Interactive landscape image projection system and control method

Abstract

The present invention relates to an interactive landscape image projection system and control method that recognizes the movement of visitors in real time, synthesizes and projects landscape images and footprint effects, and provides a multi-sensory experience. The system of the present invention utilizes an infrared camera, a vision camera, LiDAR, and a laser sensor to detect footprint locations through high-speed parallel computation and enhances accuracy using a multi-data fusion algorithm. The server synthesizes footprint effects and various landscape videos in real-time and projects them onto a projector, while synchronously outputting sound, lighting, scent, and haptics. AI environment analysis and intelligent response modules learn temperature, humidity, illuminance, noise, etc., to automatically optimize content, thereby enabling immersive interactions for visitors.

Description

Technology Field

[0001] The present invention relates to an interactive landscape image projection system and control method, and more specifically, it belongs to the field of technology that recognizes the movement or location of a visitor in real time, projects landscape images such as waves and footprint effects in response, and provides an immersive experience by linking lighting, sound, scent, and haptic effects.

[0002] This system utilizes multiple sensors to detect user footprints and activities within a space at high speed and with precision, and by controlling various output devices, it enables the creation of landscapes that allow for real-time interaction in public places, exhibition halls, theme parks, and the like. Background Technology

[0004] Recently, there has been a surge in demand for immersive content that reflects visitors' movements and reactions in real-time in various experiential spaces, such as public places, theme parks, and exhibition halls.

[0005] In particular, systems that link visual and auditory effects by allowing video projected onto the floor or walls to interact with the movements of the audience are attracting attention as landscape design solutions that go beyond simple audiovisual experiences and change the atmosphere of the entire space in real time.

[0006] Here, “landscape” is understood as a concept encompassing not only natural scenery such as mountains, fields, rivers, and seas, but also the entire landscape that reflects specific regional and cultural elements.

[0007] The present invention is not limited to a specific theme for such landscapes, and can be extended to various natural or artificial landscapes as subjects for production, using seaside waves and footprint effects as examples.

[0008] For example, not only beach waves but also themes such as snowfields, forest paths, deserts, and city squares can be implemented through the same system.

[0009] However, conventional interactive video projection systems have been limited to relying primarily on a single sensor or synchronizing only video and audio, and thus had limitations in the ability to recognize each footprint at high speed and precision when multiple viewers move simultaneously, and to naturally synthesize and project landscape images and footprint effects based on their locations.

[0010] Furthermore, features to expand into a multi-sensory experience by synchronizing various output devices such as lighting, scent, and haptics with video, or to automatically optimize content based on environmental data such as temperature, humidity, illuminance, and noise of the space, were not sufficiently implemented.

[0011] There is a growing need for technology that learns visitors' experience patterns to actively recommend highly preferred themes and flexibly adjusts the mixing method of landscape images and footprint effects according to spatial conditions.

[0012] Accordingly, in order to provide a consistent and immersive experience for visitors while they are in the space, there is a need for a new type of interactive landscape video projection system and control method that organically integrates and manages various sensor data and output devices, and can mix landscape images and footprint effects at high speed and project them in real time.

[0014] Prior art regarding conventional interactive video systems is characterized by the following configuration.

[0015] Prior Art 1 (Registered Patent No. 10-2711579, published on September 30, 2024) relates to an interactive projector control system that detects user movements and distances through an infrared sensor and a recognition camera, and recognizes user gestures using a gesture learning algorithm and motion history images. This technology focuses on functions that change content or generate screen effects in conjunction with specific movements or poses.

[0016] However, this technology does not have the function to individually identify the location of each visitor's footprints in real time when multiple visitors walk on the floor simultaneously, and to immediately generate and project wave or footprint effects in response to those coordinates.

[0017] Prior Art 2 (Published Patent No. 10-2023-0126040, published on August 29, 2023) relates to an omnidirectional experience space that projects 360-degree video onto the walls and floor inside a box-shaped space and outputs an interaction screen according to the user's actions using motion detection sensors and touch detection sensors. This technology focuses on a method of changing the content of the entire screen when the user moves their hand or body inside the space or touches the wall.

[0018] However, this technology also does not include a technical configuration that specifically distinguishes and recognizes the location of each visitor's footprint in real time the moment multiple visitors step on the floor, and generates individual wave or footprint effects corresponding to that location.

[0019] Prior Art 3 (Registered Patent No. 10-2103269, published on April 23, 2020) relates to a landscape lighting control system that applies various sensors to automatically control the lighting, brightness, and color of landscape lighting. This system aims to detect the surrounding environment using cameras and fine dust sensors and to control lighting and sound effects, but it does not provide a function to project visual effects onto specific locations on the floor in conjunction with the movement of visitors. In particular, technical means to distinguish and track the location of each footprint in real time to generate individual wave effects when multiple visitors move simultaneously are not presented.

[0020] As such, conventional technologies have focused only on gesture recognition, behavior recognition of the entire space, and lighting and sound control, respectively, but they have limitations in that they cannot realize the function of precisely identifying the locations of footprints left by multiple visitors walking on the floor in real time and individually generating and projecting wave effects that immediately correspond to each location. Prior art literature

[0022] Registered Patent No. 10-2711579 (Publication Date: September 30, 2024) Published Patent No. 10-2023-0126040 (Publication Date: August 29, 2023) Registered Patent No. 10-2103269 (Publication Date: April 23, 2020) The problem to be solved

[0023] The present invention aims to solve the problems of the aforementioned prior art and has as its main objective the provision of a system capable of implementing an immersive interactive landscape image that responds immediately to the movement of a visitor by accurately recognizing in real time the location where a visitor’s foot touches while walking on an open floor space, immediately generating a footprint effect at that location, and naturally synthesizing and projecting it with a wave image.

[0024] In addition, another objective of the present invention is to provide a natural and intuitive interaction experience even in large-scale spaces by distinguishing and detecting the location of each visitor's footprints in real time, even when multiple visitors are moving simultaneously, and rapidly outputting a corresponding individual footprint effect.

[0025] In addition, the present invention aims to provide an interactive landscape image projection system and a control method thereof that can enhance both the immersion and operational convenience of an experience space by efficiently combining various types of sensors to improve recognition accuracy and response speed, and by enabling users to intuitively manage images, sounds, and effects through a system operation terminal. means of solving the problem

[0027] To solve the above problem, the present invention provides an interactive landscape image projection system and a control method thereof that detects in real time the location where a visitor’s foot touches while the visitor moves freely in an open floor space, immediately generates a footprint effect at that location, and naturally blends and projects it with a wave image.

[0028] According to one embodiment of the present invention, an interactive landscape image projection system comprises a grid floor having a grid pattern formed with an infrared photosensitive material, and at least one left and right projector that projects a wave image onto the grid floor. To detect the location of footprints in real time when a visitor walks on the floor, various detection sensors such as a laser sensor, a LiDAR sensor, a camera, and an infrared camera are provided.

[0029] As the primary means of recognizing footprint locations, an infrared camera captures infrared patterns projected onto a grid floor and detects in real-time changes where the pattern is obscured or distorted by the visitor's foot. This allows for the calculation of the coordinates of the foot contact location in high resolution. Additionally, a vision camera captures the floor in real-time and uses image processing algorithms to detect the contours and colors of the visitor's legs and feet, thereby identifying the individual location of each footprint. This method enables independent distinction of footprints without confusion, even when multiple visitors are moving simultaneously.

[0030] LiDAR and laser sensors are used to detect the overall presence of visitors or to recognize height differences between the soles of the feet and the floor as an auxiliary means, and the signals from these sensors are utilized to complement the recognition data of infrared and vision cameras.

[0031] To rapidly identify the locations of multiple visitors' footprints, the present invention adopts a parallel computing method based on high-speed frame processing, and the server operates with a multi-threaded structure that simultaneously processes real-time data collected from each sensor. Accordingly, even if visitors step at multiple locations simultaneously, the coordinates of each footprint can be extracted with a fast response speed, and individual footprint effects can be generated at each coordinate in real time. Furthermore, by using a multi-data fusion algorithm that comprehensively analyzes high-resolution camera images and changes in infrared patterns, recognition errors can be minimized and the distinguishability of each footprint location can be enhanced.

[0032] The server according to the present invention includes an image unit that stores wave images and footprint effects, a footprint detection unit that detects the location of a visitor's footprints, an image mixing unit that synthesizes wave images and footprint effects in real time, a projector output unit that outputs the synthesized image to left and right projectors, and an audio output unit that synchronizes and outputs wave sounds and footprint sounds. Through this, processing is performed so that a footprint effect is displayed at the corresponding location as soon as a visitor steps on it, and even if multiple visitors move simultaneously, each footprint location is individually recognized to naturally generate corresponding effects.

[0033] Furthermore, the present invention provides visitors with an experience as if they were walking along the seashore by generating and projecting a footprint effect in real time that corresponds to the location of the visitor's footprint, harmonizing with the image of waves crashing onto the shore. The wave image depicts continuously moving waves across the entire floor, while the footprint effect realistically portrays the visual effect of footprints left on the water when a visitor steps at a specific location.

[0034] In addition, the present invention is equipped with a function to operate a control panel through a user terminal, enabling an operator to manage the system's video, audio, and effect output status in real time. Specifically, by performing the steps of: the terminal operating the control panel according to user input (S101); the server receiving operation information (S102); the detection sensor detecting the footprint location (S103); the server synthesizing the wave image and the footprint effect (S104); and the left and right projectors projecting the synthesized image (S105), stable and intuitive system operation is supported.

[0035] As such, the present invention enables the simultaneous achievement of high responsiveness and immersion, which were difficult to realize in conventional technology, by rapidly detecting and distinguishing the locations where the feet of multiple visitors touch, immediately generating a footprint effect corresponding to that location, and naturally synthesizing and projecting it with a wave image. Effects of the invention

[0037] According to the present invention, as a visitor moves through an open floor space, the location where their foot touches is accurately recognized in real time, and a footprint effect is immediately generated at that location and naturally synthesized and projected with a wave image, thereby providing an intuitive and synchronized response to the visitor's movements.

[0038] In addition, by combining high-resolution recognition based on infrared and vision cameras with a method of analyzing changes in grid patterns, it is possible to independently distinguish the location of each footprint even when multiple visitors move on the floor simultaneously, and to process them at high speed to smoothly output individual effects corresponding to each coordinate.

[0039] Furthermore, by utilizing the server's multi-threaded parallel processing and multi-data fusion algorithms, the response speed and accuracy of footprint recognition are simultaneously improved, and high responsiveness and realism, which were difficult to implement in existing gesture-based interaction systems or simple lighting systems, can be realized.

[0040] In addition, since video, audio, and effects can be intuitively managed through control panel operation via user terminals, it enhances the operator's convenience in system operation and enables the stable provision of various experiential content.

[0041] As such, the present invention can significantly improve the quality and reliability of immersive interactive experiences through precise location recognition centered on the location of visitors' footprints and real-time video and audio synthesis. Brief explanation of the drawing

[0043] FIG. 1 is a block diagram illustrating the overall configuration of an interactive landscape image projection system according to the present invention. FIG. 2 is a schematic perspective view showing the installation state of a grid floor, a projector, and a detection sensor according to an embodiment of the present invention. FIG. 3 is a diagram illustrating the detailed configuration of an interactive landscape image projection system according to one embodiment of the present invention. FIG. 4 is a diagram illustrating the control operation flow of an interactive landscape image projection system according to one embodiment of the present invention. Specific details for implementing the invention

[0044] Hereinafter, the present invention will be described in detail with reference to the drawings. The present invention may be subject to various modifications and may take various forms, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0045] FIG. 1 is a block diagram schematically illustrating the overall configuration of an interactive landscape image projection system according to the present invention.

[0046] This system is centered on a function that recognizes the location where a visitor's foot touches in real time, generates a footprint effect at that spot, and projects it along with wave footage.

[0047] The main components of the system include a detection sensor (3), a server (4), a terminal (5), and a plurality of output devices (200). The detection sensor (3) detects the location of a visitor's footprints using an infrared camera, a vision camera, etc., and the detected location information is transmitted to the server (4).

[0048] The server (4) serves as the core control unit of the system, recognizes the location of footprints, processes the stored wave images and footprint effects in real time, and then transmits the result to the output device (200). Additionally, the server is connected to the terminal (5) to allow the operator to manage video, sound, and production effects through the control panel of the terminal (5).

[0049] The output device (200) includes a projector (202), a sound device (203), a lighting / fog device (204), a haptic device (205), a smell device (206), etc., and receives an output signal from a server (4) to provide various visual, auditory, and sensory effects to the visitor experience space.

[0050] With such a configuration, the present invention can implement interactive effects tailored to the location of footprints by immediately responding to the movement of visitors, and provide highly immersive experience content.

[0051] FIG. 2 is a schematic perspective view illustrating the installation state of a grid floor, a projector, and a detection sensor of an interactive landscape image projection system according to an embodiment of the present invention.

[0052] As illustrated in the drawing, the system has a grid floor (1) with a grid pattern formed from an infrared photosensitive material placed in the center, and a plurality of projectors (202) and detection sensors (3) are installed on both the left and right sides, respectively.

[0053] The grid floor (1) is one of the core components of the present invention and is a structure designed to precisely detect the position where a visitor's foot touches and to clearly display an image projected by a projector.

[0054] In terms of function, the grid floor (1) performs two main roles.

[0055] First, a grid pattern made of infrared photosensitive material is formed on the surface, allowing an infrared camera to detect changes in the pattern projected onto the floor. When a visitor walks on the floor, the grid pattern is partially obscured or distorted by their feet; this deformation information is converted into the precise coordinates where the foot landed during the camera image processing.

[0056] Second, the reflectivity and color contrast of the surface are optimized so that the landscape image and footprint effect projected from the projector (202) are displayed clearly and without distortion on the grid floor surface. This ensures that the image quality remains constant.

[0057] The present invention is not limited to specific themes for landscape images, and can be applied to various natural or artificial landscapes as targets for production, using seaside waves and footprint effects as examples.

[0058] For example, not only beach waves but also themes such as snowfields, forest paths, deserts, and city squares can be implemented through the same system.

[0059] In terms of composition, the grid floor (1) is configured in a form in which an infrared photosensitive material or a reflector is arranged in a grid pattern at regular intervals on a flat plate made of high-strength transparent or translucent material. The grid pattern is repeated at regular intervals and sizes so that the position and spacing of the pattern can be used as a reference coordinate system in the camera's shooting screen. Additionally, the pattern can be processed so that it is visually fine or the color is adjusted to be similar to the floor color so that it does not interfere with the viewer's field of vision during the experience.

[0060] When the system of the present invention is in operation, the projector (202) projects a wave image onto the grid floor (1), and an infrared camera or vision camera captures the state in real time where the projected image and the grid pattern overlap. When a visitor's foot touches a specific grid, the corresponding grid is obscured or its brightness and pattern shape are partially deformed.

[0061] The footprint detection unit (42) of the server (4), which will be described later, analyzes the difference between the reference pattern and the real-time pattern in the camera image to calculate the center coordinates of the obscured area and transmits this to the server as footprint location data. Subsequently, the server (4) generates a footprint effect corresponding to the coordinates and outputs it again through the projector (202), so that the footprint effect is displayed immediately the moment a visitor steps on it.

[0062] In this way, the grid floor (1) simultaneously supports real-time recognition of footprints and clear image projection, thereby serving as a core foundation for the interactive experience of the present invention.

[0064] The projector (202) is a core output device that projects wave images and footprint effects received from the server (4) onto a grid floor in the interactive landscape image projection system of the present invention, thereby enabling visitors to immerse themselves in the experience space.

[0065] In terms of function, the projector (202) outputs a composite image received in real-time from the server (4) in high resolution and projects it onto the grid floor (1). The projected image is a combination of the movement of waves and a footprint effect created at the location where the viewer's feet touch, and the image is updated immediately whenever the viewer moves. Through this, the user can experience a visual reaction that looks like footprints appearing on the waves whenever they step.

[0066] Additionally, the projector (202) automatically adjusts the brightness, color, and projection angle of the image to display a uniform and natural effect on the floor surface, and performs software edge blending so that the boundary between the two projected images is smoothly connected when multiple projectors are installed.

[0067] In terms of configuration, the projector (202) uses a uniform light source and includes at least the following elements:

[0068] - Light source: High-brightness LED, laser light source, or hybrid light source

[0069] - Optical engine: Color wheel, DLP (Digital Light Processing) chip, or LCD panel

[0070] - Lens: Ultra-short focus lens or fixed focus lens, manual or automatic focusing function

[0071] - Image processor: An image processing circuit that decodes and corrects the composite image input from the server (4).

[0072] - Network Module: HDMI, DisplayPort, LAN, or Wireless Data Receiver

[0073] Projectors (202) are mainly installed on both sides of the space, and two or more projectors each cover half of the floor or a certain area to ensure continuity of the wave image.

[0074] Looking at the operation process of the projector (202), the server (4) transmits the composite image, which is processed in real time by the footprint detection unit (42) and the video mixing unit (43) described later, to the projector (202) through the projector output unit (44) of the server described later. The projector (202) transmits the received video signal to the optical engine via an internal image processor, and the optical engine generates a high-resolution image by matching each color and gradation information to the light source and lens. The generated image is projected onto the grid floor through the lens unit, creating a visual experience in the viewer's field of vision that looks like footprints are forming on a real beach.

[0075] At this time, when multiple projectors (202) are used, each projector (202) automatically corrects the overlapping area of ​​the projected screen and synchronizes the projection brightness or color to create a seamless, integrated image. Additionally, the projector (202) can respond to environmental changes by adjusting the brightness, color temperature, and screen size according to the server's command as needed.

[0076] In this way, the projector (202) serves as the core output means of the present invention, clearly displaying a real-time interactive video on the floor that immediately responds to the location of the visitor's footprints, and plays a role in maximizing the sense of immersion in the experience.

[0078] The detection sensor (3) is a core input means configured to accurately detect in real time the location where a visitor steps on the floor in the interactive landscape image projection system of the present invention, and to independently identify the location of each footprint even when multiple visitors experience it simultaneously.

[0079] In terms of function, the detection sensor (3) quickly recognizes the point where a visitor's foot touches the grid floor (1) as the visitor moves, and transmits the corresponding coordinate information to the server (4), thereby enabling the wave image and footprint effect to be output immediately. In particular, an important functional feature is that even when there are multiple visitors, each footprint is distinguished without confusion and individual coordinates are calculated.

[0080] In terms of configuration, the detection sensor (3) is configured by combining multiple sensors such as an infrared camera, a vision camera, a LiDAR sensor, and a laser sensor. Among these, the infrared camera is the primary sensor for detecting footprint locations with high resolution, capturing infrared patterns projected onto the grid floor (1) in real time and detecting changes where the pattern is obscured or distorted at the point where the visitor's foot touches. Through this, the coordinates of the location where the foot made contact can be calculated with high precision. Two or more infrared cameras are fixedly installed on the upper sides or side walls of the grid floor to minimize blind spots on the floor, and each camera is positioned to capture images while tilted at a certain angle toward the center of the floor.

[0081] The vision camera captures the floor in real time and uses image processing algorithms to recognize the contours and colors of visitors' feet, identifying the individual locations of each footprint. This method ensures that each footprint is distinguished as an independent object without confusion, even when multiple visitors are experiencing the experience simultaneously.

[0082] Two vision cameras are primarily used and are fixedly installed on upper columns or ceiling structures located diagonally across the grid floor. Each camera is tilted at a certain angle toward the center of the floor to form intersecting fields of view, allowing for the overlapping capture of the entire area of ​​the grid floor.

[0083] Furthermore, vision cameras are often installed at a higher elevation than infrared cameras, while infrared cameras are typically installed on floor-side pillars or at lower positions to precisely detect changes in infrared patterns. This relative positioning allows the two camera systems to capture the same area from different angles and heights, thereby ensuring both recognition accuracy and stability simultaneously.

[0084] In this invention, LiDAR and laser sensors serve the role of auxiliaryly detecting the location of visitors' footprints or detecting the presence of visitors and changes in height in real time. They are used together as auxiliary detection means to complement the reliability of data collected by infrared cameras or vision cameras and to increase recognition speed.

[0085] A LiDAR sensor is a device capable of measuring the distance between the floor surface and the soles of visitors' feet at multiple points by emitting laser light, primarily generating two-dimensional or three-dimensional distance data (point clouds). Based on the characteristic that the distance value momentarily shortens when a foot makes contact, the LiDAR sensor can rapidly determine whether a foot is present at a specific location. Furthermore, since LiDAR covers a relatively wide area using rotation or scanning methods, it is useful for distinguishing between areas where a foot has made contact and areas where it has not, even in situations where there are multiple visitors. In this invention, one LiDAR sensor is installed and fixed to the center of the side of the grid floor, positioned to scan nearly horizontally with the floor surface or to scan downward at a certain angle. This allows for continuous monitoring of height changes across the entire floor area.

[0086] A laser sensor is a device that performs high-speed distance measurement in a single line or a small scan area, capable of rapidly detecting changes in distance at the moment a foot contacts the floor. Compared to LiDAR, the laser sensor detects a narrower range at a faster rate and immediately recognizes the moment of contact based on distance values ​​that change in real time. In this invention, one or two laser sensors are used and installed on both sides or the lower side of a grid floor to emit a beam at a certain angle to the floor surface. This allows for the rapid detection of the state at the moment the foot touches the floor and the moment it falls.

[0087] Distance change information detected by the lidar sensor and laser sensor is transmitted to the server (4) and processed in an integrated manner together with image data collected from the infrared camera and vision camera. In particular, the lidar and laser provide auxiliary information on the time of contact and height status, thereby compensating for instantaneous movements or missing data at boundary areas that are difficult for the camera to recognize.

[0088] As such, LiDAR sensors and laser sensors play a crucial role in enhancing the precision and reliability of footprint recognition by rapidly detecting distance changes over a wide area or determining contact points within a narrow range at high speed, respectively.

[0089] Looking at the operation process of the detection sensor (3), when the interactive landscape image projection system is activated, the infrared camera detects pattern deformation by comparing the grid pattern of the reference state with the real-time pattern, thereby extracting the center coordinates of the area where the foot touches. The vision camera recognizes the outline of the visitor's foot through image processing and calculates the coordinates of the footprint by analyzing color, shape, and movement information. The LiDAR sensor and laser sensor detect changes in height and the presence or absence of the visitor's foot at the moment it approaches or comes into contact with the ground, generating a rapid response signal. The server (4) fuses and analyzes the data collected from these multiple sensors in real time to distinguish the location of each footprint at high speed and converts it into coordinates.

[0090] The footprint coordinates identified in this way are transmitted to the footprint detection unit of the server, and the server generates a footprint effect at the corresponding location and combines it with a wave image to immediately output it through the projector (202). In this way, the detection sensor (3) is designed to independently and quickly recognize the location of footprints even when multiple visitors move simultaneously, thereby playing an important role in providing base data for the immersive interactive experience of the present invention.

[0092] Next, the server (4) communicates with the terminal (5), and through the terminal (5), the system operator can manage the status of the video and effects or control the settings, which will be examined in more detail through FIG. 3.

[0093] The server (4) is the central control unit of the entire system and communicates with the detection sensor (3), the left and right projectors (202), and the terminal (5), processes the footprint location in real time, and performs video and audio output and system control. The server (4) is equipped with several functional modules internally, and each module performs the following roles.

[0094] 1. Video section (41)

[0095] The video unit (41) is configured to store and manage wave video data and footprint effect video data. When the server starts operating, the video unit loads pre-registered high-resolution video data and provides it to the video mixing unit (43) when necessary. This module also provides a linkage function so that the system operator can select or update content through a terminal.

[0097] 2. Footprint detection unit (42)

[0098] The footprint detection unit (42) recognizes the location of the visitor's footprint in real time and performs the function of precisely integrating and verifying data acquired from various sensors to generate coordinate information necessary for subsequent image processing.

[0099] The footprint detection unit (42) adopts a parallel processing method based on high-speed frame processing and a multi-threaded processing structure to accurately distinguish the location of each footprint, especially in situations where multiple visitors experience it simultaneously.

[0100] Specifically, when the footprint detection unit (42) receives data collected in real time from various detection sensors (3), such as an infrared camera, a vision camera, a lidar, and a laser sensor, it processes the data stream for each sensor simultaneously in an independent thread.

[0101] The server operates on a multi-threaded architecture, with each thread analyzing data in high-speed frame units to minimize processing waiting time.

[0102] This method supports the extraction of the coordinates of each footprint with almost no delay, even if multiple visitors step at various locations.

[0103] In addition, the footprint detection unit (42) applies a multi-data fusion algorithm that analyzes changes in high-resolution camera images and infrared patterns in combination.

[0104] This algorithm compares and verifies coordinate information acquired from each sensor with foot contour data and reflection distance data, and assigns reliability by prioritizing the adoption of data with a high degree of agreement.

[0105] In this process, time synchronization and spatial correction of sensor-specific data are performed in parallel to minimize recognition errors of footprint locations and enhance the distinguishability of individual footprints.

[0106] Thanks to this processing method, the footprint detection unit (42) supports each footprint being identified as a separate object even when multiple visitors are moving at the same time, and stably transmits high-speed and high-precision coordinate data to the video mixing unit (43).

[0108] The footprint detection unit (42) recognizes the location of the visitor's footprint at high resolution and high speed, and performs the processing in the following four steps to generate real-time coordinate data.

[0109] This module applies a parallel computing method based on high-speed frame processing, a multi-threaded processing structure, and a multi-data fusion algorithm to enable fast and accurate identification, especially in situations where multiple visitors experience it simultaneously.

[0111] 1) Data collection stage;

[0112] First, the footprint detection unit (42) collects data in real time from the detection sensor (3). The infrared camera detects changes in the grid pattern projected onto the grid floor (1), such as obscuring or distorting it by the foot, and outputs the center coordinates of the corresponding area as pixel coordinates based on the camera. The vision camera generates image data including the outline and color information of the foot to recognize the shape and location of the footprint. The LiDAR and laser sensors measure changes in the distance between the foot and the floor to quickly determine whether contact has occurred.

[0113] At this time, the data stream for each sensor is processed in parallel in an independent thread, and the server (4) is operated with a multi-threaded architecture.

[0114] This method minimizes data processing latency, enabling location detection with a response speed in milliseconds even when multiple visitors step into various locations.

[0116] 2) Internal reference coordinate system transformation and integration step;

[0117] Next, the footprint detection unit (42) converts coordinate data generated independently for each sensor into an internal reference coordinate system. The internal reference coordinate system is a common reference coordinate system established to consistently compare and fuse various forms of coordinate data acquired from the detection sensors, defined based on the center point and plane of the grid floor (1), and the coordinate units consist of actual distance units (e.g., millimeters or centimeters) rather than pixels.

[0118] In this conversion process, correction parameters and geometric transformation matrices are applied to compensate for camera mounting angles, lens distortion coefficients, and resolution differences. This enables the accurate representation of relative positions on the same plane, even for data acquired from different sensors.

[0120] 3) Data analysis and fusion stage;

[0121] At this stage, the footprint detection unit (42) compares and verifies each sensor data unified into an internal reference coordinate system and determines the final footprint coordinates by applying a multi-data fusion algorithm.

[0122] For example, if the coordinates detected by the infrared camera and the contour center of the vision camera match within a certain distance, they are determined to be the same object; if the distance value of the corresponding location from the LiDAR and laser sensors is below a threshold, the reliability is increased to that of an actual contact footprint.

[0123] This data fusion process is processed in parallel on independent threads for each sensor, enabling high-speed analysis of multi-coordinate correlations.

[0124] The footprint detection unit records the creation time (timestamp) of each footprint during this process.

[0125] This creation time is used as a key criterion for maintaining, distinguishing, and synchronizing the effects of a single footprint.

[0126] For example, if a foot remains in the same location for a certain period of time, it is considered an "ongoing footprint" based on the creation time and a new identification number is not assigned; additionally, if multiple footprints are rapidly created and disappear at close coordinates, the chronological order is distinguished based on the creation time to prevent collisions.

[0127] In addition, the generation time serves as a standard for maintaining consistent output timing of the footprint effect video and sound, and allows control to ensure that the effect lasts for a certain period of time or gradually fades away.

[0128] This data is also utilized for system logging and statistical analysis, serving as foundational data for calculating information such as dwell time, density, and preferred areas.

[0130] This analysis and fusion process is processed at high speed in parallel on independent threads for each sensor, continuously comparing and verifying the correlations between the data generated by each sensor.

[0131] When the footprint detection unit (42) detects adjacent coordinates at the same time, it calculates the degree of agreement by combining the distance error, the degree of pattern agreement, and the contact determination results of the lidar and laser.

[0132] A high confidence score is assigned to these coordinates with high agreement, and the system prioritizes adopting them to determine the center coordinates of the footprint.

[0133] In addition, each footprint is assigned a unique identification number (ID), which is linked to the time the footprint was created and used to track and manage the persistence of the same footprint.

[0134] Through this method, even when footprints from multiple visitors are generated at various locations almost simultaneously, the system can independently distinguish each footprint as a separate object, preventing effect output conflicts or coordinate overlaps.

[0135] In addition, by combining parallel computation based on high-speed frame processing with multiple data fusion algorithms, the entire process from collection and analysis to verification is completed in milliseconds.

[0136] Accordingly, reliable coordinate data is stably secured even in complex situations where visitors' movements are rapid or multiple people move simultaneously, and footprint effects and video synthesis can be synchronized and output in real time in subsequent stages.

[0138] In this way, during the data analysis and fusion stage, the footprint detection unit (42) processes the data of each sensor in parallel at high speed in an independent thread, and evaluates the consistency and reliability of the coordinates using a multiple data fusion algorithm.

[0139] Footprints are assigned a unique identification number (ID) and a creation time to track the persistence of identical footprints and accurately distinguish between the footprints of multiple visitors. Through this process, reliable coordinate data is secured in real-time even when multiple visitors move simultaneously, allowing for stable utilization in subsequent video synthesis and multi-sensory output.

[0141] 4) Data transfer step;

[0142] The coordinate data that has been verified and integrated is finalized as precise coordinate data based on the camera.

[0143] Afterward, the footprint detection unit (42) transmits metadata, including the time of creation, center coordinates, area, confidence score, and identification number (ID), to the video mixing unit (43) of the server. Based on this data, the video mixing unit (43) performs conversion and correction into the projection coordinate system of the projector and synthesizes the wave image and footprint effect in real time.

[0144] This allows the floor's video, sound, lighting, scent, and haptics to all be synchronized and output instantly the moment a visitor's foot touches it.

[0146] In this way, the footprint detection unit (42) combines parallel computation based on high-speed frame processing and a multi-data fusion algorithm to quickly and precisely identify the footprints of multiple visitors, and plays a pivotal role in providing accurate and reliable coordinate data necessary for subsequent image processing and projection by integrating and verifying sensor-specific data into an internal reference coordinate system.

[0148] 3. Video mixing section (43)

[0149] The video mixing unit (43) performs a series of processes to synthesize the wave image and footprint effect in real time and output them at an accurate location based on the coordinate data transmitted from the footprint detection unit (42).

[0150] The specific operation process is as follows.

[0151] 1) Coordinate transformation processing step:

[0152] The video mixing unit (43) receives footprint location data of the internal reference coordinate system received from the footprint detection unit (42). Since this internal reference coordinate system is based on the shooting coordinates of the camera, the coordinate system is different from the projection coordinate system of the projector.

[0153] Therefore, for accurate projection, it is necessary to transform the coordinate system using a coordinate transformation matrix. This coordinate transformation matrix is ​​obtained during the calibration process performed during system installation.

[0154] In the calibration process, a projector first projects a regular pattern (e.g., a checkerboard or dot pattern) onto a grid floor, and an infrared camera and a vision camera capture this pattern from multiple angles to collect the coordinates of each point into a camera-based pixel coordinate system.

[0155] Subsequently, the captured coordinates and the original coordinates output by the projector are matched as corresponding points, and the server's correction processing module calculates the transformation matrix between the two coordinate systems by applying the least squares method and geometric correction algorithms.

[0156] The coordinate transformation matrix generated in this way includes rotation, translation, scaling, and perspective correction information, and is stored in the image mixing unit (43).

[0157] By applying this matrix, the detected footprint coordinates are accurately converted from the internal reference coordinate system to the projector's projection coordinate system, and are subsequently used for geometric distortion correction and image synthesis.

[0158] This converts the camera's footprint coordinates into an accurate location on the actual projector screen.

[0160] 2) Geometric distortion correction step:

[0161] In the coordinate transformation processing step of the video mixing unit (43), the coordinates transformed from the internal reference coordinate system to the projector projection coordinate system may actually be distorted on the screen due to the installation angle or projection distance of the projector, lens characteristics, etc.

[0162] Typically, when the projector is not installed vertically on the grid floor (1) but is tilted at a certain angle, the output image is deformed from its original rectangular shape into a trapezoid (keystone), and if this is not corrected, the footprint effect is displayed inaccurately on the floor.

[0163] To solve this problem, the video mixing unit (43) applies a geometric distortion inverse correction algorithm.

[0164] The inverse correction process is carried out as follows.

[0165] First, during the system installation phase, geometric parameters of the projector, such as the projection angle, lens center position, and screen distance, are measured.

[0166] This data defines the projection model between the camera and the projector, and calculates not only coordinate system transformations but also distortion patterns.

[0167] For example, if a reference pattern (checkerboard pattern) is projected while tilted at a certain angle and captured by a camera, it is possible to identify that the square grid is distorted into a trapezoid in the field of view.

[0168] Based on these correction parameters, the video mixing unit simulates the coordinate-transformed footprint center point and the position and shape on the screen where the footprint effect image will actually be projected.

[0169] The distortion correction map generated at this time is a data structure that records the "correspondence between the original display position and the distorted position" for each pixel on the screen, and it performs an inverse conversion when outputting the footprint effect video.

[0170] When processing in real-time, this distortion correction map is applied immediately when the footprint effect image is generated.

[0171] For example, if the coordinates in the projection coordinate system are (X, Y), the image is corrected according to the correction map so that it corresponds to the pixel at (X', Y') during actual output.

[0172] This allows the footprint effect to be displayed in accordance with its actual location and shape on the floor without geometric distortion.

[0173] In addition, if the projector has aspherical distortion (barrel distortion, pincushion distortion) inherent in the lens itself, the distortion correction algorithm applies an additional lens distortion model to correct even minute deformations at the edges of the image.

[0174] This process is performed in real-time on a frame-by-frame basis, maintaining distortion-free projection even when the footprint effect is instantly updated according to the viewer's foot movements.

[0175] Through such geometric distortion correction, the projected image viewed by the audience is displayed in a form that exactly matches the actual grid floor, enabling a highly immersive experience.

[0177] 3) Footprint Effect Creation and Mapping Step:

[0178] The video mixing unit (43) calls the footprint effect data stored in the video unit (41) and places the center point of the footprint effect according to the converted and corrected coordinates.

[0179] Each footprint is managed by a unique identification number (ID), and individual effects are generated and placed to prevent footprints from overlapping even when multiple visitors are present.

[0181] 4) Wave image and real-time synthesis step;

[0182] The video mixing unit (43) performs processing to synthesize the basic background image and the footprint effect image in real time to generate the final image to be output to the projector.

[0183] The purpose of this stage is to provide an experience where footprint effects are naturally superimposed on the wave image as the viewer walks on the grid floor (1), making it feel as if they are stepping on a real beach.

[0184] First, the video mixing unit (43) begins compositing based on the basic background video. The basic background video is a high-resolution wave video stored in the video unit (41), and is played in chronological order so that the flow and texture of the waves change continuously. The wave video may be played repeatedly periodically or switched to various theme videos depending on the situation.

[0185] Next, a footprint effect image to be displayed at the corresponding coordinates is generated by referring to the footprint coordinate data transmitted from the footprint detection unit (42). The footprint effect is processed in the form of a separate transparent layer, and various attributes are assigned to the display state of each footprint.

[0186] The main attributes are as follows:

[0187] - Transparency (Alpha Value): It is displayed in a semi-transparent form that is not completely opaque, so that the footprint effect overlaps appropriately with the wave image. It starts in a sharp state immediately after display and gradually decreases in transparency over time, fading away naturally.

[0188] - Size: The size of the footprint is adjusted according to the visitor's foot size or system settings.

[0189] - Duration: Specifies the duration for which the footprint effect remains on the screen after it is displayed. This time may be set differently depending on the experience content or theme.

[0190] Each of these attributes can be adjusted in real time by an operator at the control unit (46) of the server or the control panel (51) of the terminal (5).

[0191] The video mixing unit (43) superimposes this footprint effect layer onto the wave video using a layer compositing method. During compositing, the graphics engine calculates the color value and transparency value of each pixel and smoothly fuses the two layers. Through this process, the boundary is naturally expressed when the footprint is buried in or overlaps with the waves.

[0192] In addition, when multiple footprints are displayed simultaneously, each footprint layer is managed separately, and their transparency, display time, and coordinates are maintained independently. As a result, even when multiple visitors experience the feature at the same time, each footprint effect is treated as a separate object, preventing confusion.

[0193] The final image generated as a result of the composite processing is transmitted to the projector after undergoing edge blending and synchronization processes in a subsequent step.

[0194] In this way, the video mixing unit (43) superimposes wave images and footprint effects in real time and precisely adjusts the attributes of each effect to create a visually smooth and natural immersive experience.

[0196] 5) Multi-projector edge blending and synchronization step;

[0197] In the present invention, when the experience space is large or high-resolution projection is required, two or more projectors are installed to project an image onto the grid floor (1).

[0198] At this time, the projection areas of each projector overlap to a certain extent, and if the color and brightness do not match in the boundary area, the boundary line may be clearly visible on the screen, causing a sense of incongruity.

[0199] Additionally, if the footprint effect is output across the projection areas of two projectors, the footprints will appear misaligned if the output timing or colors do not match.

[0200] To solve this problem, the video mixing unit (43) performs edge blending processing and output synchronization processing together.

[0201] First, edge blending is a process of gradually adjusting the brightness and color of each image in the overlapping area existing at the projection boundary of two projectors.

[0202] Specifically, the server measures the projection range and boundaries of each projector during initial installation and generates pixel-level blending profiles for the boundary areas.

[0203] This profile is defined in such a way that the brightness of the two images is kept the same at the center of the overlapping area, but the brightness of each projector decreases linearly as it approaches the boundary.

[0204] For example, the pixel projection brightness of one projector starts at 100% on the left side of the boundary and decreases to 0% at the right end, while the other projector increases from 0% to 100% in the opposite direction.

[0205] When the images from the two projectors adjusted in this way are superimposed, the boundary lines are inconspicuous, and they appear as a single, naturally connected screen.

[0206] In addition, the video mixing unit (43) performs color matching processing to correct differences in color temperature or color space for each projector.

[0207] To achieve this, gamma curves and color correction matrices are applied to each RGB channel to ensure that colors remain as consistent as possible when the same content is output from two projectors.

[0208] Next, output synchronization processing is the process of controlling output timing and content to ensure they exactly match when footprint effects or wave images are displayed across multiple projectors.

[0209] The server (4) manages the frame synchronization signal of each projector and unifies the timing of the generation of the output frame.

[0210] For example, if the footprint effect spans the boundary between two projectors, a sync clock is used to synchronize the output timing so that the same frame is delivered and displayed on both projectors simultaneously.

[0211] Thanks to this processing, the footprint effect does not break or waver even when visitors walk past the boundary line.

[0212] In addition, the server monitors the status of the projector in real time and immediately sends a rebalancing signal to maintain synchronization if latency or frame drops occur.

[0213] In this way, the video mixing unit (43) combines edge blending and output synchronization processing to realize natural video projection in which brightness, color, and timing are consistently maintained even in an environment where multiple projectors are installed.

[0215] 6) Final image output data generation step;

[0216] When all coordinate transformations, distortion corrections, and synthesis are completed, the video mixing unit (43) generates a final output frame and transmits it to the left and right projector output units (44).

[0217] This final output data includes location and status information of the footprint, the current frame of the wave image, and blending processing information.

[0218] In this way, the video mixing unit (43) performs coordinate transformation and distortion correction based on footprint coordinate data, and plays a key role in supporting the accurate output by naturally synthesizing the wave image and footprint effect in real time.

[0220] 4. Left and right projector output sections (44)

[0221] The left and right projector output unit (44) performs the role of accurately and stably transmitting the final composite image generated in the video mixing unit (43) to the left and right projectors (202), which are actual output devices.

[0222] This configuration goes beyond simply transmitting video signals to comprehensively handle output quality management, status monitoring, and timing synchronization.

[0223] First, in the video transmission processing, the final frame data to be output is received from the video mixing unit (43).

[0224] This data features footprint effects naturally composited with wave images and consists of separated frame buffers tailored to the screen area of ​​each projector.

[0225] The left and right projector output units (44) encode this data according to the resolution, color format, and frame cycle and transmit it to each projector.

[0226] In this process, if projectors with different resolutions are used, scaling is performed to unify the resolution and ensure display without distortion.

[0227] In addition, the left and right projector output sections manage output timing synchronization.

[0228] If multiple projectors are installed, footprint effects or wave images may appear misaligned from the boundaries if the same frame is not displayed simultaneously on different projectors.

[0229] To prevent this, the output section refers to the Sync Clock to unify the transmission and display timing of all output signals.

[0230] This process is repeated in real time, and a readjustment signal is automatically sent when the output is delayed or frame drops occur.

[0231] In addition, the left and right projector output sections (44) also perform output quality management functions.

[0232] The left and right projector output sections continuously monitor the brightness, color, and gamma values ​​of each projector and automatically adjust them as needed.

[0233] For example, if the brightness of one projector decreases, gamma correction and color correction matrices are applied to maintain consistent color with the other projector.

[0234] This correction data is shared with the control unit (46) of the server, and the operator in the control unit may manually change the settings.

[0235] In addition, the left and right projector output sections (44) include status reporting and notification functions.

[0236] Various diagnostic data such as the projector's connection status, temperature, lamp life, and signal status are periodically collected and reported to the control unit (46) of the server.

[0237] When a problem occurs, the control unit (46) immediately displays a notification on the control panel of the terminal (5) and requests the operator to respond.

[0238] In this way, the left and right projector output units (44) go beyond simple transmission functions and perform image data quality control, output synchronization, and real-time status monitoring, thereby playing a key role in ensuring that the footprint effect is always displayed at the correct location without error.

[0240] 5. Sound output unit (45)

[0241] The sound output unit (45) is responsible for generating and outputting sound that is precisely synchronized with the visual content to maximize the immersion of the footprint effect and the wave video.

[0242] This configuration goes beyond simply playing audio sources; it performs real-time synchronization to ensure sound is output immediately in accordance with the location of footprints or the timing of effect occurrences, and manages sound quality.

[0243] First, the sound output unit (45) retrieves sound data such as wave sounds, footstep sound effects, and background music from the server's internal storage or an external database.

[0244] In this case, the footstep sound effect is not a single sound source, but can be composed of multiple samples to express various surface effects or changes in intensity.

[0245] For example, samples are distinguished so that the acoustic texture varies depending on whether the footprint location is at the edge of the wave or in the center.

[0246] The sound output unit receives information regarding the timing, location, and duration of the footprint effect transmitted from the video mixing unit (43).

[0247] Referencing this data, the timing of audio playback is adjusted to match the precise timing of the video output.

[0248] For example, if a footprint effect is projected the moment a visitor steps on it, audio synchronization logic is applied so that the footprint sound is output at the same moment as the display time of that frame.

[0249] Additionally, the sound output unit (45) processes the background sound so that it is continuously modulated as the wave image changes over time.

[0250] Volume, frequency, stereo position, etc., can be adjusted in real time according to the size or period of the waves and screen transition effects.

[0251] This process is synchronized with the wave video playback state, ensuring that the sound flows naturally as the audience moves.

[0252] The sound output unit (45) finally transmits the synthesized sound signal to the sound device (203).

[0253] When transmitting, sound quality is managed including output volume, channel separation (e.g., stereo, surround), reverberation effects, etc., and if necessary, the operator can manually adjust it at the control unit (46).

[0254] In addition, the sound output unit (45) monitors the status of the sound device and immediately reports to the control unit of the server if an output abnormality or connection error occurs.

[0255] In this way, the sound output unit (45) synchronizes the sound with the display timing of the footprint effect and the wave image, and controls and outputs the sound effect in real time to provide a sense of space and realism, thereby performing a key function of maximizing the immersion of the audience.

[0257] 6. Control unit (46)

[0258] The control unit (46) manages and controls all functional modules of the server (4) and is a core control component that ensures the stability and real-time operability of the system.

[0259] First, the control unit monitors the overall system status.

[0260] The operation status, data processing speed, and communication connection status of each of the video unit (41), footprint detection unit (42), video mixing unit (43), left and right projector output unit (44), and sound output unit (45) are checked in real time.

[0261] During the monitoring process, status information periodically transmitted by each module is collected, and based on this, the normal operation status and processing load are evaluated.

[0262] Next, error detection and recovery processing are performed.

[0263] When an error code or abnormal behavior is detected in each module, the control unit immediately generates a log and sends a retry or reset signal to the corresponding module.

[0264] For example, if the footprint detection unit (42) does not receive data for a certain period of time or longer, it sends a data re-request to the detection unit and instructs the projector output unit (44) to pause output. At this time, the error status is stored in the server's internal log and is displayed as a notification on the control panel (51) of the terminal (5).

[0265] In addition, the control unit is responsible for the system start and stop procedures.

[0266] When an operator inputs a start command via the terminal, the control unit collectively manages the initialization order of each module to execute the boot sequence.

[0267] Even upon termination, the operation of all modules is safely stopped, the final state is saved, and power is released sequentially.

[0268] In addition, the control unit also performs the function of distributing control commands.

[0269] Commands entered through the control panel (51) of the terminal (5) are received by the control unit and distributed to appropriate modules according to the type of command.

[0270] For example, if the operator changes the display duration of the footprint effect, the command is transmitted to the video mixing unit (43).

[0271] When the operator adjusts the sound volume, the command is transmitted to the sound output unit (45) and immediately reflected.

[0272] Finally, the control unit performs integrated management of status information.

[0273] The status values ​​and logs reported from each module are integrated and transmitted to the terminal (5), allowing the operator to check the overall status of the system at a glance.

[0274] This information includes the operating status of each module, error history, and the status of data currently being processed.

[0275] In this way, the control unit (46) manages the overall operation flow and state of the system, ensures the stable operation of each component, and performs the pivotal function of reflecting control commands input by the operator in real time.

[0277] In addition, the server (4) further includes a module that controls a lighting / fog device (204), a haptic device (205), and a smell device (206) included in the output device (200) shown in FIG. 1, thereby supporting an immersive experience encompassing sight, touch, and smell.

[0278] First, the lighting / fog control module is a configuration that integrates the management of lighting fixtures and fog generators.

[0279] The lighting function adjusts the ambient light of the experience space or switches colors to match the themes of the wave video and footprint effects.

[0280] For example, when the wave image is crashing strongly, the lighting turns blue, and when the footprint effect is displayed, a spotlight can be projected onto a specific area.

[0281] In addition, the fog generator sprays fine water vapor from a low position to visually emphasize the texture of the waves.

[0282] The control unit (46) of the server controls lighting and fog effects in real time based on the status information of the current wave image received from the video mixing unit (43), and activates local lighting or adjusts the fog density in conjunction with the section where the footprint effect occurs.

[0283] Second, the haptic control module controls the haptic device (205) to perform the function of transmitting vibration or pressure to the soles of the spectators' feet.

[0284] This device is connected to a vibration unit embedded in the lower part of the grid floor or attached to the surface, and outputs haptic feedback synchronized to the corresponding location when a footprint effect is displayed.

[0285] For example, when a visitor steps on it, a vibration stimulus is generated at a location that matches the coordinates of the footprint effect, providing a tactile sensation as if stepping on real sand or waves.

[0286] The haptic control module refers to the coordinate information transmitted from the footprint detection unit (42) of the server and automatically adjusts the intensity and pattern of vibration according to the timing of creation, duration, and extinction of the footprint.

[0288] Third, the olfactory control module is responsible for the function of spraying scent into the experience space through the olfactory device (206).

[0289] This module refers to scent patterns stored in the server's internal effects database and automatically selects and outputs a scent suitable for the video and situation.

[0290] For example, it sprays a beach scent when video of calm waves plays, and switches to a sea breeze scent during scenes where waves crash hard.

[0291] In addition, when the footprint effect appears, the scent can be sprayed intensely in an instant to increase the user's concentration.

[0292] The olfactory control module manages the spray intensity, duration, interval, etc., and allows the operator to manually adjust the scent theme or spray cycle through the terminal (5).

[0293] In this way, the server (4) can implement a multi-sensory interaction experience in real time that goes beyond simple audiovisual experience and encompasses touch and smell by additionally providing lighting / fog, haptic, and olfactory modules.

[0295] In addition, the server (4) is equipped with an ambient lighting linkage system.

[0296] This system includes various colored LED lights and fog generators installed around the grid floor (1), on the walls or ceiling of the exhibition hall, and dynamically creates the atmosphere of the entire experience space in conjunction with video content.

[0297] The control unit (46) of the server receives status information of wave images and footprint effects transmitted from the video mixing unit (43) in real time, and transmits commands to the lighting / fog control module based on this.

[0298] The lighting / fog control module adjusts the color, brightness, and lighting patterns of each lighting fixture according to the corresponding command, and controls the spraying timing and intensity of the fog generator as needed, thereby realizing an immersive environment that responds immediately to the movement of visitors and changes in the video.

[0299] The lighting / fog control module primarily controls the status of each lighting fixture and fog generator, and performs the role of dynamically adjusting parameters such as color, brightness, and spray pattern.

[0300] For example, in scenes where waves are crashing strongly, the surrounding lights are turned blue or flashing effects are produced, and in sections where the footprints of visitors occur, the lights in that area are momentarily illuminated or the colors are changed to add visual emphasis effects.

[0301] In addition, the fog generator sprays fine water vapor from a low position to realistically express the texture of waves.

[0302] These lighting and fog effects do not simply operate as preset patterns, but are linked with the video mixing unit (43) of the server (4) and change in real time according to the situation.

[0303] For example, when a video of sunset waves is projected, the surrounding lighting color gradually changes to red or orange, creating the effect that the entire exhibition space feels like a beach at sunset.

[0304] Likewise, when the intensity and frequency of the footprint effect are high, the lighting and fog effects also react immediately, providing sensory feedback to the audience's interaction.

[0305] The control unit (46) is responsible for the upper control function of this lighting / fog effect.

[0306] The control unit receives status information of wave images and footprint effects in real time from the video mixing unit (43) and transmits a production command to the lighting / fog control module based on this.

[0307] For example, after comprehensively analyzing data such as the current wave video theme, the active coordinates of the footprint effect, and the effect intensity, it instructs which color transition or fog spray pattern to apply.

[0308] The lighting / fog control module receives this higher-level control command and finely adjusts the output status of individual lighting devices or fog generators.

[0309] Through this hierarchical control structure, the system can express visual and sensory elements within the experience space as a single integrated theme fully linked with video, providing visitors with an immersive interaction environment that goes beyond simply watching floor projections, making the entire space feel as though it is alive and breathing.

[0311] In addition, the server (4) is equipped with an olfactory and gustatory stimulation system.

[0312] This system is designed to realize a theme park-style presentation that satisfies the five senses of visitors, going beyond the main experience space.

[0313] The olfactory and gustatory stimulation system consists of an olfactory device (206) installed around a grid floor (1) that sprays a scent, and a themed food and beverage serving facility that is optionally configured, and a scent and taste database and an olfactory control module are integrated inside the server.

[0314] The control unit (46) of the server is responsible for the upper control function of the system and analyzes the current wave image status, footprint effect generation information, sound playback status, etc. received from the video mixing unit (43) in real time.

[0315] Based on this analysis information, the control unit selects a scent or taste stimulation effect suitable for the current performance situation and transmits a command to the sub-control module.

[0316] In addition, it supports the operator to manually adjust the scent theme, spray interval, duration, etc. through the terminal (5).

[0317] The olfactory control module controls the scent dispensing device in detail according to the commands of the upper control unit.

[0318] A fragrance dispenser is a device that sprays fine fragrance particles into the air and can produce various themed scents, such as beach scents (salty or cool scents), coconut scents, and sunscreen scents.

[0319] For example, when video of calm waves plays, it sprays a soft and comforting scent characteristic of the beach, and during scenes where waves crash strongly, it momentarily sprays a cooler and more vivid scent to enhance immersion.

[0320] When the footprint effect is displayed on the screen, a short, strong scent can be sprayed at that moment to maximize the audience's concentration.

[0321] The olfactory control module precisely adjusts spray intensity, duration, and spray interval, and naturally switches between various types of scents according to the scenario.

[0322] In addition, this system can be connected to food and beverage facilities in specific theme parks or event spaces through a taste experience linkage function.

[0323] Once visitors finish the interactive experience, the server recommends menu items such as 'beach-themed' ice cream or beverages, or expands the gustatory experience by linking with an automated ordering system.

[0324] This function goes beyond mere olfactory stimulation to realize a comprehensive production effect that connects the sensory immersion felt by visitors in the space to a gustatory experience.

[0325] In this way, the server (4) oversees the upper control of the olfactory and gustatory stimulation system through the control unit, and the olfactory control module performs detailed spray control, scent conversion, and taste linkage functions at the lower level, thereby supporting a multidimensional experience that encompasses olfactory and gustatory senses beyond sight and hearing in real time.

[0326] Through this, the system of the present invention can go beyond simple lighting and video devices to provide visitors with a more sensory and rich theme park-style landscape experience.

[0328] In addition, the server (4) may be equipped with a module that controls a citizen participation type interactive control system and an environment-responsive automatic production system to respond to various application environments of the present invention.

[0329] First, the citizen-participatory interactive control module collects and analyzes inputs such as visitors' movements, location information, and smartphone apps in real time, and reflects them in the wave video, lighting, and sound production.

[0330] This module operates in conjunction with motion sensors, location sensors, and communication sensors; the motion sensor detects movement patterns of visitors within the experience space, and the location sensor tracks the visitors' location coordinates in real time.

[0331] In addition, the communication sensor can receive signals transmitted from the visitor's smartphone app or wearable device and link individual user identification information to the experience content.

[0332] Based on the data collected in this way, the citizen-participatory interactive control module outputs customized interaction effects, such as changing wave video patterns or turning on lights, when users stay at a specific location or perform gestures like hand movements.

[0333] This feature encourages children and tourists to actively participate in the experience and improves the utilization of public landscapes and their length of stay.

[0334] Meanwhile, the environment-responsive automatic direction module is responsible for automatically adjusting video direction by monitoring external weather and environmental data in real time.

[0335] This module is linked with temperature sensors, humidity sensors, wind speed sensors, dust sensors, etc., and data collected from each sensor is input to the server at regular intervals.

[0336] The environment-responsive automatic production module selects preset production logic based on values ​​such as temperature, humidity, wind speed, and fine dust concentration.

[0337] For example, if the wind blows strongly, the movement of the wave image can be dynamically changed, and if the temperature drops, it can switch to a wave pattern of cool colors.

[0338] In addition, when the concentration of fine dust is high, fine mist-type bubbles are sprayed in conjunction with the lighting / fog device (204) to create an air purification effect, thereby simultaneously enhancing the comfort and symbolism of the experience space.

[0339] In this way, the server (4) is equipped with a citizen participation type interactive control module and an environment-responsive automatic production module, thereby supporting the flexible integrated operation of real-time responsive production of public spaces and customized participation functions for individual participants.

[0341] In addition, the server (4) may be equipped with a user data analysis and statistics module to improve the quality of the system operation and experience content of the present invention.

[0342] This module performs data logging and analysis functions integrated into the server and continuously records and manages all footprint location data collected in real time from the detection sensor (3).

[0343] While the system is in operation, visitor movement information is stored in a database in chronological order, and various statistical analyses are automatically performed based on this.

[0344] First, the number of visitors by time of day and day is calculated using the visitor counting function.

[0345] Based on the unique identification number (ID) data received from the footprint detection unit (42), it determines whether there is a duplicate and automatically counts how many visitors used the experience space during a specific period.

[0346] In addition, it performs the function of analyzing popular areas.

[0347] The frequency of footprints is accumulated and tallied for each area partitioned at regular intervals on the grid floor (1) to identify the ‘Hot Spot’ area where visitors stayed the longest or moved most actively.

[0348] This data is visualized in the form of a heat map, allowing operators to identify visitors' preferred locations at a glance.

[0349] In addition, it includes an interaction pattern analysis function.

[0350] This feature statistically analyzes experience patterns, such as which types of footprint effects visitors responded to more frequently and whether the movement trend is linear or circular.

[0351] The analysis results are used to objectively evaluate the preference for sound effects or video content, or to compare the popularity of specific theme content.

[0352] The dwell time measurement function individually tracks the time each visitor spends in front of the system and the frequency of interaction.

[0353] The time interval between the unique footprint identification number and location data is calculated to determine the time of stay, and the average time of stay and distribution are statistically derived.

[0354] These analysis results provide system operators with useful information for precisely understanding visitor behavior, evaluating the effectiveness of experience shows, and establishing future content planning and marketing strategies.

[0355] Furthermore, the accumulated data provides insights necessary for the development of future interactive systems and serves as foundational data to continuously improve system quality and operational efficiency.

[0356] In this way, the server (4) is equipped with a user data analysis and statistics module, thereby realizing an integrated management system that goes beyond simple real-time interaction to include the collection, analysis, and utilization of experience data.

[0358] In addition, the server (4) is equipped with an intelligent environment response system.

[0359] This system is designed to detect and analyze changes in the external environment of the experience space and visitor behavior patterns in real time, automatically optimize various production elements such as video, sound, lighting, scent, and haptics, and provide a consistent immersive experience.

[0360] The intelligent environmental response system is largely composed of an AI-based environmental analysis module and an intelligent environmental response module.

[0361] First, the AI-based environment analysis module is responsible for high-level analysis and learning.

[0362] This module collects real-time data such as temperature, humidity, illuminance, ambient noise, and visitor density from various sensors, and combines accumulated historical data to perform the following processes:

[0363] - Evaluate the current environmental conditions and the comfort level of the experience space

[0364] - Recommends the optimal production scenario by learning from past audience reactions and preferences

[0365] - Determine whether to switch content or adjust output when sudden environmental changes occur

[0366] For example, it generates analysis results to increase image contrast when the space becomes excessively bright, and to recommend wave images with cool tones and refreshing scents when the indoor temperature rises.

[0367] This analysis data and instructions are transmitted to the control unit (46) of the server.

[0368] The control unit (46) acts as the upper controller of the system and comprehensively reviews the results received from the AI-based environment analysis module.

[0369] Subsequently, specific control commands are distributed to intelligent environment response modules according to each condition, and the overall production status is monitored in real time.

[0370] In addition, manual settings or schedule priorities of the operator input through the terminal (5) are adjusted, and if necessary, manual operation is applied with priority over automatic direction.

[0371] The intelligent environment response module performs sub-control functions.

[0372] This module automatically controls and optimizes specific production elements based on AI analysis results and commands from the control unit, and is responsible for the following functions:

[0373] - Environment-responsive content optimization: Increases the volume of wave sounds when ambient noise increases, and adjusts video contrast and saturation when bright lighting is detected to maintain visual immersion.

[0374] - Temperature and Humidity-Based Direction Switching: When the temperature is high, it selects cool wave visuals and scents, and when it is low, it selects warm sunset visuals and soft scents to induce a pleasant experience.

[0375] - Visitor Pattern Learning: Automatically rearranges content by prioritizing highly preferred themes based on specific time slots or crowd levels.

[0376] - Multimodal Effect Synchronization: Based on the analysis results above, all effects, such as lighting color, scent dispensing, and haptic output, are consistently synchronized under the same theme.

[0377] This module reports the status of each performance element to the control unit (46) in real time and immediately notifies any status changes or abnormal signals that occur during the performance process.

[0378] In this way, the intelligent environment response system of the present invention has an AI-based environment analysis module that performs real-time data collection, evaluation, recommendation, and learning, and the intelligent environment response module automatically optimizes and controls specific production elements, and the entire process is integratedly managed by the control unit (46) of the server.

[0379] Through this, it can actively adapt to visitors and spatial conditions, reliably providing a consistent and high-quality interactive experience regardless of when a visit is made.

[0381] In the present invention, the terminal (5) is a user control device connected to the server (4) via a network to comprehensively manage and control the overall production elements of the experience space and to support real-time monitoring of the system status.

[0382] In the present invention, the terminal is implemented in the form of various general-purpose devices such as tablets, smartphones, and PCs, and is designed to allow the system to be operated flexibly regardless of location or situation.

[0383] The terminal is largely composed of a control panel (51) and a control unit (52).

[0384] First, the control panel (51) is a core section that provides a user interface (UI), allowing the operator to directly select and configure the following content and devices:

[0385] - Switch Wave Video and Theme: Change or apply a new wave video and theme currently playing.

[0386] - Footprint Effect Management: Adjust effect type, duration, transparency, etc.

[0387] - Sound content control: Adjust background music and sound effect volume, and select sound themes.

[0388] - Lighting Control: Set ambient lighting color, brightness, and flashing patterns, or switch automatically according to a schedule.

[0389] - Olfactory Control: Select and schedule fragrance type, spray intensity, duration, and spray interval.

[0390] - Haptic Control: Set the intensity and pattern of haptic feedback for footprint effects or specific animations.

[0391] Various UI elements such as buttons, sliders, list menus, and color selection tools are arranged on the control panel, and it is visualized so that even non-experts can intuitively operate the system.

[0392] The control unit (52) is responsible for sending commands input from the control panel (51) to the server (4) and receiving and displaying status information returned from the server in real time.

[0393] For example, if the lighting color is changed, the corresponding command is sent to the server's lighting / fog control module, and if scent dispensing is scheduled, the reservation data is transmitted to the server's olfactory control module.

[0394] In addition, if the haptic effect is changed, it is reflected in real time in the server's haptic control module.

[0395] The control unit continuously checks the network connection status and automatically attempts to reconnect if the connection is temporarily lost.

[0396] The command processing logic determines the priority of multiple input commands to process them sequentially without conflicts, and receives a completion response from the server to reflect it on the terminal screen.

[0397] The status information received from the server includes the following:

[0398] - Wave video playback status and theme information

[0399] - Types and quantities of currently active footprint effects

[0400] - Lighting color, brightness, and blinking status

[0401] - Fragrance spray status and remaining volume

[0402] - Haptic output status

[0403] - Overall system operation status and error information

[0404] The operator can check this information at a glance on the terminal screen and make immediate adjustments if necessary.

[0405] In this way, the terminal (5) performs the function of integrally managing and controlling various experience elements such as lighting, smell, and haptics as well as audiovisual effects, and serves as a core user interface that maximizes the sense of immersion in the experience space through stable communication with the server (4).

[0407] FIG. 4 is a diagram illustrating the control operation flow of an interactive landscape image projection system according to one embodiment of the present invention.

[0408] The entire process is broadly divided into five major stages, and each stage is performed through the organic interoperability of system operators, servers, detection sensors, projectors, and various output devices.

[0410] 1) Step 1: Input Step (S101)

[0411] The system operator uses a terminal (5) to access the control panel (51) and inputs a system start command.

[0412] At this stage, the operator can select the theme of the wave video, sound content, initial lighting settings, scent and haptic options as needed, or keep them at default values. The input commands and setting information are stored in the control unit (52) inside the terminal.

[0414] 2) Step 2: Server Transmission and Preparation Phase (S102)

[0415] At this stage, the control unit (52) of the terminal transmits the input start command and initial setting data to the server (4).

[0416] This data is reliably transmitted via a standard communication protocol, and the server's control unit (46) checks the received content and distributes initialization commands to each module of the system.

[0417] The server prepares production content based on the transmitted data. The server (4) loads a basic wave image selected from the video unit (41) and projects it onto the grid floor (1) by transmitting it to the left and right projectors (2) through the left and right projector output unit (44).

[0418] At the same time, the sound output unit (45) plays a gentle wave sound and commands the lighting / fog control module, olfactory control module, and haptic control module to an initial output state to set the basic atmosphere of the entire space.

[0419] For example, blue lights linked to the wave video are turned on, and a soft beach scent is gently sprayed.

[0421] 3) Step 3: Visitor Footprint Detection Step (S103)

[0422] When a visitor starts walking on the grid floor, the detection sensor (3) captures the movement and position of the foot in real time.

[0423] The infrared camera detects distortion in the grid pattern to generate coordinates, and the vision camera analyzes the contours and colors of the foot. LiDAR and laser sensors detect height differences to verify contact.

[0424] This data is transmitted to the footprint detection unit (42) of the server, converted and integrated into an internal reference coordinate system, and managed by assigning a unique identification number (ID) and reliability to the footprint.

[0426] 4) Step 4: Video, Audio, and Environmental Effects Mixing Step (S104)

[0427] The video mixing unit (43) of the server generates a footprint effect video based on footprint coordinate data and composites it onto the wave video in real time.

[0428] The footprint effect is equipped with transparency, duration, and animation patterns, so that ripples or glowing afterimages appear the moment a visitor's foot touches it.

[0429] At the same time, the sound output unit (45) plays the sound of splashing water or footsteps in synchronization with the footstep effect.

[0430] In addition, the control unit (46) controls the following environmental effect output together by referring to footprint generation information and image mixing status:

[0431] - Lighting / Fog Control Module: Instantly illuminates or changes the color of areas marked with footprints, and increases the fog spray intensity as needed to provide visual emphasis.

[0432] - Haptic control module: Generates vibration stimulation on the grid floor area corresponding to the footprint coordinates to provide tactile feedback to the soles of the visitors' feet.

[0433] - Olfactory Control Module: Enhances olfactory immersion by briefly spraying a beach scent or a cool sea breeze scent simultaneously with the footprint effect.

[0434] These audiovisual, tactile, and olfactory effects are synthesized and controlled in real time, ensuring that all effects are output with precise timing.

[0436] 5) Step 5: Projection and output of composite content (S105)

[0437] The finally generated composite image is transmitted to the projector (2) through the left and right projector output sections (44) and immediately projected onto the grid floor (1).

[0438] At the same time, lighting colors switch to match the production theme, and fog, haptic, and olfactory effects are displayed in synchronization with footprint coordinates.

[0439] All of this process is carried out at a processing speed in milliseconds, allowing visitors to vividly experience an immersive experience where the video and sound of the floor, ambient lighting and scent, and even tactile stimulation respond together the moment their feet touch it.

[0440] As such, the operation flow illustrated in FIG. 4 is organically carried out step by step from the start of the system to the detection of visitor movement, the generation of effects, and multi-sensory output, thereby implementing an integrated interaction in which the entire space responds in real time.

[0442] As described above, the present invention relates to a system that provides a multi-sensory experience by immediately responding to movement when a visitor walks on the floor.

[0443] In particular, the present invention increases data processing speed by applying parallel computation and a multi-threaded structure based on high-speed frame processing so that each footprint can be accurately distinguished even in situations where multiple visitors experience it simultaneously.

[0444] In addition, information collected from infrared cameras, vision cameras, LiDAR, and laser sensors is analyzed using a multi-data fusion algorithm, and each footprint is assigned a creation time and a unique identification number to track its persistence and manage it independently.

[0445] In addition, in the present invention, footprint coordinates are utilized in real time so that the generated footprint effect image is composited with a wave image and projected immediately, and at the same time, sound, lighting, scent, and haptics are accurately synchronized so that visitors can experience a vivid and multi-sensory experience as if they were walking on actual waves.

[0446] In addition, AI-based environmental analysis modules and intelligent environmental response modules evaluate and learn from temperature, humidity, illuminance, noise, and visitor density in real time to automatically apply content and effects suitable for the situation. As a result, video and audio are immediately adjusted when the space brightens or noise increases, allowing the system to actively adapt to environmental changes and maintain immersion.

[0447] In addition, the system of the present invention allows an operator to easily control key performance elements and monitor their status through a terminal, thereby flexibly combining automatic operation and manual control.

[0448] This provides visitors with an immersive experience where the entire space responds immediately, and can simultaneously increase space utilization and satisfaction with their stay.

[0450] Although the present invention has been described in detail through specific embodiments, this is for the purpose of specifically explaining the invention, and the invention is not limited thereto. It is evident that modifications or improvements can be made by those skilled in the art within the technical scope of the invention. Explanation of the symbols

[0452] 1: Grid floor 2: Left and right projectors 3: Detection sensor 4: Server 5: Terminal 41: Video Department 42: Footprint detection unit 43: Video Mixing Department 44: Left and right projector output sections 45: Audio output section 46: Control unit 51: Control Panel 52: Control unit 200: Output device 202: Projector 203: Sound device 204: Lighting / Fogging System 205: Haptic device 206: Olfactory apparatus

Claims

Claim 1 In an interactive landscape video projection system that detects the location of visitors' footprints in real time, synthesizes and projects landscape video and footprint effects, and provides various output effects in synchronization, the system comprises: a grid floor (1) having a grid pattern formed with an infrared photosensitive material for footprint detection; a detection sensor (3) for detecting the location of visitors' footprints; a server (4) that processes footprint location data collected from the detection sensor (3) to calculate center coordinate data that can individually distinguish the footprints of multiple visitors, and mixes landscape video and footprint effects in real time based on the center coordinate data and transmits a control signal to an output device (200); the server (4) includes a footprint detection unit (42) that processes data collected from the detection sensor (3) in parallel using a multi-threaded structure and calculates the center coordinate data, creation time, and unique identification number of the footprints through a multi-data fusion algorithm; and the server (4) includes an AI-based environmental analysis module that learns and evaluates at least one environmental data among temperature, humidity, illuminance, noise, and visitor density to automatically optimize the performance content, and output conditions for the footprint effects and landscape video according to the analysis results of the environmental analysis module. An interactive landscape image projection system comprising: a terminal (5) that includes an intelligent environment response module for controlling, communicates with a server (4) to input system start commands and production settings, and displays status information; and a plurality of output devices (200) that receive output signals from the server (4) and provide visual, auditory, and sensory effects to a visitor experience space, wherein the output devices (200) include at least one of a projector (202) and a sound device (203), and at least two of a lighting / fog device (204), a haptic device (205), and an olfactory device (206). Claim 2 An interactive landscape image projection system according to claim 1, wherein the server (4) generates a footprint effect image based on data calculated by a footprint detection unit (42) and includes an image mixing unit (43) that synthesizes and outputs the image in real time with a landscape image. Claim 3 An interactive landscape image projection system according to claim 1 or 2, wherein the terminal (5) comprises a control panel (51) capable of selecting or setting production content, and a control unit (52) that transmits an input signal to a server (4) and receives and displays status information of the server (4). Claim 4 A control method for an interactive landscape video projection system that recognizes the location of a visitor's footprints in real time, synthesizes and projects landscape video and footprint effects, and synchronizes output effects, comprising: an input step (S101) of inputting a system start command and production settings from a control panel (51) using a terminal (5); a server transmission and preparation step (S102) in which a control unit (52) of the terminal (5) transmits the command and setting data to a server (4), and the server (4) prepares production content based on the received data; a footprint detection step (S103) in which a detection sensor (3) detects a visitor's footprints in real time on a grid floor (1), and a footprint detection unit (42) of the server (4) performs parallel computation processing of the data collected from the detection sensor (3) in a multi-threaded structure, calculates center coordinate data capable of individually distinguishing multiple visitors' footprints, a creation time, and a unique identification number through a multi-data fusion algorithm, and generates a footprint effect based on the center coordinate data; and a video mixing unit (43) of the server (4) generates a footprint effect video based on the center coordinate data. A control method for an interactive landscape image projection system, characterized by comprising: a video, audio, and environmental effect mixing step (S104) in which an AI-based environmental analysis module of a server (4) generates and synthesizes the landscape image in real time, and an intelligent environmental response module controls the output conditions of the footprint effect and the landscape image according to the analysis result; and a composite content projection and output step (S105) in which the synthesized image is transmitted to a projector (202) of an output device (200), and one or more of an audio device (203) synchronized with the footprint effect, and two or more of a lighting / fog device (204), a haptic device (205), and an olfactory device (206). Claim 5 A control method for an interactive landscape image projection system according to claim 4, wherein the server (4) generates a footprint effect image based on data calculated by a footprint detection unit (42) and includes an image mixing unit (43) that synthesizes and outputs the image in real time with a landscape image. Claim 6 A method for controlling an interactive landscape image projection system according to claim 4 or 5, wherein the terminal (5) comprises a control panel (51) capable of selecting or setting production content, and a control unit (52) that transmits an input signal to a server (4) and receives and displays status information of the server (4). Claim 7 delete

Images