system

Abstract

We provide the system. [Solution] A theme selection means for selecting a different theme each week, A video data preparation means for preparing video data based on the selected theme, A video data distribution method that distributes prepared video data to terminals on each floor, A projection method that uses received video data to perform projection mapping on a staircase, A means of setting quest elements, such as quizzes or mini-games on specific staircases, A data collection method that collects user behavior data, quiz answer data, etc. A system that includes analytical means to analyze collected data and evaluate exercise levels and health status.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional stairs in office buildings and commercial facilities do not function sufficiently as a means for users to eliminate lack of exercise and promote health. In addition, the use of stairs is monotonous and lacks entertainment, so there is a tendency to rely on elevators. As a result, the overall amount of exercise is insufficient and the health risk increases. In addition, conventional measures to promote the use of stairs often have limited effects and lack sustainability. There is a need for a new approach to solve this problem, make the use of stairs enjoyable, and promote the health of users.

Means for Solving the Problems

[0005] This invention provides a system including a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, and an analysis means. The theme selection means selects a different theme each week, and the video data preparation means prepares video data based on that theme. The video data distribution means distributes this data to the projection means on each floor. The projection means performs projection mapping on the stairs, displaying different themes. The quest element setting means sets quizzes and mini-games on specific stairs to provide entertainment for users. The data collection means collects user behavior data, and the analysis means analyzes this data to understand the user's exercise level and health status. In this way, the system makes using the stairs enjoyable, alleviates users' lack of exercise, and promotes their health.

[0006] A "theme selection method" is a component that has the function of selecting a different theme each week.

[0007] A "video data preparation means" is a component that has the function of preparing video data based on a selected theme.

[0008] A "video data distribution means" is a component that has the function of distributing prepared video data to terminals on each floor.

[0009] A "projection means" is a component that has the function of performing projection mapping on a staircase using received video data.

[0010] A "quest element setting means" is a component that has the function of setting quizzes or mini-games on specific staircases.

[0011] A "data collection means" is a component that has the function of collecting user behavior data, quiz answer data, and so on.

[0012] "Analysis tools" are components that have the function of analyzing collected data and evaluating exercise levels and health status. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes and quest elements, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[0035] Server Role

[0036] The server acts as the controller for the entire system and performs the following main tasks:

[0037] 1. Theme selection:

[0038] The server selects a different theme each week (for example, mountain climbing, underwater, space, etc.) and prepares video data for it.

[0039] 2. Preparation and distribution of video data:

[0040] Based on the selected theme, the server prepares the video data and distributes it to the projection terminals installed on each floor.

[0041] 3. Setting Quest Elements:

[0042] The server sets up quizzes and mini-games on specific staircases and delivers them to the terminals.

[0043] 4. Data collection and analysis:

[0044] The server receives data collected from each terminal and analyzes the user's activity level and health status.

[0045] Terminal role

[0046] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server:

[0047] 1. Play back the video:

[0048] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs.

[0049] 2. Display of quizzes and mini-games:

[0050] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[0051] 3. User interaction handling:

[0052] The terminal receives and processes the user's quiz answers and mini-game results, and sends that data to the server.

[0053] User roles

[0054] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[0055] 1. Use of stairs:

[0056] Users climb the stairs while enjoying visual effects created by projection mapping.

[0057] 2. Participation in quizzes and mini-games:

[0058] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[0059] Specific example

[0060] For example, if the theme for the following week is set to "mountain climbing," the server will prepare video data related to the "mountain climbing" theme and distribute it to terminals on each floor from the 1st to the 10th floor.

[0061] On the first floor, a terminal projects an image of "the foot of a mountain" onto the staircase, and as you go up from the second floor, the scenery changes from "mountain path" to "valley" to "steep rocky area" to "snowy mountain." When a user reaches the third floor, a quiz is displayed asking, "Which of the following mountains is over 1,000 meters high?" If the user selects "Mount Fuji" from the options and answers correctly, a special effect (such as fireworks or special visual effects) is displayed on the staircase to the next floor.

[0062] The server collects data from terminals on each floor and analyzes user response data and stair usage data to evaluate exercise levels and health status. This data is used for user health improvement programs and as feedback for the entire company.

[0063] The following describes the processing flow.

[0064] Step 1:

[0065] Theme selection (server)

[0066] The server checks the current week and selects a new theme. For example, the theme for next week might be set to "Space Exploration".

[0067] Step 2:

[0068] Preparation of video data (server)

[0069] The server collects and prepares relevant video data based on the selected theme. For example, it prepares a video dataset for space exploration (spaceships, starry skies, planets, etc.).

[0070] Step 3:

[0071] Video data distribution (server)

[0072] The server sequentially distributes the prepared video data to the projection terminals on each floor. For example, it sends video data from the first floor to the tenth floor.

[0073] Step 4:

[0074] Receiving and saving video data (on the device)

[0075] The device receives video data transmitted from the server and saves it to local storage. Once saving is complete, it begins preparing for playback.

[0076] Step 5:

[0077] Projection Start (Device)

[0078] The device activates the projector at a specified time and projects images onto the walls and floor of the staircase. For example, an image of a spaceship is displayed on the staircase on the first floor.

[0079] Step 6:

[0080] Quest element settings (server)

[0081] The server prepares quiz or mini-game data for specific staircases and sends it to the terminals. For example, the 3rd floor might have a quiz like, "What is the distance from the Earth to the Moon?"

[0082] Step 7:

[0083] Quest data reception and display preparation (device)

[0084] The terminal receives data for quizzes and mini-games and prepares to display it when the user reaches the desired location. It verifies the validity of the data and prepares the interactive screen.

[0085] Step 8:

[0086] User interaction (user)

[0087] The user climbs the stairs while enjoying projection mapping. A quiz is displayed when they reach a specific floor (for example, the 3rd floor). The user then answers the quiz.

[0088] Step 9:

[0089] Receiving and evaluating responses (device)

[0090] The device receives the user's answer and determines whether it is correct or incorrect. If correct, a special effect is added to the video leading up to the next floor. For example, a special star effect is displayed when the answer is correct.

[0091] Step 10:

[0092] Data collection (device)

[0093] The terminal collects user response data and stair usage data. The collected data is then prepared for transmission to the server.

[0094] Step 11:

[0095] Data transmission (device)

[0096] The device sends the collected data to the server. This data may include, for example, each user's quiz answers and stair usage time.

[0097] Step 12:

[0098] Data reception and analysis (server)

[0099] The server receives data sent from the terminal and stores it in a database. It then begins analysis to evaluate exercise levels and health status.

[0100] Step 13:

[0101] Feedback of results (server)

[0102] The server generates a report based on the analysis results and notifies administrators and relevant parties. It reports on each user's health status and activity levels over time.

[0103] Through the steps described above, the system aims to make using stairs enjoyable for users, alleviate a lack of exercise, and promote health.

[0104] (Example 1)

[0105] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0106] In modern office buildings and commercial facilities, users' reliance on elevators leads to insufficient daily exercise and a decline in health. Furthermore, using stairs is psychologically monotonous, causing users to avoid it. Additionally, there is a lack of systems to efficiently monitor and improve health conditions and exercise levels.

[0107] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0108] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a quest element setting means, a data collection means, an analysis means, and a user interaction processing means. This makes using stairs enjoyable, helps alleviate lack of exercise, and promotes health. Furthermore, it provides psychological enjoyment by offering quizzes and mini-games, and enables efficient collection and analysis of each user's exercise data.

[0109] "Theme selection method" refers to the process of selecting a theme from the library that is appropriate for the year or season, and deciding on it as the theme for the following week.

[0110] "Video data preparation means" refers to a means for generating relevant video data based on a selected theme and preparing it for projection on each floor.

[0111] "Video data distribution means" refers to a means for distributing generated video data to terminals on each floor.

[0112] "Projection means" refers to a method for transmitting video data received by a terminal to a projector device and projecting the image onto the stairs.

[0113] A "quest element setting method" is a means of placing interactive elements such as quizzes or mini-games on a specific staircase and presenting them to the user.

[0114] "Data collection means" refers to the means of collecting user interaction and stair usage data from each terminal.

[0115] "Analysis methods" refer to methods for analyzing each user's exercise volume and quiz accuracy rate based on collected data, and generating reports to address lack of exercise and promote health.

[0116] "User interaction processing means" refers to a means for receiving and processing input data when a user answers a quiz or participates in a mini-game, and sending the results to a server.

[0117] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes and quest elements, transforming the use of stairs into an enjoyable experience, thereby addressing users' lack of exercise and promoting their health. The system as a whole is mainly divided into three elements: server, terminal, and user.

[0118] Server Role

[0119] The server acts as the controller for the entire system and performs the following main tasks:

[0120] Theme selection method:

[0121] The server's role is to select the theme for the following week from a theme library that is appropriate for the year and season. For example, it might select one theme from options such as "mountain climbing," "underwater," or "space."

[0122] Methods for preparing video data:

[0123] After a theme is selected, the server retrieves relevant video data from cloud storage based on the selected theme. It then generates video data suitable for each floor. For example, for a "mountain climbing" theme, it generates video from the "foot of the mountain" on the 1st floor to the "summit" on the 10th floor.

[0124] Video data distribution method:

[0125] The generated video data is distributed from the server to projection terminals on each floor. This makes it possible to project theme-based visual effects onto each staircase.

[0126] Quest element setting method:

[0127] The server is responsible for placing interactive elements such as quizzes and mini-games on specific staircases. For example, on the third floor, a quiz might be set up asking, "Which of the following mountains is over 1,000 meters high?"

[0128] Data acquisition methods:

[0129] The server collects user quiz answers and stair usage data from each terminal in real time. This allows for monitoring of user behavior.

[0130] Analysis means:

[0131] The server analyzes each user's exercise level and quiz accuracy rate based on the collected data. The analysis results are generated as reports to help address lack of exercise and promote health.

[0132] User interaction processing means:

[0133] The system receives input data from users when they answer quizzes or participate in mini-games, and sends the results to the server.

[0134] Terminal role

[0135] Terminals are installed on each floor and stairwell, and perform the following tasks:

[0136] Play video:

[0137] The terminal receives video data streamed from the server and transmits that data to a projector device to project the image onto the staircase. For example, the terminal on the first floor projects an image showing "the foot of the mountain."

[0138] Display of quizzes and mini-games:

[0139] The device receives quest content delivered from the server and displays quizzes or mini-games to users who reach specific locations on the stairs.

[0140] User interaction handling:

[0141] The terminal receives the user's quiz answers and mini-game participation data, and sends the results to the server.

[0142] User roles

[0143] The user performs the following tasks:

[0144] Use of stairs:

[0145] Users climb the stairs while enjoying visual effects created by projection mapping. By experiencing various themed landscapes, they can also get some exercise.

[0146] Participation in quizzes and mini-games:

[0147] When users reach a specific location on a floor, they can participate in quizzes or mini-games displayed by a projector. For example, on the third floor, they might answer a quiz asking, "Which of the following mountains is over 1,000 meters high?"

[0148] Specific examples and prompt statements

[0149] Specific example:

[0150] If the theme for the following week is set to "mountain climbing," the server prepares video data related to the "mountain climbing" theme and distributes it to terminals on each floor from the 1st to the 10th floor. The terminal on the 1st floor projects an image of "the foot of a mountain" onto the stairs, and as the user goes up from the 2nd floor, the scenery changes from "mountain path" → "valley" → "steep rocky area" → "snowy mountain." When the user reaches the 3rd floor, a quiz is displayed asking "Which of the following mountains has an altitude of over 1,000 meters?", and the user selects "Mount Fuji" from the options.

[0151] Examples of prompts for a generative AI model:

[0152] Please describe a system where projection mapping is used on the staircase from the 1st to the 10th floor, with the theme being "mountain climbing." Provide specific examples of scenery related to the theme of each floor, and detail how quizzes and mini-games are conducted on each floor.

[0153] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0154] Server processing steps

[0155] Step 1: Theme Selection

[0156] Input: Theme data from the previous week, year and season information, theme library

[0157] Processing: The server selects the theme for the following week from the theme library based on the year and season. Specifically, it scans the list of themes in the library and runs an algorithm that evaluates how well each theme fits the given conditions.

[0158] Output: Selected theme (e.g., "Mountain climbing", "Underwater", "Space")

[0159] Specific action: The server sets the theme for the following week to "mountain climbing".

[0160] Step 2: Prepare video data

[0161] Input: Selected theme (e.g., "Mountain climbing")

[0162] Processing: The server retrieves theme-related video data from cloud storage. Then, it generates video data appropriate for each floor. Specifically, for the "mountain climbing" theme, it runs an image processing algorithm to generate scenery data for floors 1 through 10.

[0163] Output: Video data corresponding to each floor (e.g., from the "foot of the mountain" on the 1st floor to the "summit" on the 10th floor)

[0164] Specific operation: The server generates video of the "foot of the mountain" for the first floor, and then generates video of the "mountain path" for the second floor.

[0165] Step 3: Video data distribution

[0166] Input: Generated video data

[0167] Processing: The server distributes video data to terminals on each floor. Specifically, it uses a protocol that transmits data to terminals via a network connection.

[0168] Output: Video data has been distributed to each terminal.

[0169] Specific operation: The server distributes the corresponding video data to each terminal from the 1st floor to the 10th floor.

[0170] Step 4: Setting Quest Elements

[0171] Input: Database of selected themes, quizzes, and mini-games

[0172] Processing: The server retrieves quizzes and mini-games related to the theme from the database and places them on specific floors. It then distributes them to projection terminals.

[0173] Output: Quest elements placed on each staircase

[0174] Specific operation: The server sets a quiz as a quest on the 3rd floor: "Which of the following mountains has an elevation of over 1,000 meters?" and delivers it to the terminal on the 3rd floor.

[0175] Step 5: Data Acquisition

[0176] Input: User quiz answer data, stair usage data

[0177] Processing: The server receives data collected from each terminal in real time. This includes the process of collecting data using network communication protocols.

[0178] Output: Collected data (e.g., user response data, usage data)

[0179] Specific operation: The server receives the user's quiz answer, "Mount Fuji," in real time from a terminal on the third floor.

[0180] Step 6: Data Analysis

[0181] Input: Collected data

[0182] Processing: The server analyzes each user's activity level and quiz accuracy rate. Specifically, it processes each user's data through a statistical analysis algorithm and generates a report.

[0183] Output: Report of analysis results

[0184] Specific operation: The server records and analyzes that user A walked 1,000 steps and achieved an 85% correct answer rate on the quiz.

[0185] Terminal processing steps

[0186] Step 1: Play the video

[0187] Input: Video data streamed from the server

[0188] Processing: The terminal verifies the received video data and sends it to the projector device. This includes a procedure to check that the data has been transferred correctly.

[0189] Output: Image projected onto the stairs

[0190] Specific operation: The terminal projects an image of "the foot of the mountain" onto the first floor.

[0191] Step 2: Display the quiz / mini-game

[0192] Input: Quest content delivered from the server

[0193] Processing: The terminal checks the received quest content and displays it when the user reaches a specific floor location.

[0194] Output: Quizzes and mini-games displayed on the stairs

[0195] Specific action: The terminal displays a quiz to the user upon reaching the 3rd floor: "Which of the following mountains has an elevation of over 1,000 meters?"

[0196] Step 3: Handling User Interactions

[0197] Input: User's quiz answers, mini-game results

[0198] Processing: The terminal receives user input data, processes it immediately, and sends the results to the server.

[0199] Output: User input data sent to the server

[0200] Specific operation: The terminal sends the result of the user selecting "Mt. Fuji" on the 3rd floor to the server.

[0201] User processing steps

[0202] Step 1: Use the stairs

[0203] Input: Projection-mapped stairs

[0204] Processing: Users enjoy the visual effects of the stairs as they climb, which can increase their physical activity.

[0205] Output: Exercise data of users using stairs

[0206] Specific action: The user climbs the stairs from the first floor to the second floor.

[0207] Step 2: Participate in quizzes and mini-games

[0208] Input: Quizzes and mini-games displayed on the stairs

[0209] Processing: Users participate in quizzes and mini-games displayed on specific floors, and perform actions such as answering questions.

[0210] Output: Quiz answers, mini-game results

[0211] Specific action: On the third floor, the user selects "Mount Fuji" in response to the quiz, "Which of the following mountains has an elevation of over 1,000 meters?"

[0212] (Application Example 1)

[0213] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0214] In modern society, opportunities for daily exercise have decreased due to the presence of office buildings and commercial facilities, leading to a serious problem of lack of physical activity. Furthermore, while fitness and health promotion initiatives continue, many people lack the motivation to maintain them. Traditional methods struggle to balance health promotion with enjoyment, and opportunities for exercise during travel, particularly when using food delivery services, are limited.

[0215] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0216] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, an analysis means, a virtual environment display means, an exercise path generation means, and an exercise data evaluation means. This allows users to enjoy healthy activities through projection mapping while waiting for food delivery or while delivery personnel are on their way. By participating in quizzes and mini-games at specific points, users are provided with an experience that increases their exercise level, evaluates their health status, and provides feedback.

[0217] A "theme selection method" refers to a device or program for selecting a weekly theme and preparing video data based on that theme.

[0218] "Video data preparation means" refers to devices or programs for generating and preparing video data related to a selected theme.

[0219] "Video data distribution means" refers to devices or programs for transmitting prepared video data to each terminal.

[0220] "Projection means" refers to devices or programs that use received video data to project images onto real-world objects or spaces.

[0221] A "quest element setting means" is a device or program for setting interactive elements such as quizzes and mini-games and placing them at specific points.

[0222] "Data collection means" refers to devices or programs used to collect user interaction and behavioral data.

[0223] "Analysis tools" refer to devices or programs that analyze collected data and evaluate the user's exercise level and health status.

[0224] A "virtual environment display means" refers to a device or program that uses a smartphone or smart glasses to display a virtual environment and enrich the user experience.

[0225] A "movement path generation means" is a device or program for setting a movement path for a user or delivery person and placing projection mapping or quizzes along that path.

[0226] "Exercise data evaluation means" refers to a device or program for evaluating a user's exercise data and providing feedback on the results.

[0227] This invention is a system that uses projection mapping technology to help users enjoy themselves while using food delivery services, thereby combating a sedentary lifestyle and promoting their health. The system consists of the following main components:

[0228] Server Role

[0229] The server acts as the central controller for the entire system. The server performs the following main tasks:

[0230] 1. Theme Selection Method: The server selects a different theme each week (e.g., space travel, underwater exploration, etc.).

[0231] 2. Video data preparation means: Based on the selected theme, the server generates and prepares the relevant video data.

[0232] 3. Video data distribution method: The prepared video data is sent to each terminal.

[0233] 4. Quest element setting method: Quizzes and mini-games are set on specific walking routes or staircases, and the data is distributed to the terminal.

[0234] 5. Data collection methods: User interaction data and behavioral data will be collected from each terminal.

[0235] 6. Analysis methods: The collected data is analyzed to evaluate the user's exercise level and health status.

[0236] 7. Virtual environment display means: A virtual environment is displayed on a smartphone, smart glasses, etc.

[0237] 8. Movement path generation means: Sets the movement path of the user or delivery person and places projection mapping along that path.

[0238] 9. Exercise data evaluation means: Evaluate the user's exercise data and provide feedback on the results.

[0239] Terminal role

[0240] The terminal will be installed on the smartphones or smart glasses of delivery personnel and users. The terminal will perform the following tasks:

[0241] 1. Virtual environment display means: Based on video data received from the server, projection mapping is performed onto real-world stairs and walking paths.

[0242] 2. Displaying quest elements: Display quizzes or mini-games at specific points to encourage user interaction.

[0243] 3. User interaction processing: Receive and process user quiz answers and mini-game results, and send them to the server.

[0244] User roles

[0245] Users can use this system to make good use of their waiting time for food delivery while enjoying fitness. The following tasks will be performed:

[0246] 1. Use of food delivery: Users order food delivery through the app.

[0247] 2. Projection Mapping Experience: While waiting for the delivery person to arrive, users can enjoy projection mapping while exercising.

[0248] 3. Participate in quizzes and mini-games: Participate in quizzes and mini-games at specific points and enjoy special effects depending on the results.

[0249] Specific example

[0250] For example, if a user launches a fitness app and orders a pizza, the delivery person's route is given a "space travel" theme. The user then jogs or walks while experiencing projection mapping using their smartphone or smart glasses. At certain points, a quiz appears asking, "How many times larger is Mars than Earth?" If the user answers correctly, a special effect occurs, boosting their motivation to exercise.

[0251] Example of a prompt

[0252] Please provide specific ideas for quizzes and mini-games that can be applied to health promotion applications using projection mapping. For example, quizzes and mini-games themed around "space travel" are possible.

[0253] This system is a new food delivery method that encourages users to exercise healthily and provides an enjoyable experience.

[0254] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0255] Step 1:

[0256] The server selects a theme each week using a theme selection mechanism. The input is index information including the start date of the week, and the output is the selected theme. Examples of themes include "space travel" and "underwater exploration." The server determines the theme based on the previous week's theme and user feedback.

[0257] Step 2:

[0258] The server uses a video data preparation mechanism to generate video data related to the selected theme. The input is the selected theme, and the output is video data corresponding to that theme. For example, if the theme is "space travel," video footage of outer space and planets will be generated. This video data will be delivered in a later step.

[0259] Step 3:

[0260] The server transmits the prepared video data to each terminal using a video data distribution method. The input is the video data, and the output is a delivery success message to each terminal. The video data is transmitted from the server to the terminals via the network.

[0261] Step 4:

[0262] The device projects received video data onto real-world stairs or walking paths using projection technology. The input is video data, and the output is the projected image. The user views this projected image through a smartphone or smart glasses.

[0263] Step 5:

[0264] The server uses a quest element setting mechanism to configure quizzes and mini-games and distributes the data to the terminal. The input is the configuration information, and the output is a message indicating successful distribution to the terminal. For example, if the theme is "Space Travel," a quiz such as "How many times larger is Mars than Earth?" will be set.

[0265] Step 6:

[0266] The device displays received quizzes or mini-games to the user when they reach a specific point. The input is the received quiz data, and the output is the quiz screen. When the user answers the quiz, the result is sent to the device.

[0267] Step 7:

[0268] The terminal sends the user's answer to the server. The input is the user's answer data, and the output is a message indicating successful transmission to the server. The answer data is transmitted from the terminal to the server via the network.

[0269] Step 8:

[0270] The server collects data transmitted from each terminal using data collection methods. The input consists of user response data and behavioral data, while the output is the collected data. The server collects user response data and analyzes it in the next step.

[0271] Step 9:

[0272] The server analyzes the collected data using analytical tools to evaluate the user's exercise level and health status. The input is the collected data, and the output is the analysis results. Based on the analysis results, feedback on health status and exercise advice are generated.

[0273] Step 10:

[0274] The server uses an exercise data evaluation system to provide the user with analysis results. The input is the analysis results, and the output is a feedback message. The feedback message is notified to the user through the app, providing information that can help improve their health.

[0275] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0276] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes, quest elements, and an emotion engine that recognizes the user's emotions, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[0277] Server Role

[0278] The server acts as the controller for the entire system and performs the following main tasks:

[0279] 1. Theme selection:

[0280] The server selects different themes every week (such as mountain climbing, underwater, space, etc.) and prepares video data.

[0281] 2. Video Data Preparation and Distribution:

[0282] Based on the selected theme, the server prepares video data and distributes it to the projection terminals installed on each floor.

[0283] 3. Setting of Quest Elements:

[0284] The server sets quizzes and mini - games on specific stairs and distributes them to the terminals.

[0285] 4. Data Collection and Analysis:

[0286] The server receives the emotion data transmitted from the emotion engine and analyzes it in combination with the user's exercise volume and health status.

[0287] Role of the Terminal

[0288] The terminals are installed on each floor and on the stairs, and execute the following tasks based on the data received from the server:

[0289] 1. Video Playback:

[0290] The terminal transmits the received video data to the projector device and performs projection mapping on the stairs.

[0291] 2. Display of Quizzes and Mini - games:

[0292] On specific stairs, the terminal displays quizzes and mini - games so that users can participate.

[0293] 3. User Emotion Recognition:

[0294] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine. The emotion engine analyzes this data to recognize the user's emotional state.

[0295] 4. Interaction handling:

[0296] The device provides feedback based on the user's emotional state and quiz answers, and adjusts the projected images and sound effects accordingly.

[0297] User roles

[0298] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[0299] 1. Use of stairs:

[0300] Users climb the stairs while enjoying visual effects created by projection mapping.

[0301] 2. Participation in quizzes and mini-games:

[0302] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[0303] 3. Emotion Recognition Interaction:

[0304] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system. For example, if the user is feeling happy, the projected image becomes more vivid, enhancing the entertainment value.

[0305] Specific example

[0306] For example, if the theme for the following week is set to "Space Exploration," the server will prepare video data related to the "Space Exploration" theme and distribute it to terminals on each floor from the first to the tenth floor.

[0307] The terminal on the first floor projects an image of a "spaceship" onto the staircase, and as one goes up from the second floor, the scenery changes from "starry sky" → "planet" → "galaxy". When the user reaches the third floor, a quiz "What is the distance from the Earth to the Moon?" is displayed. The user selects "about 380,000 kilometers" from the options, and if the answer is correct, a special effect (e.g., star effect) is displayed on the staircase to the next floor.

[0308] Furthermore, since an emotion engine is incorporated, it recognizes emotions from the user's expressions and voices. For example, when the user shows a surprised expression, the terminal senses it and transmits emotion data to the server. Based on this data, the server adjusts the next projection video to provide an entertainment experience suitable for the user's emotion.

[0309] The server collects data from the terminals on each floor and evaluates the momentum and health status by analyzing the user's answer data, emotion data, and staircase usage data. This data is utilized as a user's health improvement program and as feedback for the entire company.

[0310] [[ID=1X]] As described above, this invention provides a staircase utilization promotion system that also takes into account the user's emotions, eliminates lack of exercise, promotes health, and enhances entertainment at the same time.

[0311] The following describes the processing flow.

[0312] Step 1:

[0313] Theme selection (server)

[0314] The server checks the current week and selects a new theme. For example, the theme for next week is set to "space exploration".

[0315] Step 2:

[0316] X Video data preparation (server)

[0317] The server collects and prepares relevant video data based on the selected theme. For example, it prepares a video dataset for space exploration (spaceships, starry skies, planets, etc.).

[0318] Step 3:

[0319] Video data distribution (server)

[0320] The server sequentially distributes the prepared video data to the projection terminals on each floor. For example, it sends video data from the first floor to the tenth floor.

[0321] Step 4:

[0322] Receiving and saving video data (on the device)

[0323] The device receives video data transmitted from the server and saves it to local storage. Once saving is complete, it begins preparing for playback.

[0324] Step 5:

[0325] Projection Start (Device)

[0326] The device activates the projector at a specified time and projects images onto the walls and floor of the staircase. For example, an image of a spaceship is displayed on the staircase on the first floor.

[0327] Step 6:

[0328] Quest element settings (server)

[0329] The server prepares quiz or mini-game data for specific staircases and sends it to the terminals. For example, the 3rd floor might have a quiz like, "What is the distance from the Earth to the Moon?"

[0330] Step 7:

[0331] Quest data reception and display preparation (device)

[0332] The terminal receives data for quizzes and mini-games and prepares to display it when the user reaches the desired location. It verifies the validity of the data and prepares the interactive screen.

[0333] Step 8:

[0334] Emotional recognition (emotional engine)

[0335] Sensors such as cameras and microphones installed on the device capture the user's facial expressions, voice, and body movements, and transmit this data to the emotion engine. The emotion engine analyzes this data to recognize the user's emotions.

[0336] Step 9:

[0337] User interaction (user)

[0338] The user climbs the stairs while enjoying projection mapping. A quiz is displayed when they reach a specific floor (for example, the 3rd floor). The user answers the quiz. Additionally, an emotion recognition system analyzes the user's emotional state.

[0339] Step 10:

[0340] Receiving and evaluating responses (device)

[0341] The device receives the user's answer and determines whether it is correct or incorrect. If correct, a special effect is added to the video leading up to the next floor. For example, a special star effect is displayed when the answer is correct. At the same time, it also receives emotional data from the emotion engine.

[0342] Step 11:

[0343] Emotion-based feedback (device)

[0344] The device adjusts projected images and sound effects based on emotional data obtained from the emotion engine. For example, if it recognizes that the user is enjoying themselves, it makes the images more vibrant and the music more upbeat.

[0345] Step 12:

[0346] Data collection (device)

[0347] The device collects user response data, sentiment data, and stair usage data. It then prepares the collected data for transmission to the server.

[0348] Step 13:

[0349] Data transmission (device)

[0350] The device sends the collected data to the server. This data may include each user's quiz answers, emotional state, and stair climbing time.

[0351] Step 14:

[0352] Data reception and analysis (server)

[0353] The server receives data sent from the terminal and stores it in a database. It then begins analysis to assess exercise levels and health status.

[0354] Step 15:

[0355] Feedback of results (server)

[0356] The server generates a report based on the analysis results and notifies administrators and relevant parties. It reports on each user's health status, activity levels, and emotional fluctuations.

[0357] Through the steps outlined above, the system aims to encourage users to use stairs in an enjoyable way, alleviate lack of exercise, and promote health. Furthermore, it can enhance the entertainment value by providing interactions that respond to the user's emotional state.

[0358] (Example 2)

[0359] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0360] The problems of lack of exercise and poor health among modern people are serious, and it has been pointed out that stairs are not used very often, especially in office buildings and commercial facilities. Furthermore, because using stairs is monotonous, many people tend to use elevators. As a result, physical activity decreases, which has a negative impact on health. This invention aims to alleviate lack of exercise and promote health by transforming the use of stairs into an entertaining experience, thereby encouraging users to use stairs voluntarily.

[0361] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0362] In this invention, the server includes a theme selection means for automatically selecting a weekly theme, a video data preparation means for preparing video data based on the theme, a video data distribution means for distributing video data from the server to terminals installed on each floor, a projection means for projecting images onto the stairs using a projector, a quest element setting means for setting quizzes or mini-games on specific staircases, a data collection means for capturing the user's facial expressions and voice using a camera and microphone, an analysis means for analyzing the collected data, and an emotion adjustment means for adjusting the projected image based on the user's emotion data. This makes it possible to transform the use of stairs into an entertaining experience, alleviate users' lack of exercise, and promote their health.

[0363] A "theme selection means" is a device or program for automatically selecting a different theme each week.

[0364] "Video data preparation means" refers to a device or program for producing or preparing video data based on a selected theme.

[0365] "Video data distribution means" refers to a device or program for distributing prepared video data from a server to a terminal.

[0366] "Projection means" refers to a device or program for projecting received video data onto a staircase using a projector device.

[0367] A "quest element setting means" is a device or program for setting quizzes or mini-games on specific staircases and providing them to the user.

[0368] "Data collection means" refers to a device or program that uses a camera and microphone to capture the user's facial expressions and voice, and collect that data.

[0369] "Analysis means" refers to a device or program that analyzes collected emotional data, motor data, response data, etc., to evaluate the user's state and behavior.

[0370] An "emotion adjustment device" is a device or program that adjusts projected images and sound effects based on the user's emotional data to optimize the entertainment experience.

[0371] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes, quest elements, and an emotion engine that recognizes the user's emotions, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[0372] Server Role

[0373] The server acts as the controller for the entire system and performs the following main tasks:

[0374] Theme Selection Methods

[0375] The server automatically selects a different theme each week. For example, it uses a generation AI model to select themes such as "space exploration" and "seabed exploration."

[0376] Video data preparation means

[0377] The server prepares video data based on the selected theme. This video data is generated to be suitable for the staircases on each floor. Specifically, it is prepared using video editing software and CG software.

[0378] Video data distribution method

[0379] The server distributes the prepared video data to terminals installed on each floor using the HTTP protocol.

[0380] Quest element setting method

[0381] The server sets up quizzes and mini-games on specific staircases and distributes the data to the terminals. The set quest elements encourage user interaction when using the stairs.

[0382] Data acquisition methods

[0383] The server acquires user facial expressions and voice data collected using cameras and microphones. This allows it to understand the user's emotional state.

[0384] analytical means

[0385] The server analyzes emotional data transmitted from the emotion engine and evaluates it in conjunction with the user's activity level and health status. This evaluation result is used as feedback data for the entire company.

[0386] emotional regulation means

[0387] The server adjusts the projected image based on the user's emotional data. For example, if the user is feeling happy, it adjusts the color and brightness of the image to enhance its entertainment value.

[0388] Terminal role

[0389] Terminals are installed on each floor and stairwell, and perform the following tasks based on data received from the server:

[0390] Video playback

[0391] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs. For example, it connects to the projector using an HDMI cable.

[0392] Display of quizzes and mini-games

[0393] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate. A special effect is added when the user answers correctly.

[0394] User emotion recognition

[0395] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine.

[0396] Interaction Processing

[0397] The device provides feedback based on the user's emotional state and quiz answers, and adjusts the projected images and sound effects accordingly.

[0398] User roles

[0399] Users can use this system to enjoy climbing stairs:

[0400] Use of stairs

[0401] Users climb the stairs while enjoying visual effects created by projection mapping.

[0402] Participation in quizzes and mini-games

[0403] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[0404] Emotion Recognition Interaction

[0405] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system.

[0406] Specific example

[0407] For example, if the theme for the following week is set to "Space Exploration," the server will prepare video data related to the "Space Exploration" theme and distribute it to terminals on each floor from the first to the tenth floor.

[0408] On the first floor, a terminal projects an image of a "spaceship" onto the staircase, and as the user ascends from the second floor, the scenery changes from a "starry sky" to "planets" to a "galaxy." Upon reaching the third floor, the user is presented with a quiz: "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if correct, a special effect (such as a star effect) is displayed on the staircase leading to the next floor.

[0409] Furthermore, because it incorporates an emotion engine, it recognizes emotions from the user's facial expressions and voice. For example, if the user shows a surprised expression, the device detects this and sends emotion data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[0410] Example of a prompt

[0411] This week's theme is "Space Exploration." Project different images on each floor as the user climbs the stairs from the 1st to the 10th floor. Display an image of a "spaceship" on the 1st floor, a "starry sky" on the 2nd floor, and a "planet" on the 3rd floor. Also, on the 3rd floor, present a quiz: "What is the distance from Earth to the Moon?" If the user answers correctly, display a special star effect on the stairs to the next floor. Capture the user's facial expressions and voice using the camera and microphone, and if an expression of surprise is detected, send that data to the server.

[0412] Through the above steps, the system takes user emotions into consideration, addresses lack of exercise, and enhances entertainment value.

[0413] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0414] Step 1: Theme Selection

[0415] The server uses a generative AI model to select a weekly theme. To do this, the server receives various data (user interests, seasonal events, etc.) as input and selects the theme for the following week (e.g., space exploration) as output.

[0416] Step 2: Preparing the video data

[0417] Based on the selected theme, the server prepares video data using video editing software and CG software. In this step, the server receives a theme (e.g., space exploration) as input and generates corresponding video data (e.g., spaceship, starry sky, planets) as output.

[0418] Step 3: Distribution of video data

[0419] The server delivers the prepared video data to the terminals using the HTTP protocol. In this step, the server receives video data (e.g., spaceship, starry sky, planets) as input and outputs a distribution request to send it to the terminals on each floor.

[0420] Step 4: Setting up quest elements

[0421] The server sets up quizzes and mini-games on specific staircases and distributes the data to the terminals. In this step, the server receives quest elements (e.g., What is the distance from Earth to the Moon?) as input and outputs a distribution request to send them to the target terminals.

[0422] Step 5: Play the video

[0423] The terminal sends the received video data to the projector device to perform projection mapping on the stairs. In this step, the terminal receives video data (e.g., spaceship, starry sky, planets) from the server as input and outputs a video transfer request to send to the projector.

[0424] Step 6: Displaying quizzes and mini-games

[0425] At specific points along the staircase, the terminal displays quizzes or mini-games, allowing the user to participate. In this step, the terminal receives quiz data from the server (e.g., "What is the distance from the Earth to the Moon?") as input and sends a display request to output it to the display. It also receives the user's answer and feeds it back into the next action.

[0426] Step 7: User emotion recognition

[0427] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and sends them to the emotion engine. In this step, the device receives data from sensors (e.g., facial expressions, voice) as input, captures it as emotion data, and outputs a data transfer request to send it to the emotion engine.

[0428] Step 8: Interaction Processing

[0429] The device provides feedback based on the user's emotional state and quiz results, adjusting the projected images and sound effects accordingly. In this step, the device receives emotional data and quiz results as input and outputs requests to adjust the images and sounds based on them. Specifically, if the user is feeling happy, the device adjusts the colors and brightness of the images.

[0430] Step 9: Data Collection and Analysis

[0431] The server analyzes emotional data sent from the emotion engine along with the user's exercise data and response data. In this step, the server receives emotional data, exercise data, and response data as input and outputs the analysis results. Specifically, this is used to evaluate the user's health status and entertainment experience.

[0432] Because each step involves a detailed description of specific actions, this system's program transforms using stairs into an entertaining experience, providing a means to alleviate users' lack of exercise and promote their health.

[0433] (Application Example 2)

[0434] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0435] This project aims to address the lack of exercise among stair users, promote their health, and make stair use enjoyable by enhancing its entertainment value. However, conventional methods have limited effectiveness in motivating stair users and improving the user experience, necessitating new technological solutions. Furthermore, technologies for appropriate emotion recognition and feedback based on those results are still underdeveloped. A system is needed to address these challenges.

[0436] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0437] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, an analysis means, a user recognition means, an emotion recognition means, and an emotion-based feedback means. This makes it possible to alleviate the user's lack of exercise, improve their health, and provide a dynamic entertainment experience based on emotions.

[0438] The "theme selection method" is a function that selects a different theme for each specified period and prepares the necessary video data and public relations information based on that theme.

[0439] The "video data preparation means" is a function that creates and edits video data to be used for projection mapping based on the selected theme.

[0440] "Video data distribution means" refers to a function that transmits prepared video data to a projection means.

[0441] "Projection means" refers to a function that displays received video data on stairs or walls using projection mapping technology.

[0442] The "quest element setting method" is a function that allows users to set up quizzes or mini-games at specific locations on stairs or floors, enabling them to participate in them.

[0443] "Data collection methods" refer to functions that collect user behavior data, quest participation data, and emotional data.

[0444] "Analysis tools" refer to functions that analyze collected data to evaluate the user's health status, exercise level, emotional state, and other factors.

[0445] "User recognition means" refers to a function that recognizes the user's face and body movements to identify the individual.

[0446] "Emotion recognition means" refers to a function that analyzes the user's facial expressions and voice to determine their emotional state.

[0447] "Emotion-based feedback mechanisms" are functions that adjust projection images and sound effects based on recognized emotions to provide appropriate feedback.

[0448] This invention is designed as a system that enhances the entertainment value of using stairs, alleviates lack of exercise, and promotes health. Specific embodiments using each component of this system are described below.

[0449] Server Role

[0450] The server acts as the controller for the entire system, performing the following key tasks.

[0451] 1. Theme Selection Methods

[0452] The server selects a different theme for each specified period. For example, themes might include "space exploration" or "underwater exploration," and based on these, it prepares the necessary video data and quizzes.

[0453] 2. Means for preparing video data

[0454] The server creates and edits video data to be used for projection mapping based on the selected theme. The video data is prepared to correspond to each floor of the staircase. For example, if the theme is "Space Exploration," the first floor will be set to "Earth," the second floor to "Moon," and the third floor to "Mars."

[0455] 3. Video data distribution method

[0456] The server transmits the prepared video data to projection terminals installed on each floor. The video data is distributed in real time via the network.

[0457] 4. Quest element setting means

[0458] The server sets up quizzes and mini-games at specific stairwells or floor locations and delivers them to projection terminals. For example, a quiz such as "What is the distance from the Earth to the Moon?" might be displayed at the third floor, and users can answer it using their smartphones.

[0459] 5. Data Collection Methods

[0460] The server collects user behavior data, quiz participation data, and emotional data. This allows the server to understand user activity levels and quest participation status.

[0461] 6. Analysis tools

[0462] The server analyzes the collected data to evaluate the user's health, activity level, emotional state, and other factors. This helps in providing more effective feedback and selecting future topics.

[0463] Terminal role

[0464] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server.

[0465] 1. Play back video

[0466] The terminal transmits the received video data to the projector device, which then performs projection mapping on the stairs.

[0467] 2. Display of quizzes and mini-games

[0468] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[0469] 3. User Recognition Means

[0470] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and recognizes the user.

[0471] 4. Emotion recognition means

[0472] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine. The emotion engine analyzes this data to recognize the user's emotional state.

[0473] 5. Emotion-based feedback methods

[0474] The device provides feedback based on the user's emotional state and quiz answers, adjusting the projected images and sound effects accordingly. For example, if the user is enjoying themselves, the images become more vibrant and the sound effects are enhanced.

[0475] User roles

[0476] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[0477] 1. Use of stairs

[0478] Users climb the stairs while enjoying visual effects created by projection mapping.

[0479] 2. Participation in quizzes and mini-games

[0480] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[0481] 3. Emotion Recognition Interaction

[0482] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system. For example, if the user shows a surprised expression, the device detects this and sends emotional data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[0483] Hardware and software to be used

[0484] Hardware: Smartphones (e.g., iPhone®, Google Pixel®), projector devices, sensor terminals (cameras, microphones, etc.)

[0485] Software: EmotionRecognition library (for emotion recognition), OpenCV library (for camera input and video processing), server-side web frameworks such as Flask and Django.

[0486] Specific example

[0487] For example, if the theme is set to "Space Exploration," the server prepares video data related to the "Space Exploration" theme and distributes it to terminals on each floor from the 1st to the 10th floor. The terminal on the 1st floor projects an image of a "spaceship" onto the staircase, and as the user goes up from the 2nd floor, the scenery changes from a "starry sky" to "planets" to "galaxies." When the user reaches the 3rd floor, a quiz is displayed asking, "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if they answer correctly, a special effect (for example, a star effect) is displayed on the staircase to the next floor.

[0488] Example of a prompt

[0489] Example of quiz generation: "To the generation AI model: Please create 3 quiz questions related to space exploration."

[0490] Example of emotion data analysis: "To the generative AI model: Please suggest suitable entertainment content when the emotional state is happy."

[0491] The above describes specific embodiments for carrying out this invention.

[0492] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0493] Step 1:

[0494] The server selects the theme.

[0495] The server selects a different theme for each specified period, prompting the AI ​​model with the message, "Please suggest a theme for next week," and then receives the generated theme. This theme information is used for the following processes.

[0496] Input: Theme selection prompt message

[0497] Data processing / manipulation: Themes are generated using a generative AI model.

[0498] Output: Selected theme (e.g., "Space Exploration")

[0499] Step 2:

[0500] The server prepares the video data.

[0501] The server creates video data for projection mapping using video data preparation means, based on the selected theme. It collects the necessary video materials and edits them to match the theme.

[0502] Input: Selected theme

[0503] Data processing / editing: Creation and editing of video data

[0504] Output: Prepared video data

[0505] Step 3:

[0506] The server distributes the video data.

[0507] The server distributes the prepared video data to terminals installed on each floor via the network.

[0508] Input: Prepared video data

[0509] Data processing / processing: Video data distribution

[0510] Output: Supply of video data to the terminal

[0511] Step 4:

[0512] The device plays the video.

[0513] The terminal transmits the received video data to a projector device, performing projection mapping on designated staircases or walls. The video is displayed according to the selected theme.

[0514] Input: Video data from the server

[0515] Data processing / editing: Video display using projection mapping

[0516] Output: Video displays on stairs and walls

[0517] Step 5:

[0518] The server sets the quest elements.

[0519] The server requests a generation AI model to create quizzes and mini-games related to the theme. For example, it might input a prompt like, "Create three quiz questions related to space exploration," and then receive the generated quest content.

[0520] Input: Quest generation prompt text

[0521] Data processing / manipulation: Generating quest elements using a generative AI model.

[0522] Output: Generated quest content (e.g., "What is the distance from Earth to the Moon?")

[0523] Step 6:

[0524] The server will deliver quest elements.

[0525] The server distributes the generated quest details to terminals installed on specific staircases.

[0526] Input: Generated quest content

[0527] Data processing / editing: Distribution of quest content

[0528] Output: Supplying quest content to the terminal

[0529] Step 7:

[0530] The device displays a quiz / mini-game.

[0531] The terminal displays received quest content on a projector device or terminal screen, allowing users to participate. Users answer quizzes and mini-games in real time.

[0532] Input: Quest details from the server

[0533] Data processing / manipulation: Displaying quizzes and mini-games

[0534] Output: User participation in quizzes and mini-games

[0535] Step 8:

[0536] The device recognizes the user's emotions.

[0537] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and analyzes them using emotion recognition technology.

[0538] Input: User's facial expressions, voice, and motion data

[0539] Data processing / analysis: Recognition and analysis of emotional states

[0540] Output: Recognized emotion data

[0541] Step 9:

[0542] The device sends emotional data to the server.

[0543] The device sends the recognized emotion data to the server. The server receives this data and uses it for the next processing step.

[0544] Input: Recognized emotion data

[0545] Data processing / processing: Sending emotional data

[0546] Output: Supplying emotion data to the server

[0547] Step 10:

[0548] The server provides emotion-based feedback.

[0549] The server analyzes the received emotional data and adjusts the next projection images and sound effects based on the emotions. It inputs a prompt message to the generating AI model, "Please suggest suitable entertainment content when the emotional state is happy," and sends the generated content to the terminal.

[0550] Input: Sentiment data, entertainment content suggestion prompt text

[0551] Data Processing / Editing: Analysis and generation of emotional data, content proposals using AI models.

[0552] Output: Adjusted entertainment content

[0553] Step 11:

[0554] The device displays the feedback content.

[0555] The device plays back the adjusted projection images and sound effects received from the server and provides feedback to the user. This is designed to create an entertainment experience that responds to the user's emotional state.

[0556] Input: Coordinated entertainment content from the server

[0557] Data processing / editing: Playback of projection images and sound effects.

[0558] Output: Providing feedback to users

[0559] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0560] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0561] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0562] [Second Embodiment]

[0563] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0564] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0565] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0566] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0567] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0568] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0569] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0570] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0571] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0572] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0573] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0574] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0575] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes and quest elements, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[0576] Server Role

[0577] The server acts as the controller for the entire system and performs the following main tasks:

[0578] 1. Theme selection:

[0579] The server selects a different theme each week (for example, mountain climbing, underwater, space, etc.) and prepares video data for it.

[0580] 2. Preparation and distribution of video data:

[0581] Based on the selected theme, the server prepares the video data and distributes it to the projection terminals installed on each floor.

[0582] 3. Setting Quest Elements:

[0583] The server sets up quizzes and mini-games on specific staircases and delivers them to the terminals.

[0584] 4. Data collection and analysis:

[0585] The server receives data collected from each terminal and analyzes the user's activity level and health status.

[0586] Terminal role

[0587] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server:

[0588] 1. Play back the video:

[0589] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs.

[0590] 2. Display of quizzes and mini-games:

[0591] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[0592] 3. User interaction handling:

[0593] The terminal receives and processes the user's quiz answers and mini-game results, and sends that data to the server.

[0594] User roles

[0595] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[0596] 1. Use of stairs:

[0597] Users climb the stairs while enjoying visual effects created by projection mapping.

[0598] 2. Participation in quizzes and mini-games:

[0599] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[0600] Specific example

[0601] For example, if the theme for the following week is set to "mountain climbing," the server will prepare video data related to the "mountain climbing" theme and distribute it to terminals on each floor from the 1st to the 10th floor.

[0602] On the first floor, a terminal projects an image of "the foot of a mountain" onto the staircase, and as you go up from the second floor, the scenery changes from "mountain path" to "valley" to "steep rocky area" to "snowy mountain." When a user reaches the third floor, a quiz is displayed asking, "Which of the following mountains is over 1,000 meters high?" If the user selects "Mount Fuji" from the options and answers correctly, a special effect (such as fireworks or special visual effects) is displayed on the staircase to the next floor.

[0603] The server collects data from terminals on each floor and analyzes user response data and stair usage data to evaluate exercise levels and health status. This data is used for user health improvement programs and as feedback for the entire company.

[0604] The following describes the processing flow.

[0605] Step 1:

[0606] Theme selection (server)

[0607] The server checks the current week and selects a new theme. For example, the theme for next week might be set to "Space Exploration".

[0608] Step 2:

[0609] Preparation of video data (server)

[0610] The server collects and prepares relevant video data based on the selected theme. For example, it prepares a video dataset for space exploration (spaceships, starry skies, planets, etc.).

[0611] Step 3:

[0612] Video data distribution (server)

[0613] The server sequentially distributes the prepared video data to the projection terminals on each floor. For example, it sends video data from the first floor to the tenth floor.

[0614] Step 4:

[0615] Receiving and saving video data (on the device)

[0616] The device receives video data transmitted from the server and saves it to local storage. Once saving is complete, it begins preparing for playback.

[0617] Step 5:

[0618] Projection Start (Device)

[0619] The device activates the projector at a specified time and projects images onto the walls and floor of the staircase. For example, an image of a spaceship is displayed on the staircase on the first floor.

[0620] Step 6:

[0621] Quest element settings (server)

[0622] The server prepares quiz or mini-game data for specific staircases and sends it to the terminals. For example, the 3rd floor might have a quiz like, "What is the distance from the Earth to the Moon?"

[0623] Step 7:

[0624] Quest data reception and display preparation (device)

[0625] The terminal receives data for quizzes and mini-games and prepares to display it when the user reaches the desired location. It verifies the validity of the data and prepares the interactive screen.

[0626] Step 8:

[0627] User interaction (user)

[0628] The user climbs the stairs while enjoying projection mapping. A quiz is displayed when they reach a specific floor (for example, the 3rd floor). The user then answers the quiz.

[0629] Step 9:

[0630] Receiving and evaluating responses (device)

[0631] The device receives the user's answer and determines whether it is correct or incorrect. If correct, a special effect is added to the video leading up to the next floor. For example, a special star effect is displayed when the answer is correct.

[0632] Step 10:

[0633] Data collection (device)

[0634] The terminal collects user response data and stair usage data. The collected data is then prepared for transmission to the server.

[0635] Step 11:

[0636] Data transmission (device)

[0637] The device sends the collected data to the server. This data may include, for example, each user's quiz answers and stair usage time.

[0638] Step 12:

[0639] Data reception and analysis (server)

[0640] The server receives data sent from the terminal and stores it in a database. It then begins analysis to evaluate exercise levels and health status.

[0641] Step 13:

[0642] Feedback of results (server)

[0643] The server generates a report based on the analysis results and notifies administrators and relevant parties. It reports on each user's health status and activity levels over time.

[0644] Through the steps described above, the system aims to make using stairs enjoyable for users, alleviate a lack of exercise, and promote health.

[0645] (Example 1)

[0646] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0647] In modern office buildings and commercial facilities, users' reliance on elevators leads to insufficient daily exercise and a decline in health. Furthermore, using stairs is psychologically monotonous, causing users to avoid it. Additionally, there is a lack of systems to efficiently monitor and improve health conditions and exercise levels.

[0648] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0649] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a quest element setting means, a data collection means, an analysis means, and a user interaction processing means. This makes using stairs enjoyable, helps alleviate lack of exercise, and promotes health. Furthermore, it provides psychological enjoyment by offering quizzes and mini-games, and enables efficient collection and analysis of each user's exercise data.

[0650] "Theme selection method" refers to the process of selecting a theme from the library that is appropriate for the year or season, and deciding on it as the theme for the following week.

[0651] "Video data preparation means" refers to a means for generating relevant video data based on a selected theme and preparing it for projection on each floor.

[0652] "Video data distribution means" refers to a means for distributing generated video data to terminals on each floor.

[0653] "Projection means" refers to a method for transmitting video data received by a terminal to a projector device and projecting the image onto the stairs.

[0654] A "quest element setting method" is a means of placing interactive elements such as quizzes or mini-games on a specific staircase and presenting them to the user.

[0655] "Data collection means" refers to the means of collecting user interaction and stair usage data from each terminal.

[0656] "Analysis methods" refer to methods for analyzing each user's exercise volume and quiz accuracy rate based on collected data, and generating reports to address lack of exercise and promote health.

[0657] "User interaction processing means" refers to a means for receiving and processing input data when a user answers a quiz or participates in a mini-game, and sending the results to a server.

[0658] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes and quest elements, transforming the use of stairs into an enjoyable experience, thereby addressing users' lack of exercise and promoting their health. The system as a whole is mainly divided into three elements: server, terminal, and user.

[0659] Server Role

[0660] The server acts as the controller for the entire system and performs the following main tasks:

[0661] Theme selection method:

[0662] The server's role is to select the theme for the following week from a theme library that is appropriate for the year and season. For example, it might select one theme from options such as "mountain climbing," "underwater," or "space."

[0663] Methods for preparing video data:

[0664] After a theme is selected, the server retrieves relevant video data from cloud storage based on the selected theme. It then generates video data suitable for each floor. For example, for a "mountain climbing" theme, it generates video from the "foot of the mountain" on the 1st floor to the "summit" on the 10th floor.

[0665] Video data distribution method:

[0666] The generated video data is distributed from the server to projection terminals on each floor. This makes it possible to project theme-based visual effects onto each staircase.

[0667] Quest element setting method:

[0668] The server is responsible for placing interactive elements such as quizzes and mini-games on specific staircases. For example, on the third floor, a quiz might be set up asking, "Which of the following mountains is over 1,000 meters high?"

[0669] Data acquisition methods:

[0670] The server collects user quiz answers and stair usage data from each terminal in real time. This allows for monitoring of user behavior.

[0671] Analysis means:

[0672] The server analyzes each user's exercise level and quiz accuracy rate based on the collected data. The analysis results are generated as reports to help address lack of exercise and promote health.

[0673] User interaction processing means:

[0674] The system receives input data from users when they answer quizzes or participate in mini-games, and sends the results to the server.

[0675] Terminal role

[0676] Terminals are installed on each floor and stairwell, and perform the following tasks:

[0677] Play video:

[0678] The terminal receives video data streamed from the server and transmits that data to a projector device to project the image onto the staircase. For example, the terminal on the first floor projects an image showing "the foot of the mountain."

[0679] Display of quizzes and mini-games:

[0680] The device receives quest content delivered from the server and displays quizzes or mini-games to users who reach specific locations on the stairs.

[0681] User interaction handling:

[0682] The terminal receives the user's quiz answers and mini-game participation data, and sends the results to the server.

[0683] User roles

[0684] The user performs the following tasks:

[0685] Use of stairs:

[0686] Users climb the stairs while enjoying visual effects created by projection mapping. By experiencing various themed landscapes, they can also get some exercise.

[0687] Participation in quizzes and mini-games:

[0688] When users reach a specific location on a floor, they can participate in quizzes or mini-games displayed by a projector. For example, on the third floor, they might answer a quiz asking, "Which of the following mountains is over 1,000 meters high?"

[0689] Specific examples and prompt statements

[0690] Specific example:

[0691] If the theme for the following week is set to "mountain climbing," the server prepares video data related to the "mountain climbing" theme and distributes it to terminals on each floor from the 1st to the 10th floor. The terminal on the 1st floor projects an image of "the foot of a mountain" onto the stairs, and as the user goes up from the 2nd floor, the scenery changes from "mountain path" → "valley" → "steep rocky area" → "snowy mountain." When the user reaches the 3rd floor, a quiz is displayed asking "Which of the following mountains has an altitude of over 1,000 meters?", and the user selects "Mount Fuji" from the options.

[0692] Examples of prompts for a generative AI model:

[0693] Please describe a system where projection mapping is used on the staircase from the 1st to the 10th floor, with the theme being "mountain climbing." Provide specific examples of scenery related to the theme of each floor, and detail how quizzes and mini-games are conducted on each floor.

[0694] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0695] Server processing steps

[0696] Step 1: Theme Selection

[0697] Input: Theme data from the previous week, year and season information, theme library

[0698] Processing: The server selects the theme for the following week from the theme library based on the year and season. Specifically, it scans the list of themes in the library and runs an algorithm that evaluates how well each theme fits the given conditions.

[0699] Output: Selected theme (e.g., "Mountain climbing", "Underwater", "Space")

[0700] Specific action: The server sets the theme for the following week to "mountain climbing".

[0701] Step 2: Prepare video data

[0702] Input: Selected theme (e.g., "Mountain climbing")

[0703] Processing: The server retrieves theme-related video data from cloud storage. Then, it generates video data appropriate for each floor. Specifically, for the "mountain climbing" theme, it runs an image processing algorithm to generate scenery data for floors 1 through 10.

[0704] Output: Video data corresponding to each floor (e.g., from the "foot of the mountain" on the 1st floor to the "summit" on the 10th floor)

[0705] Specific operation: The server generates video of the "foot of the mountain" for the first floor, and then generates video of the "mountain path" for the second floor.

[0706] Step 3: Video data distribution

[0707] Input: Generated video data

[0708] Processing: The server distributes video data to terminals on each floor. Specifically, it uses a protocol that transmits data to terminals via a network connection.

[0709] Output: Video data has been distributed to each terminal.

[0710] Specific operation: The server distributes the corresponding video data to each terminal from the 1st floor to the 10th floor.

[0711] Step 4: Setting Quest Elements

[0712] Input: Database of selected themes, quizzes, and mini-games

[0713] Processing: The server retrieves quizzes and mini-games related to the theme from the database and places them on specific floors. It then distributes them to projection terminals.

[0714] Output: Quest elements placed on each staircase

[0715] Specific operation: The server sets a quiz as a quest on the 3rd floor: "Which of the following mountains has an elevation of over 1,000 meters?" and delivers it to the terminal on the 3rd floor.

[0716] Step 5: Data Acquisition

[0717] Input: User quiz answer data, stair usage data

[0718] Processing: The server receives data collected from each terminal in real time. This includes the process of collecting data using network communication protocols.

[0719] Output: Collected data (e.g., user response data, usage data)

[0720] Specific operation: The server receives the user's quiz answer, "Mount Fuji," in real time from a terminal on the third floor.

[0721] Step 6: Data Analysis

[0722] Input: Collected data

[0723] Processing: The server analyzes each user's activity level and quiz accuracy rate. Specifically, it processes each user's data through a statistical analysis algorithm and generates a report.

[0724] Output: Report of analysis results

[0725] Specific operation: The server records and analyzes that user A walked 1,000 steps and achieved an 85% correct answer rate on the quiz.

[0726] Terminal processing steps

[0727] Step 1: Play the video

[0728] Input: Video data streamed from the server

[0729] Processing: The terminal verifies the received video data and sends it to the projector device. This includes a procedure to check that the data has been transferred correctly.

[0730] Output: Image projected onto the stairs

[0731] Specific operation: The terminal projects an image of "the foot of the mountain" onto the first floor.

[0732] Step 2: Display the quiz / mini-game

[0733] Input: Quest content delivered from the server

[0734] Processing: The terminal checks the received quest content and displays it when the user reaches a specific floor location.

[0735] Output: Quizzes and mini-games displayed on the stairs

[0736] Specific action: The terminal displays a quiz to the user upon reaching the 3rd floor: "Which of the following mountains has an elevation of over 1,000 meters?"

[0737] Step 3: Handling User Interactions

[0738] Input: User's quiz answers, mini-game results

[0739] Processing: The terminal receives user input data, processes it immediately, and sends the results to the server.

[0740] Output: User input data sent to the server

[0741] Specific operation: The terminal sends the result of the user selecting "Mt. Fuji" on the 3rd floor to the server.

[0742] User processing steps

[0743] Step 1: Use the stairs

[0744] Input: Projection-mapped stairs

[0745] Processing: Users enjoy the visual effects of the stairs as they climb, which can increase their physical activity.

[0746] Output: Exercise data of users using stairs

[0747] Specific action: The user climbs the stairs from the first floor to the second floor.

[0748] Step 2: Participate in quizzes and mini-games

[0749] Input: Quizzes and mini-games displayed on the stairs

[0750] Processing: Users participate in quizzes and mini-games displayed on specific floors, and perform actions such as answering questions.

[0751] Output: Quiz answers, mini-game results

[0752] Specific action: On the third floor, the user selects "Mount Fuji" in response to the quiz, "Which of the following mountains has an elevation of over 1,000 meters?"

[0753] (Application Example 1)

[0754] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0755] In modern society, opportunities for daily exercise have decreased due to the presence of office buildings and commercial facilities, leading to a serious problem of lack of physical activity. Furthermore, while fitness and health promotion initiatives continue, many people lack the motivation to maintain them. Traditional methods struggle to balance health promotion with enjoyment, and opportunities for exercise during travel, particularly when using food delivery services, are limited.

[0756] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0757] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, an analysis means, a virtual environment display means, an exercise path generation means, and an exercise data evaluation means. This allows users to enjoy healthy activities through projection mapping while waiting for food delivery or while delivery personnel are on their way. By participating in quizzes and mini-games at specific points, users are provided with an experience that increases their exercise level, evaluates their health status, and provides feedback.

[0758] A "theme selection method" refers to a device or program for selecting a weekly theme and preparing video data based on that theme.

[0759] "Video data preparation means" refers to devices or programs for generating and preparing video data related to a selected theme.

[0760] "Video data distribution means" refers to devices or programs for transmitting prepared video data to each terminal.

[0761] "Projection means" refers to devices or programs that use received video data to project images onto real-world objects or spaces.

[0762] A "quest element setting means" is a device or program for setting interactive elements such as quizzes and mini-games and placing them at specific points.

[0763] "Data collection means" refers to devices or programs used to collect user interaction and behavioral data.

[0764] "Analysis tools" refer to devices or programs that analyze collected data and evaluate the user's exercise level and health status.

[0765] A "virtual environment display means" refers to a device or program that uses a smartphone or smart glasses to display a virtual environment and enrich the user experience.

[0766] A "movement path generation means" is a device or program for setting a movement path for a user or delivery person and placing projection mapping or quizzes along that path.

[0767] "Exercise data evaluation means" refers to a device or program for evaluating a user's exercise data and providing feedback on the results.

[0768] This invention is a system that uses projection mapping technology to help users enjoy themselves while using food delivery services, thereby combating a sedentary lifestyle and promoting their health. The system consists of the following main components:

[0769] Server Role

[0770] The server acts as the central controller for the entire system. The server performs the following main tasks:

[0771] 1. Theme Selection Method: The server selects a different theme each week (e.g., space travel, underwater exploration, etc.).

[0772] 2. Video data preparation means: Based on the selected theme, the server generates and prepares the relevant video data.

[0773] 3. Video data distribution method: The prepared video data is sent to each terminal.

[0774] 4. Quest element setting method: Quizzes and mini-games are set on specific walking routes or staircases, and the data is distributed to the terminal.

[0775] 5. Data collection methods: User interaction data and behavioral data will be collected from each terminal.

[0776] 6. Analysis methods: The collected data is analyzed to evaluate the user's exercise level and health status.

[0777] 7. Virtual environment display means: A virtual environment is displayed on a smartphone, smart glasses, etc.

[0778] 8. Movement path generation means: Sets the movement path of the user or delivery person and places projection mapping along that path.

[0779] 9. Exercise data evaluation means: Evaluate the user's exercise data and provide feedback on the results.

[0780] Terminal role

[0781] The terminal will be installed on the smartphones or smart glasses of delivery personnel and users. The terminal will perform the following tasks:

[0782] 1. Virtual environment display means: Based on video data received from the server, projection mapping is performed onto real-world stairs and walking paths.

[0783] 2. Displaying quest elements: Display quizzes or mini-games at specific points to encourage user interaction.

[0784] 3. User interaction processing: Receive and process user quiz answers and mini-game results, and send them to the server.

[0785] User roles

[0786] Users can use this system to make good use of their waiting time for food delivery while enjoying fitness. The following tasks will be performed:

[0787] 1. Use of food delivery: Users order food delivery through the app.

[0788] 2. Projection Mapping Experience: While waiting for the delivery person to arrive, users can enjoy projection mapping while exercising.

[0789] 3. Participate in quizzes and mini-games: Participate in quizzes and mini-games at specific points and enjoy special effects depending on the results.

[0790] Specific example

[0791] For example, if a user launches a fitness app and orders a pizza, the delivery person's route is given a "space travel" theme. The user then jogs or walks while experiencing projection mapping using their smartphone or smart glasses. At certain points, a quiz appears asking, "How many times larger is Mars than Earth?" If the user answers correctly, a special effect occurs, boosting their motivation to exercise.

[0792] Example of a prompt

[0793] Please provide specific ideas for quizzes and mini-games that can be applied to health promotion applications using projection mapping. For example, quizzes and mini-games themed around "space travel" are possible.

[0794] This system is a new food delivery method that encourages users to exercise healthily and provides an enjoyable experience.

[0795] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0796] Step 1:

[0797] The server selects a theme each week using a theme selection mechanism. The input is index information including the start date of the week, and the output is the selected theme. Examples of themes include "space travel" and "underwater exploration." The server determines the theme based on the previous week's theme and user feedback.

[0798] Step 2:

[0799] The server uses a video data preparation mechanism to generate video data related to the selected theme. The input is the selected theme, and the output is video data corresponding to that theme. For example, if the theme is "space travel," video footage of outer space and planets will be generated. This video data will be delivered in a later step.

[0800] Step 3:

[0801] The server transmits the prepared video data to each terminal using a video data distribution method. The input is the video data, and the output is a delivery success message to each terminal. The video data is transmitted from the server to the terminals via the network.

[0802] Step 4:

[0803] The device projects received video data onto real-world stairs or walking paths using projection technology. The input is video data, and the output is the projected image. The user views this projected image through a smartphone or smart glasses.

[0804] Step 5:

[0805] The server uses a quest element setting mechanism to configure quizzes and mini-games and distributes the data to the terminal. The input is the configuration information, and the output is a message indicating successful distribution to the terminal. For example, if the theme is "Space Travel," a quiz such as "How many times larger is Mars than Earth?" will be set.

[0806] Step 6:

[0807] The device displays received quizzes or mini-games to the user when they reach a specific point. The input is the received quiz data, and the output is the quiz screen. When the user answers the quiz, the result is sent to the device.

[0808] Step 7:

[0809] The terminal sends the user's answer to the server. The input is the user's answer data, and the output is a message indicating successful transmission to the server. The answer data is transmitted from the terminal to the server via the network.

[0810] Step 8:

[0811] The server collects data transmitted from each terminal using data collection methods. The input consists of user response data and behavioral data, while the output is the collected data. The server collects user response data and analyzes it in the next step.

[0812] Step 9:

[0813] The server analyzes the collected data using analytical tools to evaluate the user's exercise level and health status. The input is the collected data, and the output is the analysis results. Based on the analysis results, feedback on health status and exercise advice are generated.

[0814] Step 10:

[0815] The server uses an exercise data evaluation system to provide the user with analysis results. The input is the analysis results, and the output is a feedback message. The feedback message is notified to the user through the app, providing information that can help improve their health.

[0816] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0817] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes, quest elements, and an emotion engine that recognizes the user's emotions, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[0818] Server Role

[0819] The server acts as the controller for the entire system and performs the following main tasks:

[0820] 1. Theme selection:

[0821] The server selects a different theme each week (for example, mountain climbing, underwater, space, etc.) and prepares video data for it.

[0822] 2. Preparation and distribution of video data:

[0823] Based on the selected theme, the server prepares the video data and distributes it to the projection terminals installed on each floor.

[0824] 3. Setting Quest Elements:

[0825] The server sets up quizzes and mini-games on specific staircases and delivers them to the terminals.

[0826] 4. Data collection and analysis:

[0827] The server receives emotional data transmitted from the emotion engine and analyzes it in conjunction with the user's activity level and health status.

[0828] Terminal role

[0829] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server:

[0830] 1. Play back the video:

[0831] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs.

[0832] 2. Display of quizzes and mini-games:

[0833] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[0834] 3. User emotion recognition:

[0835] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine. The emotion engine analyzes this data to recognize the user's emotional state.

[0836] 4. Interaction handling:

[0837] The device provides feedback based on the user's emotional state and quiz answers, and adjusts the projected images and sound effects accordingly.

[0838] User roles

[0839] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[0840] 1. Use of stairs:

[0841] Users climb the stairs while enjoying visual effects created by projection mapping.

[0842] 2. Participation in quizzes and mini-games:

[0843] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[0844] 3. Emotion Recognition Interaction:

[0845] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system. For example, if the user is feeling happy, the projected image becomes more vivid, enhancing the entertainment value.

[0846] Specific example

[0847] For example, if the theme for the following week is set to "Space Exploration," the server will prepare video data related to the "Space Exploration" theme and distribute it to terminals on each floor from the first to the tenth floor.

[0848] On the first floor, a terminal projects an image of a "spaceship" onto the staircase, and as the user ascends from the second floor, the scenery changes from a "starry sky" to "planets" to a "galaxy." Upon reaching the third floor, the user is presented with a quiz: "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if correct, a special effect (such as a star effect) is displayed on the staircase leading to the next floor.

[0849] Furthermore, because it incorporates an emotion engine, it recognizes emotions from the user's facial expressions and voice. For example, if the user shows a surprised expression, the device detects this and sends emotion data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[0850] The server collects data from terminals on each floor and analyzes user response data, emotional data, and stair usage data to evaluate exercise levels and health status. This data is used for user health improvement programs and as feedback for the entire company.

[0851] As described above, this invention provides a stair-use promotion system that also takes into account the user's emotions, thereby alleviating lack of exercise, promoting health, and simultaneously enhancing entertainment value.

[0852] The following describes the processing flow.

[0853] Step 1:

[0854] Theme selection (server)

[0855] The server checks the current week and selects a new theme. For example, the theme for next week might be set to "Space Exploration".

[0856] Step 2:

[0857] Preparation of video data (server)

[0858] The server collects and prepares relevant video data based on the selected theme. For example, it prepares a video dataset for space exploration (spaceships, starry skies, planets, etc.).

[0859] Step 3:

[0860] Video data distribution (server)

[0861] The server sequentially distributes the prepared video data to the projection terminals on each floor. For example, it sends video data from the first floor to the tenth floor.

[0862] Step 4:

[0863] Receiving and saving video data (on the device)

[0864] The device receives video data transmitted from the server and saves it to local storage. Once saving is complete, it begins preparing for playback.

[0865] Step 5:

[0866] Projection Start (Device)

[0867] The device activates the projector at a specified time and projects images onto the walls and floor of the staircase. For example, an image of a spaceship is displayed on the staircase on the first floor.

[0868] Step 6:

[0869] Quest element settings (server)

[0870] The server prepares quiz or mini-game data for specific staircases and sends it to the terminals. For example, the 3rd floor might have a quiz like, "What is the distance from the Earth to the Moon?"

[0871] Step 7:

[0872] Quest data reception and display preparation (device)

[0873] The terminal receives data for quizzes and mini-games and prepares to display it when the user reaches the desired location. It verifies the validity of the data and prepares the interactive screen.

[0874] Step 8:

[0875] Emotional recognition (emotional engine)

[0876] Sensors such as cameras and microphones installed on the device capture the user's facial expressions, voice, and body movements, and transmit this data to the emotion engine. The emotion engine analyzes this data to recognize the user's emotions.

[0877] Step 9:

[0878] User interaction (user)

[0879] The user climbs the stairs while enjoying projection mapping. A quiz is displayed when they reach a specific floor (for example, the 3rd floor). The user answers the quiz. Additionally, an emotion recognition system analyzes the user's emotional state.

[0880] Step 10:

[0881] Receiving and evaluating responses (device)

[0882] The device receives the user's answer and determines whether it is correct or incorrect. If correct, a special effect is added to the video leading up to the next floor. For example, a special star effect is displayed when the answer is correct. At the same time, it also receives emotional data from the emotion engine.

[0883] Step 11:

[0884] Emotion-based feedback (device)

[0885] The device adjusts projected images and sound effects based on emotional data obtained from the emotion engine. For example, if it recognizes that the user is enjoying themselves, it makes the images more vibrant and the music more upbeat.

[0886] Step 12:

[0887] Data collection (device)

[0888] The device collects user response data, sentiment data, and stair usage data. It then prepares the collected data for transmission to the server.

[0889] Step 13:

[0890] Data transmission (device)

[0891] The device sends the collected data to the server. This data may include each user's quiz answers, emotional state, and stair climbing time.

[0892] Step 14:

[0893] Data reception and analysis (server)

[0894] The server receives data sent from the terminal and stores it in a database. It then begins analysis to assess exercise levels and health status.

[0895] Step 15:

[0896] Feedback of results (server)

[0897] The server generates a report based on the analysis results and notifies administrators and relevant parties. It reports on each user's health status, activity levels, and emotional fluctuations.

[0898] Through the steps outlined above, the system aims to encourage users to use stairs in an enjoyable way, alleviate lack of exercise, and promote health. Furthermore, it can enhance the entertainment value by providing interactions that respond to the user's emotional state.

[0899] (Example 2)

[0900] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0901] The problems of lack of exercise and poor health among modern people are serious, and it has been pointed out that stairs are not used very often, especially in office buildings and commercial facilities. Furthermore, because using stairs is monotonous, many people tend to use elevators. As a result, physical activity decreases, which has a negative impact on health. This invention aims to alleviate lack of exercise and promote health by transforming the use of stairs into an entertaining experience, thereby encouraging users to use stairs voluntarily.

[0902] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0903] In this invention, the server includes a theme selection means for automatically selecting a weekly theme, a video data preparation means for preparing video data based on the theme, a video data distribution means for distributing video data from the server to terminals installed on each floor, a projection means for projecting images onto the stairs using a projector, a quest element setting means for setting quizzes or mini-games on specific staircases, a data collection means for capturing the user's facial expressions and voice using a camera and microphone, an analysis means for analyzing the collected data, and an emotion adjustment means for adjusting the projected image based on the user's emotion data. This makes it possible to transform the use of stairs into an entertaining experience, alleviate users' lack of exercise, and promote their health.

[0904] A "theme selection means" is a device or program for automatically selecting a different theme each week.

[0905] "Video data preparation means" refers to a device or program for producing or preparing video data based on a selected theme.

[0906] "Video data distribution means" refers to a device or program for distributing prepared video data from a server to a terminal.

[0907] "Projection means" refers to a device or program for projecting received video data onto a staircase using a projector device.

[0908] A "quest element setting means" is a device or program for setting quizzes or mini-games on specific staircases and providing them to the user.

[0909] "Data collection means" refers to a device or program that uses a camera and microphone to capture the user's facial expressions and voice, and collect that data.

[0910] "Analysis means" refers to a device or program that analyzes collected emotional data, motor data, response data, etc., to evaluate the user's state and behavior.

[0911] An "emotion adjustment device" is a device or program that adjusts projected images and sound effects based on the user's emotional data to optimize the entertainment experience.

[0912] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes, quest elements, and an emotion engine that recognizes the user's emotions, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[0913] Server Role

[0914] The server acts as the controller for the entire system and performs the following main tasks:

[0915] Theme Selection Methods

[0916] The server automatically selects a different theme each week. For example, it uses a generation AI model to select themes such as "space exploration" and "seabed exploration."

[0917] Video data preparation means

[0918] The server prepares video data based on the selected theme. This video data is generated to be suitable for the staircases on each floor. Specifically, it is prepared using video editing software and CG software.

[0919] Video data distribution method

[0920] The server distributes the prepared video data to terminals installed on each floor using the HTTP protocol.

[0921] Quest element setting method

[0922] The server sets up quizzes and mini-games on specific staircases and distributes the data to the terminals. The set quest elements encourage user interaction when using the stairs.

[0923] Data acquisition methods

[0924] The server acquires user facial expressions and voice data collected using cameras and microphones. This allows it to understand the user's emotional state.

[0925] analytical means

[0926] The server analyzes emotional data transmitted from the emotion engine and evaluates it in conjunction with the user's activity level and health status. This evaluation result is used as feedback data for the entire company.

[0927] emotional regulation means

[0928] The server adjusts the projected image based on the user's emotional data. For example, if the user is feeling happy, it adjusts the color and brightness of the image to enhance its entertainment value.

[0929] Terminal role

[0930] Terminals are installed on each floor and stairwell, and perform the following tasks based on data received from the server:

[0931] Video playback

[0932] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs. For example, it connects to the projector using an HDMI cable.

[0933] Display of quizzes and mini-games

[0934] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate. A special effect is added when the user answers correctly.

[0935] User emotion recognition

[0936] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine.

[0937] Interaction Processing

[0938] The device provides feedback based on the user's emotional state and quiz answers, and adjusts the projected images and sound effects accordingly.

[0939] User roles

[0940] Users can use this system to enjoy climbing stairs:

[0941] Use of stairs

[0942] Users climb the stairs while enjoying visual effects created by projection mapping.

[0943] Participation in quizzes and mini-games

[0944] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[0945] Emotion Recognition Interaction

[0946] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system.

[0947] Specific example

[0948] For example, if the theme for the following week is set to "Space Exploration," the server will prepare video data related to the "Space Exploration" theme and distribute it to terminals on each floor from the first to the tenth floor.

[0949] On the first floor, a terminal projects an image of a "spaceship" onto the staircase, and as the user ascends from the second floor, the scenery changes from a "starry sky" to "planets" to a "galaxy." Upon reaching the third floor, the user is presented with a quiz: "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if correct, a special effect (such as a star effect) is displayed on the staircase leading to the next floor.

[0950] Furthermore, because it incorporates an emotion engine, it recognizes emotions from the user's facial expressions and voice. For example, if the user shows a surprised expression, the device detects this and sends emotion data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[0951] Example of a prompt

[0952] This week's theme is "Space Exploration." Project different images on each floor as the user climbs the stairs from the 1st to the 10th floor. Display an image of a "spaceship" on the 1st floor, a "starry sky" on the 2nd floor, and a "planet" on the 3rd floor. Also, on the 3rd floor, present a quiz: "What is the distance from Earth to the Moon?" If the user answers correctly, display a special star effect on the stairs to the next floor. Capture the user's facial expressions and voice using the camera and microphone, and if an expression of surprise is detected, send that data to the server.

[0953] Through the above steps, the system takes user emotions into consideration, addresses lack of exercise, and enhances entertainment value.

[0954] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0955] Step 1: Theme Selection

[0956] The server uses a generative AI model to select a weekly theme. To do this, the server receives various data (user interests, seasonal events, etc.) as input and selects the theme for the following week (e.g., space exploration) as output.

[0957] Step 2: Preparing the video data

[0958] Based on the selected theme, the server prepares video data using video editing software and CG software. In this step, the server receives a theme (e.g., space exploration) as input and generates corresponding video data (e.g., spaceship, starry sky, planets) as output.

[0959] Step 3: Distribution of video data

[0960] The server delivers the prepared video data to the terminals using the HTTP protocol. In this step, the server receives video data (e.g., spaceship, starry sky, planets) as input and outputs a distribution request to send it to the terminals on each floor.

[0961] Step 4: Setting up quest elements

[0962] The server sets up quizzes and mini-games on specific staircases and distributes the data to the terminals. In this step, the server receives quest elements (e.g., What is the distance from Earth to the Moon?) as input and outputs a distribution request to send them to the target terminals.

[0963] Step 5: Play the video

[0964] The terminal sends the received video data to the projector device to perform projection mapping on the stairs. In this step, the terminal receives video data (e.g., spaceship, starry sky, planets) from the server as input and outputs a video transfer request to send to the projector.

[0965] Step 6: Displaying quizzes and mini-games

[0966] At specific points along the staircase, the terminal displays quizzes or mini-games, allowing the user to participate. In this step, the terminal receives quiz data from the server (e.g., "What is the distance from the Earth to the Moon?") as input and sends a display request to output it to the display. It also receives the user's answer and feeds it back into the next action.

[0967] Step 7: User emotion recognition

[0968] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and sends them to the emotion engine. In this step, the device receives data from sensors (e.g., facial expressions, voice) as input, captures it as emotion data, and outputs a data transfer request to send it to the emotion engine.

[0969] Step 8: Interaction Processing

[0970] The device provides feedback based on the user's emotional state and quiz results, adjusting the projected images and sound effects accordingly. In this step, the device receives emotional data and quiz results as input and outputs requests to adjust the images and sounds based on them. Specifically, if the user is feeling happy, the device adjusts the colors and brightness of the images.

[0971] Step 9: Data Collection and Analysis

[0972] The server analyzes emotional data sent from the emotion engine along with the user's exercise data and response data. In this step, the server receives emotional data, exercise data, and response data as input and outputs the analysis results. Specifically, this is used to evaluate the user's health status and entertainment experience.

[0973] Because each step involves a detailed description of specific actions, this system's program transforms using stairs into an entertaining experience, providing a means to alleviate users' lack of exercise and promote their health.

[0974] (Application Example 2)

[0975] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0976] This project aims to address the lack of exercise among stair users, promote their health, and make stair use enjoyable by enhancing its entertainment value. However, conventional methods have limited effectiveness in motivating stair users and improving the user experience, necessitating new technological solutions. Furthermore, technologies for appropriate emotion recognition and feedback based on those results are still underdeveloped. A system is needed to address these challenges.

[0977] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0978] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, an analysis means, a user recognition means, an emotion recognition means, and an emotion-based feedback means. This makes it possible to alleviate the user's lack of exercise, improve their health, and provide a dynamic entertainment experience based on emotions.

[0979] The "theme selection method" is a function that selects a different theme for each specified period and prepares the necessary video data and public relations information based on that theme.

[0980] The "video data preparation means" is a function that creates and edits video data to be used for projection mapping based on the selected theme.

[0981] "Video data distribution means" refers to a function that transmits prepared video data to a projection means.

[0982] "Projection means" refers to a function that displays received video data on stairs or walls using projection mapping technology.

[0983] The "quest element setting method" is a function that allows users to set up quizzes or mini-games at specific locations on stairs or floors, enabling them to participate in them.

[0984] "Data collection methods" refer to functions that collect user behavior data, quest participation data, and emotional data.

[0985] "Analysis tools" refer to functions that analyze collected data to evaluate the user's health status, exercise level, emotional state, and other factors.

[0986] "User recognition means" refers to a function that recognizes the user's face and body movements to identify the individual.

[0987] "Emotion recognition means" refers to a function that analyzes the user's facial expressions and voice to determine their emotional state.

[0988] "Emotion-based feedback mechanisms" are functions that adjust projection images and sound effects based on recognized emotions to provide appropriate feedback.

[0989] This invention is designed as a system that enhances the entertainment value of using stairs, alleviates lack of exercise, and promotes health. Specific embodiments using each component of this system are described below.

[0990] Server Role

[0991] The server acts as the controller for the entire system, performing the following key tasks.

[0992] 1. Theme Selection Methods

[0993] The server selects a different theme for each specified period. For example, themes might include "space exploration" or "underwater exploration," and based on these, it prepares the necessary video data and quizzes.

[0994] 2. Means for preparing video data

[0995] The server creates and edits video data to be used for projection mapping based on the selected theme. The video data is prepared to correspond to each floor of the staircase. For example, if the theme is "Space Exploration," the first floor will be set to "Earth," the second floor to "Moon," and the third floor to "Mars."

[0996] 3. Video data distribution method

[0997] The server transmits the prepared video data to projection terminals installed on each floor. The video data is distributed in real time via the network.

[0998] 4. Quest element setting means

[0999] The server sets up quizzes and mini-games at specific stairwells or floor locations and delivers them to projection terminals. For example, a quiz such as "What is the distance from the Earth to the Moon?" might be displayed at the third floor, and users can answer it using their smartphones.

[1000] 5. Data Collection Methods

[1001] The server collects user behavior data, quiz participation data, and emotional data. This allows the server to understand user activity levels and quest participation status.

[1002] 6. Analysis tools

[1003] The server analyzes the collected data to evaluate the user's health, activity level, emotional state, and other factors. This helps in providing more effective feedback and selecting future topics.

[1004] Terminal role

[1005] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server.

[1006] 1. Play back video

[1007] The terminal transmits the received video data to the projector device, which then performs projection mapping on the stairs.

[1008] 2. Display of quizzes and mini-games

[1009] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[1010] 3. User Recognition Means

[1011] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and recognizes the user.

[1012] 4. Emotion recognition means

[1013] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine. The emotion engine analyzes this data to recognize the user's emotional state.

[1014] 5. Emotion-based feedback methods

[1015] The device provides feedback based on the user's emotional state and quiz answers, adjusting the projected images and sound effects accordingly. For example, if the user is enjoying themselves, the images become more vibrant and the sound effects are enhanced.

[1016] User roles

[1017] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[1018] 1. Use of stairs

[1019] Users climb the stairs while enjoying visual effects created by projection mapping.

[1020] 2. Participation in quizzes and mini-games

[1021] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[1022] 3. Emotion Recognition Interaction

[1023] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system. For example, if the user shows a surprised expression, the device detects this and sends emotional data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[1024] Hardware and software to be used

[1025] Hardware: Smartphones (e.g., iPhone, Google® Pixel), projector devices, sensor terminals (cameras, microphones, etc.)

[1026] Software: EmotionRecognition library (for emotion recognition), OpenCV library (for camera input and video processing), server-side web frameworks such as Flask and Django.

[1027] Specific example

[1028] For example, if the theme is set to "Space Exploration," the server prepares video data related to the "Space Exploration" theme and distributes it to terminals on each floor from the 1st to the 10th floor. The terminal on the 1st floor projects an image of a "spaceship" onto the staircase, and as the user goes up from the 2nd floor, the scenery changes from a "starry sky" to "planets" to "galaxies." When the user reaches the 3rd floor, a quiz is displayed asking, "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if they answer correctly, a special effect (for example, a star effect) is displayed on the staircase to the next floor.

[1029] Example of a prompt

[1030] Example of quiz generation: "To the generation AI model: Please create 3 quiz questions related to space exploration."

[1031] Example of emotion data analysis: "To the generative AI model: Please suggest suitable entertainment content when the emotional state is happy."

[1032] The above describes specific embodiments for carrying out this invention.

[1033] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1034] Step 1:

[1035] The server selects the theme.

[1036] The server selects a different theme for each specified period, prompting the AI ​​model with the message, "Please suggest a theme for next week," and then receives the generated theme. This theme information is used for the following processes.

[1037] Input: Theme selection prompt message

[1038] Data processing / manipulation: Themes are generated using a generative AI model.

[1039] Output: Selected theme (e.g., "Space Exploration")

[1040] Step 2:

[1041] The server prepares the video data.

[1042] The server creates video data for projection mapping using video data preparation means, based on the selected theme. It collects the necessary video materials and edits them to match the theme.

[1043] Input: Selected theme

[1044] Data processing / editing: Creation and editing of video data

[1045] Output: Prepared video data

[1046] Step 3:

[1047] The server distributes the video data.

[1048] The server distributes the prepared video data to terminals installed on each floor via the network.

[1049] Input: Prepared video data

[1050] Data processing / processing: Video data distribution

[1051] Output: Supply of video data to the terminal

[1052] Step 4:

[1053] The device plays the video.

[1054] The terminal transmits the received video data to a projector device, performing projection mapping on designated staircases or walls. The video is displayed according to the selected theme.

[1055] Input: Video data from the server

[1056] Data processing / editing: Video display using projection mapping

[1057] Output: Video displays on stairs and walls

[1058] Step 5:

[1059] The server sets the quest elements.

[1060] The server requests a generation AI model to create quizzes and mini-games related to the theme. For example, it might input a prompt like, "Create three quiz questions related to space exploration," and then receive the generated quest content.

[1061] Input: Quest generation prompt text

[1062] Data processing / manipulation: Generating quest elements using a generative AI model.

[1063] Output: Generated quest content (e.g., "What is the distance from Earth to the Moon?")

[1064] Step 6:

[1065] The server will deliver quest elements.

[1066] The server distributes the generated quest details to terminals installed on specific staircases.

[1067] Input: Generated quest content

[1068] Data processing / editing: Distribution of quest content

[1069] Output: Supplying quest content to the terminal

[1070] Step 7:

[1071] The device displays a quiz / mini-game.

[1072] The terminal displays received quest content on a projector device or terminal screen, allowing users to participate. Users answer quizzes and mini-games in real time.

[1073] Input: Quest details from the server

[1074] Data processing / manipulation: Displaying quizzes and mini-games

[1075] Output: User participation in quizzes and mini-games

[1076] Step 8:

[1077] The device recognizes the user's emotions.

[1078] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and analyzes them using emotion recognition technology.

[1079] Input: User's facial expressions, voice, and motion data

[1080] Data processing / analysis: Recognition and analysis of emotional states

[1081] Output: Recognized emotion data

[1082] Step 9:

[1083] The device sends emotional data to the server.

[1084] The device sends the recognized emotion data to the server. The server receives this data and uses it for the next processing step.

[1085] Input: Recognized emotion data

[1086] Data processing / processing: Sending emotional data

[1087] Output: Supplying emotion data to the server

[1088] Step 10:

[1089] The server provides emotion-based feedback.

[1090] The server analyzes the received emotional data and adjusts the next projection images and sound effects based on the emotions. It inputs a prompt message to the generating AI model, "Please suggest suitable entertainment content when the emotional state is happy," and sends the generated content to the terminal.

[1091] Input: Sentiment data, entertainment content suggestion prompt text

[1092] Data Processing / Editing: Analysis and generation of emotional data, content proposals using AI models.

[1093] Output: Adjusted entertainment content

[1094] Step 11:

[1095] The device displays the feedback content.

[1096] The device plays back the adjusted projection images and sound effects received from the server and provides feedback to the user. This is designed to create an entertainment experience that responds to the user's emotional state.

[1097] Input: Coordinated entertainment content from the server

[1098] Data processing / editing: Playback of projection images and sound effects.

[1099] Output: Providing feedback to users

[1100] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1101] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1102] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1103] [Third Embodiment]

[1104] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1105] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1106] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1107] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1108] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1109] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1110] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1111] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1112] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1113] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1114] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1115] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1116] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes and quest elements, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[1117] Server Role

[1118] The server acts as the controller for the entire system and performs the following main tasks:

[1119] 1. Theme selection:

[1120] The server selects a different theme each week (for example, mountain climbing, underwater, space, etc.) and prepares video data for it.

[1121] 2. Preparation and distribution of video data:

[1122] Based on the selected theme, the server prepares the video data and distributes it to the projection terminals installed on each floor.

[1123] 3. Setting Quest Elements:

[1124] The server sets up quizzes and mini-games on specific staircases and delivers them to the terminals.

[1125] 4. Data collection and analysis:

[1126] The server receives data collected from each terminal and analyzes the user's activity level and health status.

[1127] Terminal role

[1128] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server:

[1129] 1. Play back the video:

[1130] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs.

[1131] 2. Display of quizzes and mini-games:

[1132] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[1133] 3. User interaction handling:

[1134] The terminal receives and processes the user's quiz answers and mini-game results, and sends that data to the server.

[1135] User roles

[1136] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[1137] 1. Use of stairs:

[1138] Users climb the stairs while enjoying visual effects created by projection mapping.

[1139] 2. Participation in quizzes and mini-games:

[1140] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[1141] Specific example

[1142] For example, if the theme for the following week is set to "mountain climbing," the server will prepare video data related to the "mountain climbing" theme and distribute it to terminals on each floor from the 1st to the 10th floor.

[1143] On the first floor, a terminal projects an image of "the foot of a mountain" onto the staircase, and as you go up from the second floor, the scenery changes from "mountain path" to "valley" to "steep rocky area" to "snowy mountain." When a user reaches the third floor, a quiz is displayed asking, "Which of the following mountains is over 1,000 meters high?" If the user selects "Mount Fuji" from the options and answers correctly, a special effect (such as fireworks or special visual effects) is displayed on the staircase to the next floor.

[1144] The server collects data from terminals on each floor and analyzes user response data and stair usage data to evaluate exercise levels and health status. This data is used for user health improvement programs and as feedback for the entire company.

[1145] The following describes the processing flow.

[1146] Step 1:

[1147] Theme selection (server)

[1148] The server checks the current week and selects a new theme. For example, the theme for next week might be set to "Space Exploration".

[1149] Step 2:

[1150] Preparation of video data (server)

[1151] The server collects and prepares relevant video data based on the selected theme. For example, it prepares a video dataset for space exploration (spaceships, starry skies, planets, etc.).

[1152] Step 3:

[1153] Video data distribution (server)

[1154] The server sequentially distributes the prepared video data to the projection terminals on each floor. For example, it sends video data from the first floor to the tenth floor.

[1155] Step 4:

[1156] Receiving and saving video data (on the device)

[1157] The device receives video data transmitted from the server and saves it to local storage. Once saving is complete, it begins preparing for playback.

[1158] Step 5:

[1159] Projection Start (Device)

[1160] The device activates the projector at a specified time and projects images onto the walls and floor of the staircase. For example, an image of a spaceship is displayed on the staircase on the first floor.

[1161] Step 6:

[1162] Quest element settings (server)

[1163] The server prepares quiz or mini-game data for specific staircases and sends it to the terminals. For example, the 3rd floor might have a quiz like, "What is the distance from the Earth to the Moon?"

[1164] Step 7:

[1165] Quest data reception and display preparation (device)

[1166] The terminal receives data for quizzes and mini-games and prepares to display it when the user reaches the desired location. It verifies the validity of the data and prepares the interactive screen.

[1167] Step 8:

[1168] User interaction (user)

[1169] The user climbs the stairs while enjoying projection mapping. A quiz is displayed when they reach a specific floor (for example, the 3rd floor). The user then answers the quiz.

[1170] Step 9:

[1171] Receiving and evaluating responses (device)

[1172] The device receives the user's answer and determines whether it is correct or incorrect. If correct, a special effect is added to the video leading up to the next floor. For example, a special star effect is displayed when the answer is correct.

[1173] Step 10:

[1174] Data collection (device)

[1175] The terminal collects user response data and stair usage data. The collected data is then prepared for transmission to the server.

[1176] Step 11:

[1177] Data transmission (device)

[1178] The device sends the collected data to the server. This data may include, for example, each user's quiz answers and stair usage time.

[1179] Step 12:

[1180] Data reception and analysis (server)

[1181] The server receives data sent from the terminal and stores it in a database. It then begins analysis to evaluate exercise levels and health status.

[1182] Step 13:

[1183] Feedback of results (server)

[1184] The server generates a report based on the analysis results and notifies administrators and relevant parties. It reports on each user's health status and activity levels over time.

[1185] Through the steps described above, the system aims to make using stairs enjoyable for users, alleviate a lack of exercise, and promote health.

[1186] (Example 1)

[1187] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1188] In modern office buildings and commercial facilities, users' reliance on elevators leads to insufficient daily exercise and a decline in health. Furthermore, using stairs is psychologically monotonous, causing users to avoid it. Additionally, there is a lack of systems to efficiently monitor and improve health conditions and exercise levels.

[1189] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1190] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a quest element setting means, a data collection means, an analysis means, and a user interaction processing means. This makes using stairs enjoyable, helps alleviate lack of exercise, and promotes health. Furthermore, it provides psychological enjoyment by offering quizzes and mini-games, and enables efficient collection and analysis of each user's exercise data.

[1191] "Theme selection method" refers to the process of selecting a theme from the library that is appropriate for the year or season, and deciding on it as the theme for the following week.

[1192] "Video data preparation means" refers to a means for generating relevant video data based on a selected theme and preparing it for projection on each floor.

[1193] "Video data distribution means" refers to a means for distributing generated video data to terminals on each floor.

[1194] "Projection means" refers to a method for transmitting video data received by a terminal to a projector device and projecting the image onto the stairs.

[1195] A "quest element setting method" is a means of placing interactive elements such as quizzes or mini-games on a specific staircase and presenting them to the user.

[1196] "Data collection means" refers to the means of collecting user interaction and stair usage data from each terminal.

[1197] "Analysis methods" refer to methods for analyzing each user's exercise volume and quiz accuracy rate based on collected data, and generating reports to address lack of exercise and promote health.

[1198] "User interaction processing means" refers to a means for receiving and processing input data when a user answers a quiz or participates in a mini-game, and sending the results to a server.

[1199] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes and quest elements, transforming the use of stairs into an enjoyable experience, thereby addressing users' lack of exercise and promoting their health. The system as a whole is mainly divided into three elements: server, terminal, and user.

[1200] Server Role

[1201] The server acts as the controller for the entire system and performs the following main tasks:

[1202] Theme selection method:

[1203] The server's role is to select the theme for the following week from a theme library that is appropriate for the year and season. For example, it might select one theme from options such as "mountain climbing," "underwater," or "space."

[1204] Methods for preparing video data:

[1205] After a theme is selected, the server retrieves relevant video data from cloud storage based on the selected theme. It then generates video data suitable for each floor. For example, for a "mountain climbing" theme, it generates video from the "foot of the mountain" on the 1st floor to the "summit" on the 10th floor.

[1206] Video data distribution method:

[1207] The generated video data is distributed from the server to projection terminals on each floor. This makes it possible to project theme-based visual effects onto each staircase.

[1208] Quest element setting method:

[1209] The server is responsible for placing interactive elements such as quizzes and mini-games on specific staircases. For example, on the third floor, a quiz might be set up asking, "Which of the following mountains is over 1,000 meters high?"

[1210] Data acquisition methods:

[1211] The server collects user quiz answers and stair usage data from each terminal in real time. This allows for monitoring of user behavior.

[1212] Analysis means:

[1213] The server analyzes each user's exercise level and quiz accuracy rate based on the collected data. The analysis results are generated as reports to help address lack of exercise and promote health.

[1214] User interaction processing means:

[1215] The system receives input data from users when they answer quizzes or participate in mini-games, and sends the results to the server.

[1216] Terminal role

[1217] Terminals are installed on each floor and stairwell, and perform the following tasks:

[1218] Play video:

[1219] The terminal receives video data streamed from the server and transmits that data to a projector device to project the image onto the staircase. For example, the terminal on the first floor projects an image showing "the foot of the mountain."

[1220] Display of quizzes and mini-games:

[1221] The device receives quest content delivered from the server and displays quizzes or mini-games to users who reach specific locations on the stairs.

[1222] User interaction handling:

[1223] The terminal receives the user's quiz answers and mini-game participation data, and sends the results to the server.

[1224] User roles

[1225] The user performs the following tasks:

[1226] Use of stairs:

[1227] Users climb the stairs while enjoying visual effects created by projection mapping. By experiencing various themed landscapes, they can also get some exercise.

[1228] Participation in quizzes and mini-games:

[1229] When users reach a specific location on a floor, they can participate in quizzes or mini-games displayed by a projector. For example, on the third floor, they might answer a quiz asking, "Which of the following mountains is over 1,000 meters high?"

[1230] Specific examples and prompt statements

[1231] Specific example:

[1232] If the theme for the following week is set to "mountain climbing," the server prepares video data related to the "mountain climbing" theme and distributes it to terminals on each floor from the 1st to the 10th floor. The terminal on the 1st floor projects an image of "the foot of a mountain" onto the stairs, and as the user goes up from the 2nd floor, the scenery changes from "mountain path" → "valley" → "steep rocky area" → "snowy mountain." When the user reaches the 3rd floor, a quiz is displayed asking "Which of the following mountains has an altitude of over 1,000 meters?", and the user selects "Mount Fuji" from the options.

[1233] Examples of prompts for a generative AI model:

[1234] Please describe a system where projection mapping is used on the staircase from the 1st to the 10th floor, with the theme being "mountain climbing." Provide specific examples of scenery related to the theme of each floor, and detail how quizzes and mini-games are conducted on each floor.

[1235] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1236] Server processing steps

[1237] Step 1: Theme Selection

[1238] Input: Theme data from the previous week, year and season information, theme library

[1239] Processing: The server selects the theme for the following week from the theme library based on the year and season. Specifically, it scans the list of themes in the library and runs an algorithm that evaluates how well each theme fits the given conditions.

[1240] Output: Selected theme (e.g., "Mountain climbing", "Underwater", "Space")

[1241] Specific action: The server sets the theme for the following week to "mountain climbing".

[1242] Step 2: Prepare video data

[1243] Input: Selected theme (e.g., "Mountain climbing")

[1244] Processing: The server retrieves theme-related video data from cloud storage. Then, it generates video data appropriate for each floor. Specifically, for the "mountain climbing" theme, it runs an image processing algorithm to generate scenery data for floors 1 through 10.

[1245] Output: Video data corresponding to each floor (e.g., from the "foot of the mountain" on the 1st floor to the "summit" on the 10th floor)

[1246] Specific operation: The server generates video of the "foot of the mountain" for the first floor, and then generates video of the "mountain path" for the second floor.

[1247] Step 3: Video data distribution

[1248] Input: Generated video data

[1249] Processing: The server distributes video data to terminals on each floor. Specifically, it uses a protocol that transmits data to terminals via a network connection.

[1250] Output: Video data has been distributed to each terminal.

[1251] Specific operation: The server distributes the corresponding video data to each terminal from the 1st floor to the 10th floor.

[1252] Step 4: Setting Quest Elements

[1253] Input: Database of selected themes, quizzes, and mini-games

[1254] Processing: The server retrieves quizzes and mini-games related to the theme from the database and places them on specific floors. It then distributes them to projection terminals.

[1255] Output: Quest elements placed on each staircase

[1256] Specific operation: The server sets a quiz as a quest on the 3rd floor: "Which of the following mountains has an elevation of over 1,000 meters?" and delivers it to the terminal on the 3rd floor.

[1257] Step 5: Data Acquisition

[1258] Input: User quiz answer data, stair usage data

[1259] Processing: The server receives data collected from each terminal in real time. This includes the process of collecting data using network communication protocols.

[1260] Output: Collected data (e.g., user response data, usage data)

[1261] Specific operation: The server receives the user's quiz answer, "Mount Fuji," in real time from a terminal on the third floor.

[1262] Step 6: Data Analysis

[1263] Input: Collected data

[1264] Processing: The server analyzes each user's activity level and quiz accuracy rate. Specifically, it processes each user's data through a statistical analysis algorithm and generates a report.

[1265] Output: Report of analysis results

[1266] Specific operation: The server records and analyzes that user A walked 1,000 steps and achieved an 85% correct answer rate on the quiz.

[1267] Terminal processing steps

[1268] Step 1: Play the video

[1269] Input: Video data streamed from the server

[1270] Processing: The terminal verifies the received video data and sends it to the projector device. This includes a procedure to check that the data has been transferred correctly.

[1271] Output: Image projected onto the stairs

[1272] Specific operation: The terminal projects an image of "the foot of the mountain" onto the first floor.

[1273] Step 2: Display the quiz / mini-game

[1274] Input: Quest content delivered from the server

[1275] Processing: The terminal checks the received quest content and displays it when the user reaches a specific floor location.

[1276] Output: Quizzes and mini-games displayed on the stairs

[1277] Specific action: The terminal displays a quiz to the user upon reaching the 3rd floor: "Which of the following mountains has an elevation of over 1,000 meters?"

[1278] Step 3: Handling User Interactions

[1279] Input: User's quiz answers, mini-game results

[1280] Processing: The terminal receives user input data, processes it immediately, and sends the results to the server.

[1281] Output: User input data sent to the server

[1282] Specific operation: The terminal sends the result of the user selecting "Mt. Fuji" on the 3rd floor to the server.

[1283] User processing steps

[1284] Step 1: Use the stairs

[1285] Input: Projection-mapped stairs

[1286] Processing: Users enjoy the visual effects of the stairs as they climb, which can increase their physical activity.

[1287] Output: Exercise data of users using stairs

[1288] Specific action: The user climbs the stairs from the first floor to the second floor.

[1289] Step 2: Participate in quizzes and mini-games

[1290] Input: Quizzes and mini-games displayed on the stairs

[1291] Processing: Users participate in quizzes and mini-games displayed on specific floors, and perform actions such as answering questions.

[1292] Output: Quiz answers, mini-game results

[1293] Specific action: On the third floor, the user selects "Mount Fuji" in response to the quiz, "Which of the following mountains has an elevation of over 1,000 meters?"

[1294] (Application Example 1)

[1295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1296] In modern society, opportunities for daily exercise have decreased due to the presence of office buildings and commercial facilities, leading to a serious problem of lack of physical activity. Furthermore, while fitness and health promotion initiatives continue, many people lack the motivation to maintain them. Traditional methods struggle to balance health promotion with enjoyment, and opportunities for exercise during travel, particularly when using food delivery services, are limited.

[1297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1298] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, an analysis means, a virtual environment display means, an exercise path generation means, and an exercise data evaluation means. This allows users to enjoy healthy activities through projection mapping while waiting for food delivery or while delivery personnel are on their way. By participating in quizzes and mini-games at specific points, users are provided with an experience that increases their exercise level, evaluates their health status, and provides feedback.

[1299] A "theme selection method" refers to a device or program for selecting a weekly theme and preparing video data based on that theme.

[1300] "Video data preparation means" refers to devices or programs for generating and preparing video data related to a selected theme.

[1301] "Video data distribution means" refers to devices or programs for transmitting prepared video data to each terminal.

[1302] "Projection means" refers to devices or programs that use received video data to project images onto real-world objects or spaces.

[1303] A "quest element setting means" is a device or program for setting interactive elements such as quizzes and mini-games and placing them at specific points.

[1304] "Data collection means" refers to devices or programs used to collect user interaction and behavioral data.

[1305] "Analysis tools" refer to devices or programs that analyze collected data and evaluate the user's exercise level and health status.

[1306] A "virtual environment display means" refers to a device or program that uses a smartphone or smart glasses to display a virtual environment and enrich the user experience.

[1307] A "movement path generation means" is a device or program for setting a movement path for a user or delivery person and placing projection mapping or quizzes along that path.

[1308] "Exercise data evaluation means" refers to a device or program for evaluating a user's exercise data and providing feedback on the results.

[1309] This invention is a system that uses projection mapping technology to help users enjoy themselves while using food delivery services, thereby combating a sedentary lifestyle and promoting their health. The system consists of the following main components:

[1310] Server Role

[1311] The server acts as the central controller for the entire system. The server performs the following main tasks:

[1312] 1. Theme Selection Method: The server selects a different theme each week (e.g., space travel, underwater exploration, etc.).

[1313] 2. Video data preparation means: Based on the selected theme, the server generates and prepares the relevant video data.

[1314] 3. Video data distribution method: The prepared video data is sent to each terminal.

[1315] 4. Quest element setting method: Quizzes and mini-games are set on specific walking routes or staircases, and the data is distributed to the terminal.

[1316] 5. Data collection methods: User interaction data and behavioral data will be collected from each terminal.

[1317] 6. Analysis methods: The collected data is analyzed to evaluate the user's exercise level and health status.

[1318] 7. Virtual environment display means: A virtual environment is displayed on a smartphone, smart glasses, etc.

[1319] 8. Movement path generation means: Sets the movement path of the user or delivery person and places projection mapping along that path.

[1320] 9. Exercise data evaluation means: Evaluate the user's exercise data and provide feedback on the results.

[1321] Terminal role

[1322] The terminal will be installed on the smartphones or smart glasses of delivery personnel and users. The terminal will perform the following tasks:

[1323] 1. Virtual environment display means: Based on video data received from the server, projection mapping is performed onto real-world stairs and walking paths.

[1324] 2. Displaying quest elements: Display quizzes or mini-games at specific points to encourage user interaction.

[1325] 3. User interaction processing: Receive and process user quiz answers and mini-game results, and send them to the server.

[1326] User roles

[1327] Users can use this system to make good use of their waiting time for food delivery while enjoying fitness. The following tasks will be performed:

[1328] 1. Use of food delivery: Users order food delivery through the app.

[1329] 2. Projection Mapping Experience: While waiting for the delivery person to arrive, users can enjoy projection mapping while exercising.

[1330] 3. Participate in quizzes and mini-games: Participate in quizzes and mini-games at specific points and enjoy special effects depending on the results.

[1331] Specific example

[1332] For example, if a user launches a fitness app and orders a pizza, the delivery person's route is given a "space travel" theme. The user then jogs or walks while experiencing projection mapping using their smartphone or smart glasses. At certain points, a quiz appears asking, "How many times larger is Mars than Earth?" If the user answers correctly, a special effect occurs, boosting their motivation to exercise.

[1333] Example of a prompt

[1334] Please provide specific ideas for quizzes and mini-games that can be applied to health promotion applications using projection mapping. For example, quizzes and mini-games themed around "space travel" are possible.

[1335] This system is a new food delivery method that encourages users to exercise healthily and provides an enjoyable experience.

[1336] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1337] Step 1:

[1338] The server selects a theme each week using a theme selection mechanism. The input is index information including the start date of the week, and the output is the selected theme. Examples of themes include "space travel" and "underwater exploration." The server determines the theme based on the previous week's theme and user feedback.

[1339] Step 2:

[1340] The server uses a video data preparation mechanism to generate video data related to the selected theme. The input is the selected theme, and the output is video data corresponding to that theme. For example, if the theme is "space travel," video footage of outer space and planets will be generated. This video data will be delivered in a later step.

[1341] Step 3:

[1342] The server transmits the prepared video data to each terminal using a video data distribution method. The input is the video data, and the output is a delivery success message to each terminal. The video data is transmitted from the server to the terminals via the network.

[1343] Step 4:

[1344] The device projects received video data onto real-world stairs or walking paths using projection technology. The input is video data, and the output is the projected image. The user views this projected image through a smartphone or smart glasses.

[1345] Step 5:

[1346] The server uses a quest element setting mechanism to configure quizzes and mini-games and distributes the data to the terminal. The input is the configuration information, and the output is a message indicating successful distribution to the terminal. For example, if the theme is "Space Travel," a quiz such as "How many times larger is Mars than Earth?" will be set.

[1347] Step 6:

[1348] The device displays received quizzes or mini-games to the user when they reach a specific point. The input is the received quiz data, and the output is the quiz screen. When the user answers the quiz, the result is sent to the device.

[1349] Step 7:

[1350] The terminal sends the user's answer to the server. The input is the user's answer data, and the output is a message indicating successful transmission to the server. The answer data is transmitted from the terminal to the server via the network.

[1351] Step 8:

[1352] The server collects data transmitted from each terminal using data collection methods. The input consists of user response data and behavioral data, while the output is the collected data. The server collects user response data and analyzes it in the next step.

[1353] Step 9:

[1354] The server analyzes the collected data using analytical tools to evaluate the user's exercise level and health status. The input is the collected data, and the output is the analysis results. Based on the analysis results, feedback on health status and exercise advice are generated.

[1355] Step 10:

[1356] The server uses an exercise data evaluation system to provide the user with analysis results. The input is the analysis results, and the output is a feedback message. The feedback message is notified to the user through the app, providing information that can help improve their health.

[1357] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1358] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes, quest elements, and an emotion engine that recognizes the user's emotions, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[1359] Server Role

[1360] The server acts as the controller for the entire system and performs the following main tasks:

[1361] 1. Theme selection:

[1362] The server selects a different theme each week (for example, mountain climbing, underwater, space, etc.) and prepares video data for it.

[1363] 2. Preparation and distribution of video data:

[1364] Based on the selected theme, the server prepares the video data and distributes it to the projection terminals installed on each floor.

[1365] 3. Setting Quest Elements:

[1366] The server sets up quizzes and mini-games on specific staircases and delivers them to the terminals.

[1367] 4. Data collection and analysis:

[1368] The server receives emotional data transmitted from the emotion engine and analyzes it in conjunction with the user's activity level and health status.

[1369] Terminal role

[1370] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server:

[1371] 1. Play back the video:

[1372] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs.

[1373] 2. Display of quizzes and mini-games:

[1374] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[1375] 3. User emotion recognition:

[1376] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine. The emotion engine analyzes this data to recognize the user's emotional state.

[1377] 4. Interaction handling:

[1378] The device provides feedback based on the user's emotional state and quiz answers, and adjusts the projected images and sound effects accordingly.

[1379] User roles

[1380] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[1381] 1. Use of stairs:

[1382] Users climb the stairs while enjoying visual effects created by projection mapping.

[1383] 2. Participation in quizzes and mini-games:

[1384] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[1385] 3. Emotion Recognition Interaction:

[1386] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system. For example, if the user is feeling happy, the projected image becomes more vivid, enhancing the entertainment value.

[1387] Specific example

[1388] For example, if the theme for the following week is set to "Space Exploration," the server will prepare video data related to the "Space Exploration" theme and distribute it to terminals on each floor from the first to the tenth floor.

[1389] On the first floor, a terminal projects an image of a "spaceship" onto the staircase, and as the user ascends from the second floor, the scenery changes from a "starry sky" to "planets" to a "galaxy." Upon reaching the third floor, the user is presented with a quiz: "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if correct, a special effect (such as a star effect) is displayed on the staircase leading to the next floor.

[1390] Furthermore, because it incorporates an emotion engine, it recognizes emotions from the user's facial expressions and voice. For example, if the user shows a surprised expression, the device detects this and sends emotion data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[1391] The server collects data from terminals on each floor and analyzes user response data, emotional data, and stair usage data to evaluate exercise levels and health status. This data is used for user health improvement programs and as feedback for the entire company.

[1392] As described above, this invention provides a stair-use promotion system that also takes into account the user's emotions, thereby alleviating lack of exercise, promoting health, and simultaneously enhancing entertainment value.

[1393] The following describes the processing flow.

[1394] Step 1:

[1395] Theme selection (server)

[1396] The server checks the current week and selects a new theme. For example, the theme for next week might be set to "Space Exploration".

[1397] Step 2:

[1398] Preparation of video data (server)

[1399] The server collects and prepares relevant video data based on the selected theme. For example, it prepares a video dataset for space exploration (spaceships, starry skies, planets, etc.).

[1400] Step 3:

[1401] Video data distribution (server)

[1402] The server sequentially distributes the prepared video data to the projection terminals on each floor. For example, it sends video data from the first floor to the tenth floor.

[1403] Step 4:

[1404] Receiving and saving video data (on the device)

[1405] The device receives video data transmitted from the server and saves it to local storage. Once saving is complete, it begins preparing for playback.

[1406] Step 5:

[1407] Projection Start (Device)

[1408] The device activates the projector at a specified time and projects images onto the walls and floor of the staircase. For example, an image of a spaceship is displayed on the staircase on the first floor.

[1409] Step 6:

[1410] Quest element settings (server)

[1411] The server prepares quiz or mini-game data for specific staircases and sends it to the terminals. For example, the 3rd floor might have a quiz like, "What is the distance from the Earth to the Moon?"

[1412] Step 7:

[1413] Quest data reception and display preparation (device)

[1414] The terminal receives data for quizzes and mini-games and prepares to display it when the user reaches the desired location. It verifies the validity of the data and prepares the interactive screen.

[1415] Step 8:

[1416] Emotional recognition (emotional engine)

[1417] Sensors such as cameras and microphones installed on the device capture the user's facial expressions, voice, and body movements, and transmit this data to the emotion engine. The emotion engine analyzes this data to recognize the user's emotions.

[1418] Step 9:

[1419] User interaction (user)

[1420] The user climbs the stairs while enjoying projection mapping. A quiz is displayed when they reach a specific floor (for example, the 3rd floor). The user answers the quiz. Additionally, an emotion recognition system analyzes the user's emotional state.

[1421] Step 10:

[1422] Receiving and evaluating responses (device)

[1423] The device receives the user's answer and determines whether it is correct or incorrect. If correct, a special effect is added to the video leading up to the next floor. For example, a special star effect is displayed when the answer is correct. At the same time, it also receives emotional data from the emotion engine.

[1424] Step 11:

[1425] Emotion-based feedback (device)

[1426] The device adjusts projected images and sound effects based on emotional data obtained from the emotion engine. For example, if it recognizes that the user is enjoying themselves, it makes the images more vibrant and the music more upbeat.

[1427] Step 12:

[1428] Data collection (device)

[1429] The device collects user response data, sentiment data, and stair usage data. It then prepares the collected data for transmission to the server.

[1430] Step 13:

[1431] Data transmission (device)

[1432] The device sends the collected data to the server. This data may include each user's quiz answers, emotional state, and stair climbing time.

[1433] Step 14:

[1434] Data reception and analysis (server)

[1435] The server receives data sent from the terminal and stores it in a database. It then begins analysis to assess exercise levels and health status.

[1436] Step 15:

[1437] Feedback of results (server)

[1438] The server generates a report based on the analysis results and notifies administrators and relevant parties. It reports on each user's health status, activity levels, and emotional fluctuations.

[1439] Through the steps outlined above, the system aims to encourage users to use stairs in an enjoyable way, alleviate lack of exercise, and promote health. Furthermore, it can enhance the entertainment value by providing interactions that respond to the user's emotional state.

[1440] (Example 2)

[1441] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1442] The problems of lack of exercise and poor health among modern people are serious, and it has been pointed out that stairs are not used very often, especially in office buildings and commercial facilities. Furthermore, because using stairs is monotonous, many people tend to use elevators. As a result, physical activity decreases, which has a negative impact on health. This invention aims to alleviate lack of exercise and promote health by transforming the use of stairs into an entertaining experience, thereby encouraging users to use stairs voluntarily.

[1443] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1444] In this invention, the server includes a theme selection means for automatically selecting a weekly theme, a video data preparation means for preparing video data based on the theme, a video data distribution means for distributing video data from the server to terminals installed on each floor, a projection means for projecting images onto the stairs using a projector, a quest element setting means for setting quizzes or mini-games on specific staircases, a data collection means for capturing the user's facial expressions and voice using a camera and microphone, an analysis means for analyzing the collected data, and an emotion adjustment means for adjusting the projected image based on the user's emotion data. This makes it possible to transform the use of stairs into an entertaining experience, alleviate users' lack of exercise, and promote their health.

[1445] A "theme selection means" is a device or program for automatically selecting a different theme each week.

[1446] "Video data preparation means" refers to a device or program for producing or preparing video data based on a selected theme.

[1447] "Video data distribution means" refers to a device or program for distributing prepared video data from a server to a terminal.

[1448] "Projection means" refers to a device or program for projecting received video data onto a staircase using a projector device.

[1449] A "quest element setting means" is a device or program for setting quizzes or mini-games on specific staircases and providing them to the user.

[1450] "Data collection means" refers to a device or program that uses a camera and microphone to capture the user's facial expressions and voice, and collect that data.

[1451] "Analysis means" refers to a device or program that analyzes collected emotional data, motor data, response data, etc., to evaluate the user's state and behavior.

[1452] An "emotion adjustment device" is a device or program that adjusts projected images and sound effects based on the user's emotional data to optimize the entertainment experience.

[1453] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes, quest elements, and an emotion engine that recognizes the user's emotions, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[1454] Server Role

[1455] The server acts as the controller for the entire system and performs the following main tasks:

[1456] Theme Selection Methods

[1457] The server automatically selects a different theme each week. For example, it uses a generation AI model to select themes such as "space exploration" and "seabed exploration."

[1458] Video data preparation means

[1459] The server prepares video data based on the selected theme. This video data is generated to be suitable for the staircases on each floor. Specifically, it is prepared using video editing software and CG software.

[1460] Video data distribution method

[1461] The server distributes the prepared video data to terminals installed on each floor using the HTTP protocol.

[1462] Quest element setting method

[1463] The server sets up quizzes and mini-games on specific staircases and distributes the data to the terminals. The set quest elements encourage user interaction when using the stairs.

[1464] Data acquisition methods

[1465] The server acquires user facial expressions and voice data collected using cameras and microphones. This allows it to understand the user's emotional state.

[1466] analytical means

[1467] The server analyzes emotional data transmitted from the emotion engine and evaluates it in conjunction with the user's activity level and health status. This evaluation result is used as feedback data for the entire company.

[1468] emotional regulation means

[1469] The server adjusts the projected image based on the user's emotional data. For example, if the user is feeling happy, it adjusts the color and brightness of the image to enhance its entertainment value.

[1470] Terminal role

[1471] Terminals are installed on each floor and stairwell, and perform the following tasks based on data received from the server:

[1472] Video playback

[1473] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs. For example, it connects to the projector using an HDMI cable.

[1474] Display of quizzes and mini-games

[1475] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate. A special effect is added when the user answers correctly.

[1476] User emotion recognition

[1477] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine.

[1478] Interaction Processing

[1479] The device provides feedback based on the user's emotional state and quiz answers, and adjusts the projected images and sound effects accordingly.

[1480] User roles

[1481] Users can use this system to enjoy climbing stairs:

[1482] Use of stairs

[1483] Users climb the stairs while enjoying visual effects created by projection mapping.

[1484] Participation in quizzes and mini-games

[1485] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[1486] Emotion Recognition Interaction

[1487] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system.

[1488] Specific example

[1489] For example, if the theme for the following week is set to "Space Exploration," the server will prepare video data related to the "Space Exploration" theme and distribute it to terminals on each floor from the first to the tenth floor.

[1490] On the first floor, a terminal projects an image of a "spaceship" onto the staircase, and as the user ascends from the second floor, the scenery changes from a "starry sky" to "planets" to a "galaxy." Upon reaching the third floor, the user is presented with a quiz: "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if correct, a special effect (such as a star effect) is displayed on the staircase leading to the next floor.

[1491] Furthermore, because it incorporates an emotion engine, it recognizes emotions from the user's facial expressions and voice. For example, if the user shows a surprised expression, the device detects this and sends emotion data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[1492] Example of a prompt

[1493] This week's theme is "Space Exploration." Project different images on each floor as the user climbs the stairs from the 1st to the 10th floor. Display an image of a "spaceship" on the 1st floor, a "starry sky" on the 2nd floor, and a "planet" on the 3rd floor. Also, on the 3rd floor, present a quiz: "What is the distance from Earth to the Moon?" If the user answers correctly, display a special star effect on the stairs to the next floor. Capture the user's facial expressions and voice using the camera and microphone, and if an expression of surprise is detected, send that data to the server.

[1494] Through the above steps, the system takes user emotions into consideration, addresses lack of exercise, and enhances entertainment value.

[1495] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1496] Step 1: Theme Selection

[1497] The server uses a generative AI model to select a weekly theme. To do this, the server receives various data (user interests, seasonal events, etc.) as input and selects the theme for the following week (e.g., space exploration) as output.

[1498] Step 2: Preparing the video data

[1499] Based on the selected theme, the server prepares video data using video editing software and CG software. In this step, the server receives a theme (e.g., space exploration) as input and generates corresponding video data (e.g., spaceship, starry sky, planets) as output.

[1500] Step 3: Distribution of video data

[1501] The server delivers the prepared video data to the terminals using the HTTP protocol. In this step, the server receives video data (e.g., spaceship, starry sky, planets) as input and outputs a distribution request to send it to the terminals on each floor.

[1502] Step 4: Setting up quest elements

[1503] The server sets up quizzes and mini-games on specific staircases and distributes the data to the terminals. In this step, the server receives quest elements (e.g., What is the distance from Earth to the Moon?) as input and outputs a distribution request to send them to the target terminals.

[1504] Step 5: Play the video

[1505] The terminal sends the received video data to the projector device to perform projection mapping on the stairs. In this step, the terminal receives video data (e.g., spaceship, starry sky, planets) from the server as input and outputs a video transfer request to send to the projector.

[1506] Step 6: Displaying quizzes and mini-games

[1507] At specific points along the staircase, the terminal displays quizzes or mini-games, allowing the user to participate. In this step, the terminal receives quiz data from the server (e.g., "What is the distance from the Earth to the Moon?") as input and sends a display request to output it to the display. It also receives the user's answer and feeds it back into the next action.

[1508] Step 7: User emotion recognition

[1509] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and sends them to the emotion engine. In this step, the device receives data from sensors (e.g., facial expressions, voice) as input, captures it as emotion data, and outputs a data transfer request to send it to the emotion engine.

[1510] Step 8: Interaction Processing

[1511] The device provides feedback based on the user's emotional state and quiz results, adjusting the projected images and sound effects accordingly. In this step, the device receives emotional data and quiz results as input and outputs requests to adjust the images and sounds based on them. Specifically, if the user is feeling happy, the device adjusts the colors and brightness of the images.

[1512] Step 9: Data Collection and Analysis

[1513] The server analyzes emotional data sent from the emotion engine along with the user's exercise data and response data. In this step, the server receives emotional data, exercise data, and response data as input and outputs the analysis results. Specifically, this is used to evaluate the user's health status and entertainment experience.

[1514] Because each step involves a detailed description of specific actions, this system's program transforms using stairs into an entertaining experience, providing a means to alleviate users' lack of exercise and promote their health.

[1515] (Application Example 2)

[1516] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1517] This project aims to address the lack of exercise among stair users, promote their health, and make stair use enjoyable by enhancing its entertainment value. However, conventional methods have limited effectiveness in motivating stair users and improving the user experience, necessitating new technological solutions. Furthermore, technologies for appropriate emotion recognition and feedback based on those results are still underdeveloped. A system is needed to address these challenges.

[1518] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1519] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, an analysis means, a user recognition means, an emotion recognition means, and an emotion-based feedback means. This makes it possible to alleviate the user's lack of exercise, improve their health, and provide a dynamic entertainment experience based on emotions.

[1520] The "theme selection method" is a function that selects a different theme for each specified period and prepares the necessary video data and public relations information based on that theme.

[1521] The "video data preparation means" is a function that creates and edits video data to be used for projection mapping based on the selected theme.

[1522] "Video data distribution means" refers to a function that transmits prepared video data to a projection means.

[1523] "Projection means" refers to a function that displays received video data on stairs or walls using projection mapping technology.

[1524] The "quest element setting method" is a function that allows users to set up quizzes or mini-games at specific locations on stairs or floors, enabling them to participate in them.

[1525] "Data collection methods" refer to functions that collect user behavior data, quest participation data, and emotional data.

[1526] "Analysis tools" refer to functions that analyze collected data to evaluate the user's health status, exercise level, emotional state, and other factors.

[1527] "User recognition means" refers to a function that recognizes the user's face and body movements to identify the individual.

[1528] "Emotion recognition means" refers to a function that analyzes the user's facial expressions and voice to determine their emotional state.

[1529] "Emotion-based feedback mechanisms" are functions that adjust projection images and sound effects based on recognized emotions to provide appropriate feedback.

[1530] This invention is designed as a system that enhances the entertainment value of using stairs, alleviates lack of exercise, and promotes health. Specific embodiments using each component of this system are described below.

[1531] Server Role

[1532] The server acts as the controller for the entire system, performing the following key tasks.

[1533] 1. Theme Selection Methods

[1534] The server selects a different theme for each specified period. For example, themes might include "space exploration" or "underwater exploration," and based on these, it prepares the necessary video data and quizzes.

[1535] 2. Means for preparing video data

[1536] The server creates and edits video data to be used for projection mapping based on the selected theme. The video data is prepared to correspond to each floor of the staircase. For example, if the theme is "Space Exploration," the first floor will be set to "Earth," the second floor to "Moon," and the third floor to "Mars."

[1537] 3. Video data distribution method

[1538] The server transmits the prepared video data to projection terminals installed on each floor. The video data is distributed in real time via the network.

[1539] 4. Quest element setting means

[1540] The server sets up quizzes and mini-games at specific stairwells or floor locations and delivers them to projection terminals. For example, a quiz such as "What is the distance from the Earth to the Moon?" might be displayed at the third floor, and users can answer it using their smartphones.

[1541] 5. Data Collection Methods

[1542] The server collects user behavior data, quiz participation data, and emotional data. This allows the server to understand user activity levels and quest participation status.

[1543] 6. Analysis tools

[1544] The server analyzes the collected data to evaluate the user's health, activity level, emotional state, and other factors. This helps in providing more effective feedback and selecting future topics.

[1545] Terminal role

[1546] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server.

[1547] 1. Play back video

[1548] The terminal transmits the received video data to the projector device, which then performs projection mapping on the stairs.

[1549] 2. Display of quizzes and mini-games

[1550] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[1551] 3. User Recognition Means

[1552] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and recognizes the user.

[1553] 4. Emotion recognition means

[1554] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine. The emotion engine analyzes this data to recognize the user's emotional state.

[1555] 5. Emotion-based feedback methods

[1556] The device provides feedback based on the user's emotional state and quiz answers, adjusting the projected images and sound effects accordingly. For example, if the user is enjoying themselves, the images become more vibrant and the sound effects are enhanced.

[1557] User roles

[1558] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[1559] 1. Use of stairs

[1560] Users climb the stairs while enjoying visual effects created by projection mapping.

[1561] 2. Participation in quizzes and mini-games

[1562] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[1563] 3. Emotion Recognition Interaction

[1564] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system. For example, if the user shows a surprised expression, the device detects this and sends emotional data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[1565] Hardware and software to be used

[1566] Hardware: Smartphones (e.g., iPhone, Google Pixel), projector devices, sensor terminals (cameras, microphones, etc.)

[1567] Software: EmotionRecognition library (for emotion recognition), OpenCV library (for camera input and video processing), server-side web frameworks such as Flask and Django.

[1568] Specific example

[1569] For example, if the theme is set to "Space Exploration," the server prepares video data related to the "Space Exploration" theme and distributes it to terminals on each floor from the 1st to the 10th floor. The terminal on the 1st floor projects an image of a "spaceship" onto the staircase, and as the user goes up from the 2nd floor, the scenery changes from a "starry sky" to "planets" to "galaxies." When the user reaches the 3rd floor, a quiz is displayed asking, "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if they answer correctly, a special effect (for example, a star effect) is displayed on the staircase to the next floor.

[1570] Example of a prompt

[1571] Example of quiz generation: "To the generation AI model: Please create 3 quiz questions related to space exploration."

[1572] Example of emotion data analysis: "To the generative AI model: Please suggest suitable entertainment content when the emotional state is happy."

[1573] The above describes specific embodiments for carrying out this invention.

[1574] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1575] Step 1:

[1576] The server selects the theme.

[1577] The server selects a different theme for each specified period, prompting the AI ​​model with the message, "Please suggest a theme for next week," and then receives the generated theme. This theme information is used for the following processes.

[1578] Input: Theme selection prompt message

[1579] Data processing / manipulation: Themes are generated using a generative AI model.

[1580] Output: Selected theme (e.g., "Space Exploration")

[1581] Step 2:

[1582] The server prepares the video data.

[1583] The server creates video data for projection mapping using video data preparation means, based on the selected theme. It collects the necessary video materials and edits them to match the theme.

[1584] Input: Selected theme

[1585] Data processing / editing: Creation and editing of video data

[1586] Output: Prepared video data

[1587] Step 3:

[1588] The server distributes the video data.

[1589] The server distributes the prepared video data to terminals installed on each floor via the network.

[1590] Input: Prepared video data

[1591] Data processing / processing: Video data distribution

[1592] Output: Supply of video data to the terminal

[1593] Step 4:

[1594] The device plays the video.

[1595] The terminal transmits the received video data to a projector device, performing projection mapping on designated staircases or walls. The video is displayed according to the selected theme.

[1596] Input: Video data from the server

[1597] Data processing / editing: Video display using projection mapping

[1598] Output: Video displays on stairs and walls

[1599] Step 5:

[1600] The server sets the quest elements.

[1601] The server requests a generation AI model to create quizzes and mini-games related to the theme. For example, it might input a prompt like, "Create three quiz questions related to space exploration," and then receive the generated quest content.

[1602] Input: Quest generation prompt text

[1603] Data processing / manipulation: Generating quest elements using a generative AI model.

[1604] Output: Generated quest content (e.g., "What is the distance from Earth to the Moon?")

[1605] Step 6:

[1606] The server will deliver quest elements.

[1607] The server distributes the generated quest details to terminals installed on specific staircases.

[1608] Input: Generated quest content

[1609] Data processing / editing: Distribution of quest content

[1610] Output: Supplying quest content to the terminal

[1611] Step 7:

[1612] The device displays a quiz / mini-game.

[1613] The terminal displays received quest content on a projector device or terminal screen, allowing users to participate. Users answer quizzes and mini-games in real time.

[1614] Input: Quest details from the server

[1615] Data processing / manipulation: Displaying quizzes and mini-games

[1616] Output: User participation in quizzes and mini-games

[1617] Step 8:

[1618] The device recognizes the user's emotions.

[1619] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and analyzes them using emotion recognition technology.

[1620] Input: User's facial expressions, voice, and motion data

[1621] Data processing / analysis: Recognition and analysis of emotional states

[1622] Output: Recognized emotion data

[1623] Step 9:

[1624] The device sends emotional data to the server.

[1625] The device sends the recognized emotion data to the server. The server receives this data and uses it for the next processing step.

[1626] Input: Recognized emotion data

[1627] Data processing / processing: Sending emotional data

[1628] Output: Supplying emotion data to the server

[1629] Step 10:

[1630] The server provides emotion-based feedback.

[1631] The server analyzes the received emotional data and adjusts the next projection images and sound effects based on the emotions. It inputs a prompt message to the generating AI model, "Please suggest suitable entertainment content when the emotional state is happy," and sends the generated content to the terminal.

[1632] Input: Sentiment data, entertainment content suggestion prompt text

[1633] Data Processing / Editing: Analysis and generation of emotional data, content proposals using AI models.

[1634] Output: Adjusted entertainment content

[1635] Step 11:

[1636] The device displays the feedback content.

[1637] The device plays back the adjusted projection images and sound effects received from the server and provides feedback to the user. This is designed to create an entertainment experience that responds to the user's emotional state.

[1638] Input: Coordinated entertainment content from the server

[1639] Data processing / editing: Playback of projection images and sound effects.

[1640] Output: Providing feedback to users

[1641] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1642] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1643] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1644] [Fourth Embodiment]

[1645] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1646] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1647] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1648] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1649] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1650] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1651] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1652] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1653] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1654] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1655] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1656] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1657] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1658] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes and quest elements, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[1659] Server Role

[1660] The server acts as the controller for the entire system and performs the following main tasks:

[1661] 1. Theme selection:

[1662] The server selects a different theme each week (for example, mountain climbing, underwater, space, etc.) and prepares video data for it.

[1663] 2. Preparation and distribution of video data:

[1664] Based on the selected theme, the server prepares the video data and distributes it to the projection terminals installed on each floor.

[1665] 3. Setting Quest Elements:

[1666] The server sets up quizzes and mini-games on specific staircases and delivers them to the terminals.

[1667] 4. Data collection and analysis:

[1668] The server receives data collected from each terminal and analyzes the user's activity level and health status.

[1669] Terminal role

[1670] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server:

[1671] 1. Play back the video:

[1672] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs.

[1673] 2. Display of quizzes and mini-games:

[1674] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[1675] 3. User interaction handling:

[1676] The terminal receives and processes the user's quiz answers and mini-game results, and sends that data to the server.

[1677] User roles

[1678] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[1679] 1. Use of stairs:

[1680] Users climb the stairs while enjoying visual effects created by projection mapping.

[1681] 2. Participation in quizzes and mini-games:

[1682] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[1683] Specific example

[1684] For example, if the theme for the following week is set to "mountain climbing," the server will prepare video data related to the "mountain climbing" theme and distribute it to terminals on each floor from the 1st to the 10th floor.

[1685] On the first floor, a terminal projects an image of "the foot of a mountain" onto the staircase, and as you go up from the second floor, the scenery changes from "mountain path" to "valley" to "steep rocky area" to "snowy mountain." When a user reaches the third floor, a quiz is displayed asking, "Which of the following mountains is over 1,000 meters high?" If the user selects "Mount Fuji" from the options and answers correctly, a special effect (such as fireworks or special visual effects) is displayed on the staircase to the next floor.

[1686] The server collects data from terminals on each floor and analyzes user response data and stair usage data to evaluate exercise levels and health status. This data is used for user health improvement programs and as feedback for the entire company.

[1687] The following describes the processing flow.

[1688] Step 1:

[1689] Theme selection (server)

[1690] The server checks the current week and selects a new theme. For example, the theme for next week might be set to "Space Exploration".

[1691] Step 2:

[1692] Preparation of video data (server)

[1693] The server collects and prepares relevant video data based on the selected theme. For example, it prepares a video dataset for space exploration (spaceships, starry skies, planets, etc.).

[1694] Step 3:

[1695] Video data distribution (server)

[1696] The server sequentially distributes the prepared video data to the projection terminals on each floor. For example, it sends video data from the first floor to the tenth floor.

[1697] Step 4:

[1698] Receiving and saving video data (on the device)

[1699] The device receives video data transmitted from the server and saves it to local storage. Once saving is complete, it begins preparing for playback.

[1700] Step 5:

[1701] Projection Start (Device)

[1702] The device activates the projector at a specified time and projects images onto the walls and floor of the staircase. For example, an image of a spaceship is displayed on the staircase on the first floor.

[1703] Step 6:

[1704] Quest element settings (server)

[1705] The server prepares quiz or mini-game data for specific staircases and sends it to the terminals. For example, the 3rd floor might have a quiz like, "What is the distance from the Earth to the Moon?"

[1706] Step 7:

[1707] Quest data reception and display preparation (device)

[1708] The terminal receives data for quizzes and mini-games and prepares to display it when the user reaches the desired location. It verifies the validity of the data and prepares the interactive screen.

[1709] Step 8:

[1710] User interaction (user)

[1711] The user climbs the stairs while enjoying projection mapping. A quiz is displayed when they reach a specific floor (for example, the 3rd floor). The user then answers the quiz.

[1712] Step 9:

[1713] Receiving and evaluating responses (device)

[1714] The device receives the user's answer and determines whether it is correct or incorrect. If correct, a special effect is added to the video leading up to the next floor. For example, a special star effect is displayed when the answer is correct.

[1715] Step 10:

[1716] Data collection (device)

[1717] The terminal collects user response data and stair usage data. The collected data is then prepared for transmission to the server.

[1718] Step 11:

[1719] Data transmission (device)

[1720] The device sends the collected data to the server. This data may include, for example, each user's quiz answers and stair usage time.

[1721] Step 12:

[1722] Data reception and analysis (server)

[1723] The server receives data sent from the terminal and stores it in a database. It then begins analysis to evaluate exercise levels and health status.

[1724] Step 13:

[1725] Feedback of results (server)

[1726] The server generates a report based on the analysis results and notifies administrators and relevant parties. It reports on each user's health status and activity levels over time.

[1727] Through the steps described above, the system aims to make using stairs enjoyable for users, alleviate a lack of exercise, and promote health.

[1728] (Example 1)

[1729] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1730] In modern office buildings and commercial facilities, users' reliance on elevators leads to insufficient daily exercise and a decline in health. Furthermore, using stairs is psychologically monotonous, causing users to avoid it. Additionally, there is a lack of systems to efficiently monitor and improve health conditions and exercise levels.

[1731] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1732] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a quest element setting means, a data collection means, an analysis means, and a user interaction processing means. This makes using stairs enjoyable, helps alleviate lack of exercise, and promotes health. Furthermore, it provides psychological enjoyment by offering quizzes and mini-games, and enables efficient collection and analysis of each user's exercise data.

[1733] "Theme selection method" refers to the process of selecting a theme from the library that is appropriate for the year or season, and deciding on it as the theme for the following week.

[1734] "Video data preparation means" refers to a means for generating relevant video data based on a selected theme and preparing it for projection on each floor.

[1735] "Video data distribution means" refers to a means for distributing generated video data to terminals on each floor.

[1736] "Projection means" refers to a method for transmitting video data received by a terminal to a projector device and projecting the image onto the stairs.

[1737] A "quest element setting method" is a means of placing interactive elements such as quizzes or mini-games on a specific staircase and presenting them to the user.

[1738] "Data collection means" refers to the means of collecting user interaction and stair usage data from each terminal.

[1739] "Analysis methods" refer to methods for analyzing each user's exercise volume and quiz accuracy rate based on collected data, and generating reports to address lack of exercise and promote health.

[1740] "User interaction processing means" refers to a means for receiving and processing input data when a user answers a quiz or participates in a mini-game, and sending the results to a server.

[1741] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes and quest elements, transforming the use of stairs into an enjoyable experience, thereby addressing users' lack of exercise and promoting their health. The system as a whole is mainly divided into three elements: server, terminal, and user.

[1742] Server Role

[1743] The server acts as the controller for the entire system and performs the following main tasks:

[1744] Theme selection method:

[1745] The server's role is to select the theme for the following week from a theme library that is appropriate for the year and season. For example, it might select one theme from options such as "mountain climbing," "underwater," or "space."

[1746] Methods for preparing video data:

[1747] After a theme is selected, the server retrieves relevant video data from cloud storage based on the selected theme. It then generates video data suitable for each floor. For example, for a "mountain climbing" theme, it generates video from the "foot of the mountain" on the 1st floor to the "summit" on the 10th floor.

[1748] Video data distribution method:

[1749] The generated video data is distributed from the server to projection terminals on each floor. This makes it possible to project theme-based visual effects onto each staircase.

[1750] Quest element setting method:

[1751] The server is responsible for placing interactive elements such as quizzes and mini-games on specific staircases. For example, on the third floor, a quiz might be set up asking, "Which of the following mountains is over 1,000 meters high?"

[1752] Data acquisition methods:

[1753] The server collects user quiz answers and stair usage data from each terminal in real time. This allows for monitoring of user behavior.

[1754] Analysis means:

[1755] The server analyzes each user's exercise level and quiz accuracy rate based on the collected data. The analysis results are generated as reports to help address lack of exercise and promote health.

[1756] User interaction processing means:

[1757] The system receives input data from users when they answer quizzes or participate in mini-games, and sends the results to the server.

[1758] Terminal role

[1759] Terminals are installed on each floor and stairwell, and perform the following tasks:

[1760] Play video:

[1761] The terminal receives video data streamed from the server and transmits that data to a projector device to project the image onto the staircase. For example, the terminal on the first floor projects an image showing "the foot of the mountain."

[1762] Display of quizzes and mini-games:

[1763] The device receives quest content delivered from the server and displays quizzes or mini-games to users who reach specific locations on the stairs.

[1764] User interaction handling:

[1765] The terminal receives the user's quiz answers and mini-game participation data, and sends the results to the server.

[1766] User roles

[1767] The user performs the following tasks:

[1768] Use of stairs:

[1769] Users climb the stairs while enjoying visual effects created by projection mapping. By experiencing various themed landscapes, they can also get some exercise.

[1770] Participation in quizzes and mini-games:

[1771] When users reach a specific location on a floor, they can participate in quizzes or mini-games displayed by a projector. For example, on the third floor, they might answer a quiz asking, "Which of the following mountains is over 1,000 meters high?"

[1772] Specific examples and prompt statements

[1773] Specific example:

[1774] If the theme for the following week is set to "mountain climbing," the server prepares video data related to the "mountain climbing" theme and distributes it to terminals on each floor from the 1st to the 10th floor. The terminal on the 1st floor projects an image of "the foot of a mountain" onto the stairs, and as the user goes up from the 2nd floor, the scenery changes from "mountain path" → "valley" → "steep rocky area" → "snowy mountain." When the user reaches the 3rd floor, a quiz is displayed asking "Which of the following mountains has an altitude of over 1,000 meters?", and the user selects "Mount Fuji" from the options.

[1775] Examples of prompts for a generative AI model:

[1776] Please describe a system where projection mapping is used on the staircase from the 1st to the 10th floor, with the theme being "mountain climbing." Provide specific examples of scenery related to the theme of each floor, and detail how quizzes and mini-games are conducted on each floor.

[1777] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1778] Server processing steps

[1779] Step 1: Theme Selection

[1780] Input: Theme data from the previous week, year and season information, theme library

[1781] Processing: The server selects the theme for the following week from the theme library based on the year and season. Specifically, it scans the list of themes in the library and runs an algorithm that evaluates how well each theme fits the given conditions.

[1782] Output: Selected theme (e.g., "Mountain climbing", "Underwater", "Space")

[1783] Specific action: The server sets the theme for the following week to "mountain climbing".

[1784] Step 2: Prepare video data

[1785] Input: Selected theme (e.g., "Mountain climbing")

[1786] Processing: The server retrieves theme-related video data from cloud storage. Then, it generates video data appropriate for each floor. Specifically, for the "mountain climbing" theme, it runs an image processing algorithm to generate scenery data for floors 1 through 10.

[1787] Output: Video data corresponding to each floor (e.g., from the "foot of the mountain" on the 1st floor to the "summit" on the 10th floor)

[1788] Specific operation: The server generates video of the "foot of the mountain" for the first floor, and then generates video of the "mountain path" for the second floor.

[1789] Step 3: Video data distribution

[1790] Input: Generated video data

[1791] Processing: The server distributes video data to terminals on each floor. Specifically, it uses a protocol that transmits data to terminals via a network connection.

[1792] Output: Video data has been distributed to each terminal.

[1793] Specific operation: The server distributes the corresponding video data to each terminal from the 1st floor to the 10th floor.

[1794] Step 4: Setting Quest Elements

[1795] Input: Database of selected themes, quizzes, and mini-games

[1796] Processing: The server retrieves quizzes and mini-games related to the theme from the database and places them on specific floors. It then distributes them to projection terminals.

[1797] Output: Quest elements placed on each staircase

[1798] Specific operation: The server sets a quiz as a quest on the 3rd floor: "Which of the following mountains has an elevation of over 1,000 meters?" and delivers it to the terminal on the 3rd floor.

[1799] Step 5: Data Acquisition

[1800] Input: User quiz answer data, stair usage data

[1801] Processing: The server receives data collected from each terminal in real time. This includes the process of collecting data using network communication protocols.

[1802] Output: Collected data (e.g., user response data, usage data)

[1803] Specific operation: The server receives the user's quiz answer, "Mount Fuji," in real time from a terminal on the third floor.

[1804] Step 6: Data Analysis

[1805] Input: Collected data

[1806] Processing: The server analyzes each user's activity level and quiz accuracy rate. Specifically, it processes each user's data through a statistical analysis algorithm and generates a report.

[1807] Output: Report of analysis results

[1808] Specific operation: The server records and analyzes that user A walked 1,000 steps and achieved an 85% correct answer rate on the quiz.

[1809] Terminal processing steps

[1810] Step 1: Play the video

[1811] Input: Video data streamed from the server

[1812] Processing: The terminal verifies the received video data and sends it to the projector device. This includes a procedure to check that the data has been transferred correctly.

[1813] Output: Image projected onto the stairs

[1814] Specific operation: The terminal projects an image of "the foot of the mountain" onto the first floor.

[1815] Step 2: Display the quiz / mini-game

[1816] Input: Quest content delivered from the server

[1817] Processing: The terminal checks the received quest content and displays it when the user reaches a specific floor location.

[1818] Output: Quizzes and mini-games displayed on the stairs

[1819] Specific action: The terminal displays a quiz to the user upon reaching the 3rd floor: "Which of the following mountains has an elevation of over 1,000 meters?"

[1820] Step 3: Handling User Interactions

[1821] Input: User's quiz answers, mini-game results

[1822] Processing: The terminal receives user input data, processes it immediately, and sends the results to the server.

[1823] Output: User input data sent to the server

[1824] Specific operation: The terminal sends the result of the user selecting "Mt. Fuji" on the 3rd floor to the server.

[1825] User processing steps

[1826] Step 1: Use the stairs

[1827] Input: Projection-mapped stairs

[1828] Processing: Users enjoy the visual effects of the stairs as they climb, which can increase their physical activity.

[1829] Output: Exercise data of users using stairs

[1830] Specific action: The user climbs the stairs from the first floor to the second floor.

[1831] Step 2: Participate in quizzes and mini-games

[1832] Input: Quizzes and mini-games displayed on the stairs

[1833] Processing: Users participate in quizzes and mini-games displayed on specific floors, and perform actions such as answering questions.

[1834] Output: Quiz answers, mini-game results

[1835] Specific action: On the third floor, the user selects "Mount Fuji" in response to the quiz, "Which of the following mountains has an elevation of over 1,000 meters?"

[1836] (Application Example 1)

[1837] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1838] In modern society, opportunities for daily exercise have decreased due to the presence of office buildings and commercial facilities, leading to a serious problem of lack of physical activity. Furthermore, while fitness and health promotion initiatives continue, many people lack the motivation to maintain them. Traditional methods struggle to balance health promotion with enjoyment, and opportunities for exercise during travel, particularly when using food delivery services, are limited.

[1839] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1840] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, an analysis means, a virtual environment display means, an exercise path generation means, and an exercise data evaluation means. This allows users to enjoy healthy activities through projection mapping while waiting for food delivery or while delivery personnel are on their way. By participating in quizzes and mini-games at specific points, users are provided with an experience that increases their exercise level, evaluates their health status, and provides feedback.

[1841] A "theme selection method" refers to a device or program for selecting a weekly theme and preparing video data based on that theme.

[1842] "Video data preparation means" refers to devices or programs for generating and preparing video data related to a selected theme.

[1843] "Video data distribution means" refers to devices or programs for transmitting prepared video data to each terminal.

[1844] "Projection means" refers to devices or programs that use received video data to project images onto real-world objects or spaces.

[1845] A "quest element setting means" is a device or program for setting interactive elements such as quizzes and mini-games and placing them at specific points.

[1846] "Data collection means" refers to devices or programs used to collect user interaction and behavioral data.

[1847] "Analysis tools" refer to devices or programs that analyze collected data and evaluate the user's exercise level and health status.

[1848] A "virtual environment display means" refers to a device or program that uses a smartphone or smart glasses to display a virtual environment and enrich the user experience.

[1849] A "movement path generation means" is a device or program for setting a movement path for a user or delivery person and placing projection mapping or quizzes along that path.

[1850] "Exercise data evaluation means" refers to a device or program for evaluating a user's exercise data and providing feedback on the results.

[1851] This invention is a system that uses projection mapping technology to help users enjoy themselves while using food delivery services, thereby combating a sedentary lifestyle and promoting their health. The system consists of the following main components:

[1852] Server Role

[1853] The server acts as the central controller for the entire system. The server performs the following main tasks:

[1854] 1. Theme Selection Method: The server selects a different theme each week (e.g., space travel, underwater exploration, etc.).

[1855] 2. Video data preparation means: Based on the selected theme, the server generates and prepares the relevant video data.

[1856] 3. Video data distribution method: The prepared video data is sent to each terminal.

[1857] 4. Quest element setting method: Quizzes and mini-games are set on specific walking routes or staircases, and the data is distributed to the terminal.

[1858] 5. Data collection methods: User interaction data and behavioral data will be collected from each terminal.

[1859] 6. Analysis methods: The collected data is analyzed to evaluate the user's exercise level and health status.

[1860] 7. Virtual environment display means: A virtual environment is displayed on a smartphone, smart glasses, etc.

[1861] 8. Movement path generation means: Sets the movement path of the user or delivery person and places projection mapping along that path.

[1862] 9. Exercise data evaluation means: Evaluate the user's exercise data and provide feedback on the results.

[1863] Terminal role

[1864] The terminal will be installed on the smartphones or smart glasses of delivery personnel and users. The terminal will perform the following tasks:

[1865] 1. Virtual environment display means: Based on video data received from the server, projection mapping is performed onto real-world stairs and walking paths.

[1866] 2. Displaying quest elements: Display quizzes or mini-games at specific points to encourage user interaction.

[1867] 3. User interaction processing: Receive and process user quiz answers and mini-game results, and send them to the server.

[1868] User roles

[1869] Users can use this system to make good use of their waiting time for food delivery while enjoying fitness. The following tasks will be performed:

[1870] 1. Use of food delivery: Users order food delivery through the app.

[1871] 2. Projection Mapping Experience: While waiting for the delivery person to arrive, users can enjoy projection mapping while exercising.

[1872] 3. Participate in quizzes and mini-games: Participate in quizzes and mini-games at specific points and enjoy special effects depending on the results.

[1873] Specific example

[1874] For example, if a user launches a fitness app and orders a pizza, the delivery person's route is given a "space travel" theme. The user then jogs or walks while experiencing projection mapping using their smartphone or smart glasses. At certain points, a quiz appears asking, "How many times larger is Mars than Earth?" If the user answers correctly, a special effect occurs, boosting their motivation to exercise.

[1875] Example of a prompt

[1876] Please provide specific ideas for quizzes and mini-games that can be applied to health promotion applications using projection mapping. For example, quizzes and mini-games themed around "space travel" are possible.

[1877] This system is a new food delivery method that encourages users to exercise healthily and provides an enjoyable experience.

[1878] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1879] Step 1:

[1880] The server selects a theme each week using a theme selection mechanism. The input is index information including the start date of the week, and the output is the selected theme. Examples of themes include "space travel" and "underwater exploration." The server determines the theme based on the previous week's theme and user feedback.

[1881] Step 2:

[1882] The server uses a video data preparation mechanism to generate video data related to the selected theme. The input is the selected theme, and the output is video data corresponding to that theme. For example, if the theme is "space travel," video footage of outer space and planets will be generated. This video data will be delivered in a later step.

[1883] Step 3:

[1884] The server transmits the prepared video data to each terminal using a video data distribution method. The input is the video data, and the output is a delivery success message to each terminal. The video data is transmitted from the server to the terminals via the network.

[1885] Step 4:

[1886] The device projects received video data onto real-world stairs or walking paths using projection technology. The input is video data, and the output is the projected image. The user views this projected image through a smartphone or smart glasses.

[1887] Step 5:

[1888] The server uses a quest element setting mechanism to configure quizzes and mini-games and distributes the data to the terminal. The input is the configuration information, and the output is a message indicating successful distribution to the terminal. For example, if the theme is "Space Travel," a quiz such as "How many times larger is Mars than Earth?" will be set.

[1889] Step 6:

[1890] The device displays received quizzes or mini-games to the user when they reach a specific point. The input is the received quiz data, and the output is the quiz screen. When the user answers the quiz, the result is sent to the device.

[1891] Step 7:

[1892] The terminal sends the user's answer to the server. The input is the user's answer data, and the output is a message indicating successful transmission to the server. The answer data is transmitted from the terminal to the server via the network.

[1893] Step 8:

[1894] The server collects data transmitted from each terminal using data collection methods. The input consists of user response data and behavioral data, while the output is the collected data. The server collects user response data and analyzes it in the next step.

[1895] Step 9:

[1896] The server analyzes the collected data using analytical tools to evaluate the user's exercise level and health status. The input is the collected data, and the output is the analysis results. Based on the analysis results, feedback on health status and exercise advice are generated.

[1897] Step 10:

[1898] The server uses an exercise data evaluation system to provide the user with analysis results. The input is the analysis results, and the output is a feedback message. The feedback message is notified to the user through the app, providing information that can help improve their health.

[1899] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1900] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes, quest elements, and an emotion engine that recognizes the user's emotions, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[1901] Server Role

[1902] The server acts as the controller for the entire system and performs the following main tasks:

[1903] 1. Theme selection:

[1904] The server selects a different theme each week (for example, mountain climbing, underwater, space, etc.) and prepares video data for it.

[1905] 2. Preparation and distribution of video data:

[1906] Based on the selected theme, the server prepares the video data and distributes it to the projection terminals installed on each floor.

[1907] 3. Setting Quest Elements:

[1908] The server sets up quizzes and mini-games on specific staircases and delivers them to the terminals.

[1909] 4. Data collection and analysis:

[1910] The server receives emotional data transmitted from the emotion engine and analyzes it in conjunction with the user's activity level and health status.

[1911] Terminal role

[1912] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server:

[1913] 1. Play back the video:

[1914] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs.

[1915] 2. Display of quizzes and mini-games:

[1916] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[1917] 3. User emotion recognition:

[1918] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine. The emotion engine analyzes this data to recognize the user's emotional state.

[1919] 4. Interaction handling:

[1920] The device provides feedback based on the user's emotional state and quiz answers, and adjusts the projected images and sound effects accordingly.

[1921] User roles

[1922] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[1923] 1. Use of stairs:

[1924] Users climb the stairs while enjoying visual effects created by projection mapping.

[1925] 2. Participation in quizzes and mini-games:

[1926] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[1927] 3. Emotion Recognition Interaction:

[1928] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system. For example, if the user is feeling happy, the projected image becomes more vivid, enhancing the entertainment value.

[1929] Specific example

[1930] For example, if the theme for the following week is set to "Space Exploration," the server will prepare video data related to the "Space Exploration" theme and distribute it to terminals on each floor from the first to the tenth floor.

[1931] On the first floor, a terminal projects an image of a "spaceship" onto the staircase, and as the user ascends from the second floor, the scenery changes from a "starry sky" to "planets" to a "galaxy." Upon reaching the third floor, the user is presented with a quiz: "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if correct, a special effect (such as a star effect) is displayed on the staircase leading to the next floor.

[1932] Furthermore, because it incorporates an emotion engine, it recognizes emotions from the user's facial expressions and voice. For example, if the user shows a surprised expression, the device detects this and sends emotion data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[1933] The server collects data from terminals on each floor and analyzes user response data, emotional data, and stair usage data to evaluate exercise levels and health status. This data is used for user health improvement programs and as feedback for the entire company.

[1934] As described above, this invention provides a stair-use promotion system that also takes into account the user's emotions, thereby alleviating lack of exercise, promoting health, and simultaneously enhancing entertainment value.

[1935] The following describes the processing flow.

[1936] Step 1:

[1937] Theme selection (server)

[1938] The server checks the current week and selects a new theme. For example, the theme for next week might be set to "Space Exploration".

[1939] Step 2:

[1940] Preparation of video data (server)

[1941] The server collects and prepares relevant video data based on the selected theme. For example, it prepares a video dataset for space exploration (spaceships, starry skies, planets, etc.).

[1942] Step 3:

[1943] Video data distribution (server)

[1944] The server sequentially distributes the prepared video data to the projection terminals on each floor. For example, it sends video data from the first floor to the tenth floor.

[1945] Step 4:

[1946] Receiving and saving video data (on the device)

[1947] The device receives video data transmitted from the server and saves it to local storage. Once saving is complete, it begins preparing for playback.

[1948] Step 5:

[1949] Projection Start (Device)

[1950] The device activates the projector at a specified time and projects images onto the walls and floor of the staircase. For example, an image of a spaceship is displayed on the staircase on the first floor.

[1951] Step 6:

[1952] Quest element settings (server)

[1953] The server prepares quiz or mini-game data for specific staircases and sends it to the terminals. For example, the 3rd floor might have a quiz like, "What is the distance from the Earth to the Moon?"

[1954] Step 7:

[1955] Quest data reception and display preparation (device)

[1956] The terminal receives data for quizzes and mini-games and prepares to display it when the user reaches the desired location. It verifies the validity of the data and prepares the interactive screen.

[1957] Step 8:

[1958] Emotional recognition (emotional engine)

[1959] Sensors such as cameras and microphones installed on the device capture the user's facial expressions, voice, and body movements, and transmit this data to the emotion engine. The emotion engine analyzes this data to recognize the user's emotions.

[1960] Step 9:

[1961] User interaction (user)

[1962] The user climbs the stairs while enjoying projection mapping. A quiz is displayed when they reach a specific floor (for example, the 3rd floor). The user answers the quiz. Additionally, an emotion recognition system analyzes the user's emotional state.

[1963] Step 10:

[1964] Receiving and evaluating responses (device)

[1965] The device receives the user's answer and determines whether it is correct or incorrect. If correct, a special effect is added to the video leading up to the next floor. For example, a special star effect is displayed when the answer is correct. At the same time, it also receives emotional data from the emotion engine.

[1966] Step 11:

[1967] Emotion-based feedback (device)

[1968] The device adjusts projected images and sound effects based on emotional data obtained from the emotion engine. For example, if it recognizes that the user is enjoying themselves, it makes the images more vibrant and the music more upbeat.

[1969] Step 12:

[1970] Data collection (device)

[1971] The device collects user response data, sentiment data, and stair usage data. It then prepares the collected data for transmission to the server.

[1972] Step 13:

[1973] Data transmission (device)

[1974] The device sends the collected data to the server. This data may include each user's quiz answers, emotional state, and stair climbing time.

[1975] Step 14:

[1976] Data reception and analysis (server)

[1977] The server receives data sent from the terminal and stores it in a database. It then begins analysis to assess exercise levels and health status.

[1978] Step 15:

[1979] Feedback of results (server)

[1980] The server generates a report based on the analysis results and notifies administrators and relevant parties. It reports on each user's health status, activity levels, and emotional fluctuations.

[1981] Through the steps outlined above, the system aims to encourage users to use stairs in an enjoyable way, alleviate lack of exercise, and promote health. Furthermore, it can enhance the entertainment value by providing interactions that respond to the user's emotional state.

[1982] (Example 2)

[1983] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1984] The problems of lack of exercise and poor health among modern people are serious, and it has been pointed out that stairs are not used very often, especially in office buildings and commercial facilities. Furthermore, because using stairs is monotonous, many people tend to use elevators. As a result, physical activity decreases, which has a negative impact on health. This invention aims to alleviate lack of exercise and promote health by transforming the use of stairs into an entertaining experience, thereby encouraging users to use stairs voluntarily.

[1985] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1986] In this invention, the server includes a theme selection means for automatically selecting a weekly theme, a video data preparation means for preparing video data based on the theme, a video data distribution means for distributing video data from the server to terminals installed on each floor, a projection means for projecting images onto the stairs using a projector, a quest element setting means for setting quizzes or mini-games on specific staircases, a data collection means for capturing the user's facial expressions and voice using a camera and microphone, an analysis means for analyzing the collected data, and an emotion adjustment means for adjusting the projected image based on the user's emotion data. This makes it possible to transform the use of stairs into an entertaining experience, alleviate users' lack of exercise, and promote their health.

[1987] A "theme selection means" is a device or program for automatically selecting a different theme each week.

[1988] "Video data preparation means" refers to a device or program for producing or preparing video data based on a selected theme.

[1989] "Video data distribution means" refers to a device or program for distributing prepared video data from a server to a terminal.

[1990] "Projection means" refers to a device or program for projecting received video data onto a staircase using a projector device.

[1991] A "quest element setting means" is a device or program for setting quizzes or mini-games on specific staircases and providing them to the user.

[1992] "Data collection means" refers to a device or program that uses a camera and microphone to capture the user's facial expressions and voice, and collect that data.

[1993] "Analysis means" refers to a device or program that analyzes collected emotional data, motor data, response data, etc., to evaluate the user's state and behavior.

[1994] An "emotion adjustment device" is a device or program that adjusts projected images and sound effects based on the user's emotional data to optimize the entertainment experience.

[1995] This invention is a system that uses projection mapping technology on stairwells in office buildings and commercial facilities to provide weekly changing themes, quest elements, and an emotion engine that recognizes the user's emotions, thereby transforming stair use into an enjoyable experience, addressing users' lack of exercise, and promoting their health.

[1996] Server Role

[1997] The server acts as the controller for the entire system and performs the following main tasks:

[1998] Theme Selection Methods

[1999] The server automatically selects a different theme each week. For example, it uses a generation AI model to select themes such as "space exploration" and "seabed exploration."

[2000] Video data preparation means

[2001] The server prepares video data based on the selected theme. This video data is generated to be suitable for the staircases on each floor. Specifically, it is prepared using video editing software and CG software.

[2002] Video data distribution method

[2003] The server distributes the prepared video data to terminals installed on each floor using the HTTP protocol.

[2004] Quest element setting method

[2005] The server sets up quizzes and mini-games on specific staircases and distributes the data to the terminals. The set quest elements encourage user interaction when using the stairs.

[2006] Data acquisition methods

[2007] The server acquires user facial expressions and voice data collected using cameras and microphones. This allows it to understand the user's emotional state.

[2008] analytical means

[2009] The server analyzes emotional data transmitted from the emotion engine and evaluates it in conjunction with the user's activity level and health status. This evaluation result is used as feedback data for the entire company.

[2010] emotional regulation means

[2011] The server adjusts the projected image based on the user's emotional data. For example, if the user is feeling happy, it adjusts the color and brightness of the image to enhance its entertainment value.

[2012] Terminal role

[2013] Terminals are installed on each floor and stairwell, and perform the following tasks based on data received from the server:

[2014] Video playback

[2015] The terminal transmits the received video data to the projector device, performing projection mapping on the stairs. For example, it connects to the projector using an HDMI cable.

[2016] Display of quizzes and mini-games

[2017] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate. A special effect is added when the user answers correctly.

[2018] User emotion recognition

[2019] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine.

[2020] Interaction Processing

[2021] The device provides feedback based on the user's emotional state and quiz answers, and adjusts the projected images and sound effects accordingly.

[2022] User roles

[2023] Users can use this system to enjoy climbing stairs:

[2024] Use of stairs

[2025] Users climb the stairs while enjoying visual effects created by projection mapping.

[2026] Participation in quizzes and mini-games

[2027] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[2028] Emotion Recognition Interaction

[2029] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system.

[2030] Specific example

[2031] For example, if the theme for the following week is set to "Space Exploration," the server will prepare video data related to the "Space Exploration" theme and distribute it to terminals on each floor from the first to the tenth floor.

[2032] On the first floor, a terminal projects an image of a "spaceship" onto the staircase, and as the user ascends from the second floor, the scenery changes from a "starry sky" to "planets" to a "galaxy." Upon reaching the third floor, the user is presented with a quiz: "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if correct, a special effect (such as a star effect) is displayed on the staircase leading to the next floor.

[2033] Furthermore, because it incorporates an emotion engine, it recognizes emotions from the user's facial expressions and voice. For example, if the user shows a surprised expression, the device detects this and sends emotion data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[2034] Example of a prompt

[2035] This week's theme is "Space Exploration." Project different images on each floor as the user climbs the stairs from the 1st to the 10th floor. Display an image of a "spaceship" on the 1st floor, a "starry sky" on the 2nd floor, and a "planet" on the 3rd floor. Also, on the 3rd floor, present a quiz: "What is the distance from Earth to the Moon?" If the user answers correctly, display a special star effect on the stairs to the next floor. Capture the user's facial expressions and voice using the camera and microphone, and if an expression of surprise is detected, send that data to the server.

[2036] Through the above steps, the system takes user emotions into consideration, addresses lack of exercise, and enhances entertainment value.

[2037] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2038] Step 1: Theme Selection

[2039] The server uses a generative AI model to select a weekly theme. To do this, the server receives various data (user interests, seasonal events, etc.) as input and selects the theme for the following week (e.g., space exploration) as output.

[2040] Step 2: Preparing the video data

[2041] Based on the selected theme, the server prepares video data using video editing software and CG software. In this step, the server receives a theme (e.g., space exploration) as input and generates corresponding video data (e.g., spaceship, starry sky, planets) as output.

[2042] Step 3: Distribution of video data

[2043] The server delivers the prepared video data to the terminals using the HTTP protocol. In this step, the server receives video data (e.g., spaceship, starry sky, planets) as input and outputs a distribution request to send it to the terminals on each floor.

[2044] Step 4: Setting up quest elements

[2045] The server sets up quizzes and mini-games on specific staircases and distributes the data to the terminals. In this step, the server receives quest elements (e.g., What is the distance from Earth to the Moon?) as input and outputs a distribution request to send them to the target terminals.

[2046] Step 5: Play the video

[2047] The terminal sends the received video data to the projector device to perform projection mapping on the stairs. In this step, the terminal receives video data (e.g., spaceship, starry sky, planets) from the server as input and outputs a video transfer request to send to the projector.

[2048] Step 6: Displaying quizzes and mini-games

[2049] At specific points along the staircase, the terminal displays quizzes or mini-games, allowing the user to participate. In this step, the terminal receives quiz data from the server (e.g., "What is the distance from the Earth to the Moon?") as input and sends a display request to output it to the display. It also receives the user's answer and feeds it back into the next action.

[2050] Step 7: User emotion recognition

[2051] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and sends them to the emotion engine. In this step, the device receives data from sensors (e.g., facial expressions, voice) as input, captures it as emotion data, and outputs a data transfer request to send it to the emotion engine.

[2052] Step 8: Interaction Processing

[2053] The device provides feedback based on the user's emotional state and quiz results, adjusting the projected images and sound effects accordingly. In this step, the device receives emotional data and quiz results as input and outputs requests to adjust the images and sounds based on them. Specifically, if the user is feeling happy, the device adjusts the colors and brightness of the images.

[2054] Step 9: Data Collection and Analysis

[2055] The server analyzes emotional data sent from the emotion engine along with the user's exercise data and response data. In this step, the server receives emotional data, exercise data, and response data as input and outputs the analysis results. Specifically, this is used to evaluate the user's health status and entertainment experience.

[2056] Because each step involves a detailed description of specific actions, this system's program transforms using stairs into an entertaining experience, providing a means to alleviate users' lack of exercise and promote their health.

[2057] (Application Example 2)

[2058] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2059] This project aims to address the lack of exercise among stair users, promote their health, and make stair use enjoyable by enhancing its entertainment value. However, conventional methods have limited effectiveness in motivating stair users and improving the user experience, necessitating new technological solutions. Furthermore, technologies for appropriate emotion recognition and feedback based on those results are still underdeveloped. A system is needed to address these challenges.

[2060] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[2061] In this invention, the server includes a theme selection means, a video data preparation means, a video data distribution means, a projection means, a quest element setting means, a data collection means, an analysis means, a user recognition means, an emotion recognition means, and an emotion-based feedback means. This makes it possible to alleviate the user's lack of exercise, improve their health, and provide a dynamic entertainment experience based on emotions.

[2062] The "theme selection method" is a function that selects a different theme for each specified period and prepares the necessary video data and public relations information based on that theme.

[2063] The "video data preparation means" is a function that creates and edits video data to be used for projection mapping based on the selected theme.

[2064] "Video data distribution means" refers to a function that transmits prepared video data to a projection means.

[2065] "Projection means" refers to a function that displays received video data on stairs or walls using projection mapping technology.

[2066] The "quest element setting method" is a function that allows users to set up quizzes or mini-games at specific locations on stairs or floors, enabling them to participate in them.

[2067] "Data collection methods" refer to functions that collect user behavior data, quest participation data, and emotional data.

[2068] "Analysis tools" refer to functions that analyze collected data to evaluate the user's health status, exercise level, emotional state, and other factors.

[2069] "User recognition means" refers to a function that recognizes the user's face and body movements to identify the individual.

[2070] "Emotion recognition means" refers to a function that analyzes the user's facial expressions and voice to determine their emotional state.

[2071] "Emotion-based feedback mechanisms" are functions that adjust projection images and sound effects based on recognized emotions to provide appropriate feedback.

[2072] This invention is designed as a system that enhances the entertainment value of using stairs, alleviates lack of exercise, and promotes health. Specific embodiments using each component of this system are described below.

[2073] Server Role

[2074] The server acts as the controller for the entire system, performing the following key tasks.

[2075] 1. Theme Selection Methods

[2076] The server selects a different theme for each specified period. For example, themes might include "space exploration" or "underwater exploration," and based on these, it prepares the necessary video data and quizzes.

[2077] 2. Means for preparing video data

[2078] The server creates and edits video data to be used for projection mapping based on the selected theme. The video data is prepared to correspond to each floor of the staircase. For example, if the theme is "Space Exploration," the first floor will be set to "Earth," the second floor to "Moon," and the third floor to "Mars."

[2079] 3. Video data distribution method

[2080] The server transmits the prepared video data to projection terminals installed on each floor. The video data is distributed in real time via the network.

[2081] 4. Quest element setting means

[2082] The server sets up quizzes and mini-games at specific stairwells or floor locations and delivers them to projection terminals. For example, a quiz such as "What is the distance from the Earth to the Moon?" might be displayed at the third floor, and users can answer it using their smartphones.

[2083] 5. Data Collection Methods

[2084] The server collects user behavior data, quiz participation data, and emotional data. This allows the server to understand user activity levels and quest participation status.

[2085] 6. Analysis tools

[2086] The server analyzes the collected data to evaluate the user's health, activity level, emotional state, and other factors. This helps in providing more effective feedback and selecting future topics.

[2087] Terminal role

[2088] Terminals are installed on each floor and in the stairwells, and they perform the following tasks based on data received from the server.

[2089] 1. Play back video

[2090] The terminal transmits the received video data to the projector device, which then performs projection mapping on the stairs.

[2091] 2. Display of quizzes and mini-games

[2092] On certain staircases, the terminal displays quizzes or mini-games, allowing users to participate.

[2093] 3. User Recognition Means

[2094] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and recognizes the user.

[2095] 4. Emotion recognition means

[2096] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and transmits them to the emotion engine. The emotion engine analyzes this data to recognize the user's emotional state.

[2097] 5. Emotion-based feedback methods

[2098] The device provides feedback based on the user's emotional state and quiz answers, adjusting the projected images and sound effects accordingly. For example, if the user is enjoying themselves, the images become more vibrant and the sound effects are enhanced.

[2099] User roles

[2100] Users can use this system to enjoy climbing stairs. The following tasks can be performed:

[2101] 1. Use of stairs

[2102] Users climb the stairs while enjoying visual effects created by projection mapping.

[2103] 2. Participation in quizzes and mini-games

[2104] On certain floors, users participate in quizzes and mini-games that are displayed, and receive rewards and special effects depending on their results.

[2105] 3. Emotion Recognition Interaction

[2106] The user's emotional state is captured by sensors such as cameras and microphones and analyzed by the system. For example, if the user shows a surprised expression, the device detects this and sends emotional data to the server. Based on this data, the server adjusts the next projection image to provide an entertainment experience that matches the user's emotions.

[2107] Hardware and software to be used

[2108] Hardware: Smartphones (e.g., iPhone, Google Pixel), projector devices, sensor terminals (cameras, microphones, etc.)

[2109] Software: EmotionRecognition library (for emotion recognition), OpenCV library (for camera input and video processing), server-side web frameworks such as Flask and Django.

[2110] Specific example

[2111] For example, if the theme is set to "Space Exploration," the server prepares video data related to the "Space Exploration" theme and distributes it to terminals on each floor from the 1st to the 10th floor. The terminal on the 1st floor projects an image of a "spaceship" onto the staircase, and as the user goes up from the 2nd floor, the scenery changes from a "starry sky" to "planets" to "galaxies." When the user reaches the 3rd floor, a quiz is displayed asking, "What is the distance from Earth to the Moon?" The user selects "approximately 380,000 kilometers" from the options, and if they answer correctly, a special effect (for example, a star effect) is displayed on the staircase to the next floor.

[2112] Example of a prompt

[2113] Example of quiz generation: "To the generation AI model: Please create 3 quiz questions related to space exploration."

[2114] Example of emotion data analysis: "To the generative AI model: Please suggest suitable entertainment content when the emotional state is happy."

[2115] The above describes specific embodiments for carrying out this invention.

[2116] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2117] Step 1:

[2118] The server selects the theme.

[2119] The server selects a different theme for each specified period, prompting the AI ​​model with the message, "Please suggest a theme for next week," and then receives the generated theme. This theme information is used for the following processes.

[2120] Input: Theme selection prompt message

[2121] Data processing / manipulation: Themes are generated using a generative AI model.

[2122] Output: Selected theme (e.g., "Space Exploration")

[2123] Step 2:

[2124] The server prepares the video data.

[2125] The server creates video data for projection mapping using video data preparation means, based on the selected theme. It collects the necessary video materials and edits them to match the theme.

[2126] Input: Selected theme

[2127] Data processing / editing: Creation and editing of video data

[2128] Output: Prepared video data

[2129] Step 3:

[2130] The server distributes the video data.

[2131] The server distributes the prepared video data to terminals installed on each floor via the network.

[2132] Input: Prepared video data

[2133] Data processing / processing: Video data distribution

[2134] Output: Supply of video data to the terminal

[2135] Step 4:

[2136] The device plays the video.

[2137] The terminal transmits the received video data to a projector device, performing projection mapping on designated staircases or walls. The video is displayed according to the selected theme.

[2138] Input: Video data from the server

[2139] Data processing / editing: Video display using projection mapping

[2140] Output: Video displays on stairs and walls

[2141] Step 5:

[2142] The server sets the quest elements.

[2143] The server requests a generation AI model to create quizzes and mini-games related to the theme. For example, it might input a prompt like, "Create three quiz questions related to space exploration," and then receive the generated quest content.

[2144] Input: Quest generation prompt text

[2145] Data processing / manipulation: Generating quest elements using a generative AI model.

[2146] Output: Generated quest content (e.g., "What is the distance from Earth to the Moon?")

[2147] Step 6:

[2148] The server will deliver quest elements.

[2149] The server distributes the generated quest details to terminals installed on specific staircases.

[2150] Input: Generated quest content

[2151] Data processing / editing: Distribution of quest content

[2152] Output: Supplying quest content to the terminal

[2153] Step 7:

[2154] The device displays a quiz / mini-game.

[2155] The terminal displays received quest content on a projector device or terminal screen, allowing users to participate. Users answer quizzes and mini-games in real time.

[2156] Input: Quest details from the server

[2157] Data processing / manipulation: Displaying quizzes and mini-games

[2158] Output: User participation in quizzes and mini-games

[2159] Step 8:

[2160] The device recognizes the user's emotions.

[2161] The device uses sensors such as cameras and microphones to capture the user's facial expressions, voice, and body movements, and analyzes them using emotion recognition technology.

[2162] Input: User's facial expressions, voice, and motion data

[2163] Data processing / analysis: Recognition and analysis of emotional states

[2164] Output: Recognized emotion data

[2165] Step 9:

[2166] The device sends emotional data to the server.

[2167] The device sends the recognized emotion data to the server. The server receives this data and uses it for the next processing step.

[2168] Input: Recognized emotion data

[2169] Data processing / processing: Sending emotional data

[2170] Output: Supplying emotion data to the server

[2171] Step 10:

[2172] The server provides emotion-based feedback.

[2173] The server analyzes the received emotional data and adjusts the next projection images and sound effects based on the emotions. It inputs a prompt message to the generating AI model, "Please suggest suitable entertainment content when the emotional state is happy," and sends the generated content to the terminal.

[2174] Input: Sentiment data, entertainment content suggestion prompt text

[2175] Data Processing / Editing: Analysis and generation of emotional data, content proposals using AI models.

[2176] Output: Adjusted entertainment content

[2177] Step 11:

[2178] The device displays the feedback content.

[2179] The device plays back the adjusted projection images and sound effects received from the server and provides feedback to the user. This is designed to create an entertainment experience that responds to the user's emotional state.

[2180] Input: Coordinated entertainment content from the server

[2181] Data processing / editing: Playback of projection images and sound effects.

[2182] Output: Providing feedback to users

[2183] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2184] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing sp...

Claims

1. A theme selection method that involves selecting a different theme each week, A video data preparation means for preparing video data based on the selected theme, A video data distribution method that distributes prepared video data to terminals on each floor, A projection method that uses received video data to perform projection mapping on a staircase, A means of setting quest elements, such as quizzes or mini-games on specific staircases, A data collection method that collects user behavior data, quiz answer data, etc. A system that includes analytical means to analyze collected data and evaluate exercise levels and health status.

2. The system according to claim 1, wherein the projection means performs projection mapping on a staircase spanning multiple floors.

Images