System

Abstract

A system is provided.SOLUTION: A system, comprising: means for obtaining indoor 360-degree video by a user device; means for inputting building structure information by the user device; means for transmitting the 360-degree video and the building structure information from the user device to a server; means for analyzing the received information by a AI model in the server; means for executing a seismic simulation based on an analysis result in the server; means for generating a video for visualizing a simulation result in the server; means for transmitting the generated video from the server to the user device; and means for displaying the received video in the user device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional methods of providing information on earthquake countermeasures have been limited to general guidelines and warnings, and have not proposed specific countermeasures tailored to individual environments, making it difficult for users to intuitively understand the risks of earthquakes and implement specific countermeasures.In addition, the proposed countermeasures were abstract, making it difficult for users to specifically understand how they should act. [Means for solving the problem]

[0005] The present invention provides a system including a means for acquiring indoor 360-degree video data using a user terminal, a means for inputting building structural information using the user terminal, a means for transmitting the 360-degree video data and building structural information from the user terminal to a server, a means for analyzing the received data using an AI model on the server, a means for performing an earthquake simulation based on the analysis results on the server, a means for generating a video on the server that visualizes the simulation results, a means for transmitting the generated video from the server to the user terminal, and a means for displaying the received video on the user terminal, thereby enabling users to concretely and intuitively understand the risks during an earthquake and encouraging them to take concrete action to implement appropriate countermeasures.

[0006] Furthermore, by including a means on the user terminal that presents specific earthquake countermeasure ideas to the user, it becomes easier for the user to learn specific countermeasure methods. Also, by including a means on the server that adds information including suggestions for countermeasure goods to the simulation video generated based on the 360-degree video data and building structure information, the user can instantly obtain the necessary countermeasure goods and easily implement them, further promoting the implementation of actual countermeasure actions.

[0007] "User terminal" means an electronic device operated by a user to collect, input, send, and receive data, including a smartphone, tablet, or personal computer.

[0008] "360-degree video data" is video data obtained by capturing a specific environment or space from all directions, and is used by users to cover every angle indoors.

[0009] "Building structural information" is information related to the structural characteristics of a building, including materials, age, number of floors, earthquake resistance, and the like.

[0010] "Server" means a central computer system for receiving, analyzing, storing, and processing data sent from user terminals, and generating and transmitting results.

[0011] An "AI model" is a system of mathematical and computational algorithms that uses artificial intelligence techniques to analyze data and predict or generate outcomes.

[0012] "Analysis" is the process of breaking down data provided by users, converting it into an understandable format, and extracting useful information from that data.

[0013] "Simulation" is a technique for reproducing real-world events in a virtual environment and predicting behavior and results under specific conditions, and in this invention refers to the virtual reproduction of the situation during an earthquake.

[0014] "Visualization video" is an output in a video format that displays the simulation results in a way that is easy for users to understand, and visually represents the movement and behavior of furniture and objects.

[0015] "Emergency supplies" are specific items and tools recommended for earthquake preparedness, such as furniture fasteners, earthquake-resistant reinforcement materials, and fire prevention devices. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This system allows users to collect 360-degree indoor video data, send it to a server along with building structural information, and then analyze the data and perform earthquake simulations on the server, generating the results as videos and providing them to users. To implement this system, a user terminal, a server, and software that connects them are required.

[0038] 1. User device operation

[0039] Users launch a dedicated app on their smartphone, tablet, or other device. In the app, the user first takes a picture of the entire room to obtain 360-degree video data of the room. At this time, the camera rotates to record video in all directions.

[0040] Next, users input building structural information through the app, including materials (e.g., wood, reinforced concrete), age, number of floors, seismic performance, etc. This improves the accuracy of the simulation.

[0041] 2. Data transmission

[0042] The user device sends the captured 360-degree video data and building structure information to a server, where the data is securely transferred over the Internet.

[0043] 3. Processing on the Server

[0044] The server receives the data sent by the user and analyzes it using an AI model. Specifically, it analyzes 360-degree video data to identify the room layout and furniture placement, and simulates shaking patterns during an earthquake based on building structural information.

[0045] 4. Earthquake Simulation

[0046] After the server analyzes the data, it runs an earthquake simulation based on the analysis results. The simulation recreates how furniture and appliances move and fall, and how books and tableware will fly apart. This process visualizes the risks in the event of an earthquake.

[0047] 5. Video Creation and Distribution

[0048] Based on the simulation results, the server uses CG technology to generate a video that shows in detail what the user's interior will be like when an earthquake occurs, and how furniture and objects will move.

[0049] The generated video is then sent back to the user's device, where it is displayed on the app and viewed by the user.

[0050] 6. User Notification and Recommendations

[0051] Users will receive a notification from the app that a simulation video has been prepared. By watching the video, they can gain a concrete understanding of the risks involved in an earthquake and recognize what countermeasures are necessary. The system also suggests countermeasure items, making it easier for users to immediately implement countermeasures.

[0052] Specific examples

[0053] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. By watching this video on the app, the user can consider specific measures to take, such as securing furniture and organizing unnecessary items.

[0054] In this way, the present invention encourages users to concretely understand the risks during earthquakes and take appropriate measures, thereby improving safety and reducing damage.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The user launches the dedicated app on their device. Following the app's instructions, the user rotates the camera to capture the entire room in order to obtain 360-degree video data of the indoor space. Once the capture is complete, the video data is temporarily saved.

[0058] Step 2:

[0059] On the user's device, the user enters building structural information into an input form within the app. Specifically, the user enters information such as the building's materials (e.g., wood, reinforced concrete), age, number of floors, and earthquake resistance. The entered information is temporarily saved along with the video data.

[0060] Step 3:

[0061] The user device sends the collected 360-degree video data and building structure information to a server. The data is securely transferred over the Internet. The data is compressed and encrypted as needed.

[0062] Step 4:

[0063] The server temporarily stores the received 360-degree video data and building structure information, then uses AI models to analyze the data, identify room layouts and furniture placements, and evaluate the building's earthquake resistance based on its structure.

[0064] Step 5:

[0065] The server then runs an earthquake simulation based on the analysis results. Specifically, it specifies the seismic intensity setting and calculates how shaking corresponding to that intensity will affect the space. This reproduces how furniture and objects will move, fall, and scatter.

[0066] Step 6:

[0067] The server generates a video based on the simulation results. The video includes the state of the interior of the building at the time of the earthquake, the movement of furniture, and the scattering of objects. The video is visualized intuitively to make it easy for users to understand.

[0068] Step 7:

[0069] The server then sends the generated video to the user's device, where it is compressed, optimized, and converted into a format that can be played on the user's device.

[0070] Step 8:

[0071] The user device receives the video sent from the server and notifies the user within the app, allowing the user to open the app and watch the generated simulation video.

[0072] Step 9:

[0073] After watching the video, the user device will present the user with specific earthquake countermeasures, including suggestions for securing furniture and emergency supplies, to help users take immediate action.

[0074] Through these steps, the present invention provides a system that allows users to concretely understand the risks during an earthquake and encourages them to take specific actions to take appropriate measures.

[0075] Example 1

[0076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0077] Conventional earthquake countermeasures have had the problem that risk assessment taking into account the specific structure and furniture layout of a home is difficult, and general guidelines and countermeasures alone cannot ensure sufficient safety. In particular, in order for users to specifically understand the situation in their home and take appropriate countermeasures, simulations that reproduce the interior and furniture layout of the home are necessary. However, conducting such simulations individually requires high costs and specialized knowledge, making it difficult for many ordinary users to implement.

[0078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0079] In this invention, the server includes means for acquiring indoor omnidirectional video data via a user terminal, means for inputting building structural information via the user terminal, means for transmitting the omnidirectional video data and the building structural information from the user terminal to the server, means for analyzing the received data using a generative machine learning model in the server, means for executing an earthquake simulation based on the analysis results in the server, means for generating a video that visualizes the simulation results in the server, means for transmitting the generated video from the server to the user terminal, and means for displaying the received video in the user terminal. This enables users to visually understand risks through earthquake simulation videos that recreate specific conditions in their homes and take appropriate measures.

[0080] A "user terminal" is an information device that can be operated by an individual, and typically takes the form of a smartphone or tablet.

[0081] "Omnidirectional video data" refers to video data captured from a 360-degree perspective, recording the entire area around the user.

[0082] "Building structure information" is data that includes detailed information about the building's structure, such as the building's materials, age, number of floors, and earthquake resistance.

[0083] "Transmission means" refers to the technical means used to send data from one point to another, typically referring to communications protocols over the Internet.

[0084] A "generative machine learning model" is a model that contains algorithms trained to perform advanced tasks such as data analysis and prediction.

[0085] "Earthquake simulation" refers to a virtual experiment that simulates the shaking and movement of buildings and furniture in the event of an earthquake.

[0086] "Means for generating animation" refers to technical means for creating animation-format data to visually represent analysis results or simulation results.

[0087] "Means of display" refers to the technical means by which users can visually confirm information, and is primarily a display or screen.

[0088] "Specific earthquake countermeasures" refers to information that includes suggestions for specific actions and countermeasures that users should take in the event of an earthquake.

[0089] "Disaster prevention supplies" refer to items designed to reduce risk, such as furniture fasteners and disaster prevention kits, which are used in the event of an earthquake.

[0090] This system allows users to acquire indoor omnidirectional video data, send it to a server along with building structural information, and then analyze the data and perform earthquake simulations on the server, generating the results as videos and providing them to users. To implement this system, a user terminal, a server, and software that links these together are required.

[0091] User device operations

[0092] The user launches the dedicated app on a user device such as a smartphone or tablet. When the app is launched for the first time, it obtains necessary permissions from the user, such as permission for the camera and internet connection. Next, the user uses the app's camera function to capture an image of the entire room to collect indoor omnidirectional video data. The user rotates the camera to record omnidirectional video. The user then enters building structural information into a form within the app. This structural information includes the building's material (e.g., wood or reinforced concrete), age, number of floors, and earthquake resistance.

[0093] Sending data

[0094] The user device combines the collected omnidirectional video data and the input building structure information into a single data package, which is then encrypted using the SSL / TLS protocol and sent to a server via the Internet.

[0095] Processing on the server

[0096] The server receives the data package sent from the user device and analyzes it using a generative machine learning model (e.g., TensorFlow or PyTorch). The analysis includes identifying the room layout and furniture placement based on the omnidirectional video data, and evaluating the building's seismic performance based on the building's structural information.

[0097] Earthquake Simulation

[0098] The server sets up a simulation using earthquake simulation software (e.g., OpenSees or FLAC) based on the analysis data. Simulation parameters include seismic intensity, seismic waveform, and ground conditions. The server runs the simulation under the set earthquake conditions and calculates the movement and fall of furniture and home appliances, as well as the scattering of books, tableware, etc.

[0099] Video Creation and Delivery

[0100] The server generates a video using CG software (e.g., Blender or Maya) based on the simulation results. The generated video includes scenes of furniture moving around and objects flying around. The server compresses the video file and sends it to the user's device. The video file is transferred using a secure protocol (e.g., HTTPS).

[0101] User notification and suggested solutions

[0102] The user device receives a notification that the video received from the server is ready. The app's notification function informs the user that the simulation results are available for viewing. The user plays the simulation video within the app to confirm specific earthquake risks. The system then recommends countermeasures (e.g., furniture fasteners, disaster prevention kits) and helps the user implement countermeasures immediately.

[0103] Specific examples

[0104] For example, if a user takes a photo of their living room with an omnidirectional camera and enters the building's structural information as "wooden construction" and "20 years old," the user's device will send this data to the server. The server will analyze the data and run a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. The user can watch this video in the app and consider specific measures, such as securing furniture and organizing unnecessary items.

[0105] Example prompt sentence:

[0106] "Take a 360-degree video of your living room and enter the building's structural information. For example, 'wooden structure' and '20 years old'. Then send the data to the server."

[0107] In this way, the present invention helps users to concretely understand the risks during earthquakes and take appropriate measures, thereby improving safety and reducing damage.

[0108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0109] Step 1:

[0110] The user launches a dedicated app on a device such as a smartphone or tablet. When the app is launched for the first time, it obtains necessary permissions from the user, such as camera and internet access. Based on this, the device obtains camera access permission and internet access permission. This becomes the input.

[0111] What happens: The user launches the app and sees a popup asking for permission.

[0112] Step 2:

[0113] The user uses the camera function in the app to collect indoor omnidirectional video data. The user rotates the camera to capture the entire room, and the device saves this video as omnidirectional video data. This is the input, and the output is omnidirectional video data.

[0114] Specific operation: The user stands in the center of the room and rotates the camera horizontally to take a picture.

[0115] Step 3:

[0116] Users enter building structural information into a form within the app. This structural information includes materials (wood, reinforced concrete, etc.), age of the building, number of floors, and earthquake resistance. The device receives this input, organizes the data, and saves it. This is the input, and the output is the building structural information.

[0117] What happens: The user enters information using text boxes and drop-down menus and presses the "Submit" button.

[0118] Step 4:

[0119] The user terminal combines the collected omnidirectional video data and the input building structure information into a single data package, then encrypts the data package using the SSL / TLS protocol and sends it to the server. This is the input, and the output is the encrypted data package.

[0120] Specific operation: The app automatically packages the data, and when you press the "Send" button, the encrypted data is sent to the server.

[0121] Step 5:

[0122] The server receives the data package sent from the user terminal, decompresses it, and obtains the omnidirectional video data and building structure information. This is the input, and the output is the decompressed omnidirectional video data and building structure information.

[0123] Specific operation: The server receives the data package and automatically begins decompressing and analyzing it.

[0124] Step 6:

[0125] The server uses a generative machine learning model (e.g., TensorFlow or PyTorch) to analyze the omnidirectional video data. The server uses this analysis to determine the room layout and furniture placement. This is the input, and the output is the room layout information and furniture placement information.

[0126] Specific operation: The server inputs omnidirectional video data into the AI ​​model and generates layout information as the analysis result.

[0127] Step 7:

[0128] The server further evaluates the seismic performance based on the building structural information. From this evaluation, the server obtains an evaluation result of the building's susceptibility to shaking and seismic performance. This is the input, and the output is the result of the seismic performance evaluation.

[0129] Specific operation: The server processes building structural information and evaluates the structural earthquake resistance using numerical values ​​and indicators.

[0130] Step 8:

[0131] The server sets up and runs a simulation using earthquake simulation software (e.g., OpenSees or FLAC) based on the analysis data and seismic performance evaluation. The server calculates the movement and fall of furniture and appliances, and the scattering of books and tableware. This is the input, and the output is the results of the earthquake simulation.

[0132] Specific operation: The server runs the simulation software and performs a simulation on the 3D model.

[0133] Step 9:

[0134] The server generates a video using CG software (e.g., Blender or Maya) based on the simulation results. The server creates a video that visually represents the movement of furniture and objects. This is the input, and the output is the generated simulation video.

[0135] Specific operation: The server inputs the simulation data into the CG software and renders the animation.

[0136] Step 10:

[0137] The server compresses the generated video and sends it to the user's device. The video file is transferred using a secure protocol (e.g. HTTPS). This is the input, and the output is the compressed video transmission.

[0138] Specific operation: The server compresses the video file and sends it to the user's device.

[0139] Step 11:

[0140] The user device receives a notification from the server that the video is ready. The user plays the simulation video in the app. This is the input, and the output is a notification to the user and the video playback.

[0141] Specific behavior: The device displays a notification, and the user presses the "play" button in the app to play the video.

[0142] Step 12:

[0143] The system displays information during the video, including suggestions for countermeasures (e.g., furniture fixings, disaster prevention kits). This helps users to easily take specific countermeasures. This is the input, and the output is the display of countermeasure suggestion information.

[0144] How it works: The app displays a simulation video and recommended countermeasures products on the screen.

[0145] (Application example 1)

[0146] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0147] The present invention relates to a simulation system for specifically recognizing earthquake risks, and further aims to utilize this system to provide users with a means of effectively proposing disaster prevention products and insurance services, enabling them to take prompt and appropriate countermeasures. Another objective is to make it easier for users to ensure safety and security in their lives by obtaining useful suggestions directly from the earthquake simulation results.

[0148] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0149] In this invention, the server includes a means for generating advertisements based on the analysis results, a means for inserting advertisements into videos, and a means for adding information including suggestions for disaster prevention goods and insurance services to the videos generated based on the analysis results. As a result, the user can receive specific suggestions for effective disaster prevention goods and insurance services while viewing the simulation results, enabling them to easily and quickly take earthquake countermeasures.

[0150] A "user terminal" is an information and communication device operated by a user, including smartphones and tablets.

[0151] "360-degree video data" refers to video data that records and displays the entirety of a specific location from every direction.

[0152] "Building structural information" refers to information about a building, including the building's materials, age, number of floors, earthquake resistance, etc.

[0153] A "generative model" refers to an artificial intelligence model that analyzes 360-degree video data and building structural information to generate the data that serves as the basis for simulations.

[0154] "Analysis" refers to the process of extracting and understanding specific information from received data.

[0155] "Earthquake simulation" refers to the process of predicting and virtually recreating the shaking and movement of objects that occur during an earthquake.

[0156] "Visualization video" refers to a video generated to display the results of analysis or simulation in a visually easy-to-understand format.

[0157] "Means for generating advertisements" refers to a mechanism that automatically creates advertisements for users, such as countermeasures products and insurance services, based on the analysis results.

[0158] "Emergency goods" refer to items used to reduce damage during earthquakes.

[0159] "Insurance services" refers to insurance products to cover damage caused by earthquakes.

[0160] "Insertion means" refers to the technical means used to add advertising or suggestion information to the generated video.

[0161] This invention is a system that allows users to collect 360-degree video data of their homes and rooms, as well as structural information about the building, using a dedicated app, and then sends that data to a server to perform earthquake simulations, visualize the risks, and insert advertisements into the video, including suggestions for disaster prevention products and insurance services.

[0162] User device operations

[0163] Users launch a dedicated app on their smartphone, tablet, or other device to first capture 360-degree indoor video data. At this time, the camera rotates to record video in all directions. Next, the user enters the building's structural information through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This improves the accuracy of the simulation.

[0164] Sending data

[0165] The user device sends the acquired 360-degree video data and building structure information to a server via the Internet, where the data is transferred securely.

[0166] Processing on the server

[0167] The server receives the data sent by the user and analyzes it using a generative model. Specifically, it analyzes the 360-degree video data to determine the room layout and furniture placement, and simulates the shaking patterns during an earthquake based on building structural information. Based on the simulation results, a video is generated using CG technology. The generated video shows in detail what state the user's indoors will be in when an earthquake occurs, and how furniture and objects will move.

[0168] Ad generation and insertion

[0169] Furthermore, the server generates advertisements for users based on the analysis results, including suggestions for countermeasures and insurance services. These advertisements are inserted into the simulation video. The generated final video is then sent back to the user's device.

[0170] Watching videos and suggesting solutions

[0171] The received video is displayed on the app on the user's device. By watching the simulation video, users can gain a concrete understanding of the risks involved in an earthquake and recognize what countermeasures are necessary. The system also suggests countermeasure products and insurance services, making it easier for users to take immediate action.

[0172] Hardware and software used

[0173] The hardware uses a smartphone and a 360-degree camera, and the software uses the requests library (for sending HTTP requests), the moviepy library (for video editing), and an AI model for earthquake simulation on the server side.

[0174] Examples of specific examples and prompts

[0175] Specific examples

[0176] Users launch a dedicated smartphone app and take a 360-degree photo of their living room. They then input the building's structural information, such as "wooden construction," "20 years old," "two stories," and "average earthquake resistance." This information is sent to a server, and a video containing advertisements for "earthquake-resistant mats," "earthquake insurance," and "furniture tip-prevention products" is displayed based on the simulation results.

[0177] Prompt Sentence Examples

[0178] Based on 360-degree video data taken by users and building structural information, simulate shaking during an earthquake and the movement of furniture and objects, and reflect the results in your advertising video. Specifically, simulate with high accuracy how furniture will fall and objects will fly off, and recommend products and services to users to mitigate the risk.

[0179] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0180] Step 1:

[0181] The user launches a dedicated app on a user device such as a smartphone or tablet. The user acquires 360-degree video data of the indoor space, rotating the camera to record video in all directions. The input is the video data acquired from the 360-degree camera, and the output is 360-degree video data stored on the device.

[0182] Step 2:

[0183] Users input building structural information through the app, including materials (e.g., wood, reinforced concrete), age, number of floors, earthquake resistance, etc. The input is the building structural information entered by the user, and the output is the building structural information stored on the device.

[0184] Step 3:

[0185] The user device transmits the acquired 360-degree video data and building structure information to a server via the Internet. The input is the 360-degree video data and building structure information stored on the device, and the output is the data transmitted to the server.

[0186] Step 4:

[0187] The server receives the data sent by the user and analyzes it using a generative AI model. Specifically, it analyzes the 360-degree video data to identify the room layout and furniture placement, and simulates the shaking patterns during an earthquake based on building structural information. The input is the sent 360-degree video data and building structural information, and the output is the analysis and simulation results.

[0188] Step 5:

[0189] The server uses CG technology to generate an earthquake simulation video based on the simulation results. The generated video shows in detail what state the user's interior will be in when an earthquake occurs, and how furniture and objects will move. The input is the simulation results, and the output is the generated simulation video.

[0190] Step 6:

[0191] Furthermore, based on the analysis results, the server generates advertisements for users, including suggestions for countermeasures and insurance services, and inserts these advertisements into the simulation video. The inputs are the simulation video and advertisement information, and the output is the final video with the advertisements inserted.

[0192] Step 7:

[0193] The server then sends the generated final video back to the user terminal. The input is the final video, and the output is the video sent to the user terminal.

[0194] Step 8:

[0195] The final video received is displayed on the app on the user's device and the user is allowed to watch it. By watching the simulation video, the user can concretely understand the risks in the event of an earthquake and recognize what countermeasures are necessary. The input is the final video received, and the output is the user's viewing and consideration of countermeasures.

[0196] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0197] This system uses an emotion engine installed in a user device to recognize the user's emotions and propose earthquake countermeasures based on those emotions. It operates by combining the user device, server, emotion engine, and software that links them together.

[0198] 1. User device operation

[0199] Users launch a dedicated app using their device, such as a smartphone or tablet. In the app, users first rotate the camera to capture the entire room to obtain 360-degree video data of the interior. Once the video is complete, the video data is temporarily saved. Next, users enter structural information about the building through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This improves the accuracy of the simulation.

[0200] 2. Operation of the Emotion Engine

[0201] While the user is recording video and inputting building structure information, the emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, and input. This data is analyzed to identify the user's current emotional state (e.g., relief, anxiety, fear). The emotion data is temporarily stored and sent to the server along with the analysis results.

[0202] 3. Data transmission

[0203] The user device transmits the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted as appropriate.

[0204] 4. Processing on the Server

[0205] The server receives and temporarily stores the data sent by the user, then uses an AI model to analyze the 360-degree video data and building structure information to determine room layout and furniture placement, and evaluate the earthquake resistance of the building based on its structure.

[0206] 5. Earthquake Simulation

[0207] After the server analyzes the data, it runs an earthquake simulation based on the analysis results. The simulation recreates how furniture and appliances move and fall, and how books and tableware will fly apart. This process visualizes the risks in the event of an earthquake.

[0208] 6. Video Creation and Distribution

[0209] Based on the simulation results, the server uses CG technology to generate a video. This video shows in detail what the interior of a building looks like when an earthquake occurs, how furniture moves, and how scattered objects are. Based on the analysis results of the emotion engine, messages and countermeasures information corresponding to the user's emotions are displayed in the video and within the app.

[0210] 7. Display on user device

[0211] The generated video is then sent back to the user's device. During transmission, the data is compressed and optimized, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[0212] 8. Emotion-based countermeasures

[0213] Users will receive a notification from the app that a simulation video has been prepared. As they watch the video, the system will customize and present specific earthquake preparedness measures to the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will suggest a more reassuring message and easy-to-implement measures. It will also suggest emergency supplies, making it easier for users to take measures immediately.

[0214] Specific examples

[0215] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine determines that the user is feeling anxious, the system will display specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[0216] In this way, the present invention helps users concretely understand the risks during earthquakes and implement appropriate countermeasures, thereby improving safety and reducing damage. A customized approach that takes into account the user's emotions can make the implementation of countermeasure actions even more effective.

[0217] The processing flow will be explained below.

[0218] Step 1:

[0219] The user launches a dedicated app on their smartphone or tablet. Following the app's instructions, the user rotates the camera to capture the entire room, capturing 360-degree video data of the indoor space. Once the video is complete, it is temporarily saved on the user's device.

[0220] Step 2:

[0221] Users input building structural information into the app, such as the building's materials (wood, reinforced concrete, etc.), age, number of floors, and earthquake resistance. The input building structural information is temporarily saved on the user's device along with the video data.

[0222] Step 3:

[0223] The emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, etc. This process is carried out while the user is entering information, and determines whether the user is feeling safe, anxious, or scared. The emotion data is analyzed and temporarily stored.

[0224] Step 4:

[0225] The user device sends the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted during transmission.

[0226] Step 5:

[0227] The server receives the 360-degree video data, building structure information, and emotion data sent by the user. The server temporarily stores this data and then begins the analysis process. First, it analyzes the 360-degree video data to identify the room layout and furniture placement.

[0228] Step 6:

[0229] The server evaluates the earthquake resistance of a building based on structural information, taking into account information such as the building's materials and age, and predicts the shaking pattern and extent of impact during an earthquake.

[0230] Step 7:

[0231] The server then runs an earthquake simulation based on the analysis results. The simulation reproduces how furniture and home appliances will move and fall, and how books, tableware, etc. will fly apart in response to a specified seismic intensity (for example, intensity 6). The simulation results are then generated.

[0232] Step 8:

[0233] The server uses CG technology to generate a video based on the simulation results, which shows in detail what the interior of a building looks like when an earthquake strikes, how furniture moves, and how objects are scattered.

[0234] Step 9:

[0235] Based on the analysis results of the emotion engine, the server inserts emotional messages and countermeasures information into videos and apps. For example, if a user is feeling anxious, the system will display a message designed to reassure them.

[0236] Step 10:

[0237] The server then sends the generated video to the user's device, where it is converted into a playable format and appropriately compressed.

[0238] Step 11:

[0239] The user's device receives the video sent from the server and notifies the user within the app. Upon receiving the notification, the user can open the app and watch the generated simulation video.

[0240] Step 12:

[0241] After watching the video, the user's device will present customized earthquake countermeasures based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will detail easy-to-implement countermeasures and display a message to provide reassurance. It will also suggest necessary countermeasure goods.

[0242] In this way, this system supports users in taking concrete and effective earthquake countermeasures while taking their emotions into consideration. The generated videos and customized countermeasures help users to concretely understand the risks of earthquakes and take appropriate measures.

[0243] Example 2

[0244] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0245] Conventional earthquake simulation systems propose earthquake countermeasures without considering the user's emotions or psychological state, which makes it difficult to alleviate users' anxiety and reduces the rate at which countermeasures are implemented. Furthermore, the system does not provide sufficient suggestions for specific countermeasures or the goods needed to implement them, so the proposals are not in a form that is easy for users to implement immediately.

[0246] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0247] A means for acquiring user emotion data by a user terminal;

[0248] A means for transmitting 360-degree video data, building structure information, and emotion data from a user terminal to a server;

[0249] A means for analyzing the received data by an AI model in a server;

[0250] This makes it possible to provide specific and feasible earthquake countermeasures that take into account the user's emotional state.

[0251] A "user terminal" is a portable information processing device operated by a user, such as a smartphone or tablet.

[0252] "360-degree video data" is data that digitally records visual information in all directions captured by a rotating camera.

[0253] "Building structural information" refers to data on the physical characteristics of a building, such as the building's materials, age, number of floors, and earthquake resistance.

[0254] "Emotional data" is data that indicates the user's psychological state, analyzed from the user's facial expressions, tone of voice, etc.

[0255] A "server" is a computer system that receives data from user terminals via a network and analyzes and processes it.

[0256] An "AI model" is a program that uses artificial intelligence algorithms to analyze data and make predictions and simulations.

[0257] "Earthquake simulation" is a computational process that recreates in a virtual environment the conditions inside a building during an earthquake.

[0258] "Movie" refers to dynamic video data generated to visually visualize the simulation results.

[0259] "Countermeasures proposals" are proposals that specifically outline the actions users should take and the preparations they should make in the event of an earthquake.

[0260] "Earthquake prevention goods" refer to equipment and devices necessary for earthquake prevention, which help users implement earthquake prevention measures.

[0261] This invention is a system that uses an emotion engine installed in a user terminal to recognize a user's emotions and proposes earthquake countermeasures based on those emotions. This system operates by combining a user terminal, a server, an emotion engine, and software that links these together.

[0262] First, the user launches a dedicated app on their device, such as a smartphone or tablet. To obtain 360-degree indoor video data within the app, the user rotates the camera to capture the entire room. Once the video is complete, it is temporarily saved. Next, the user enters the building's structural information. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and earthquake resistance.

[0263] Next, while the user records video and inputs building structure information, the emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, and input content. This data is analyzed to identify the user's current emotional state (e.g., relief, anxiety, fear). The emotion data is temporarily stored and sent to the server along with the analysis results.

[0264] The user device transmits the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted as appropriate.

[0265] The server receives the data sent by the user and temporarily stores it. The server then uses an AI model to analyze the 360-degree video data and building structural information to determine room layout and furniture placement, and evaluate the building's earthquake resistance based on its structure. Specific technologies used include image processing algorithms and machine learning models. This provides the basic data needed to accurately simulate risks in the event of an earthquake.

[0266] Once the analysis is complete, the server runs an earthquake simulation based on the results. The simulation recreates how furniture and appliances will move and fall, and how books, tableware, etc. will fly apart. This provides a visually easy-to-understand display of specific earthquake risks.

[0267] Based on the simulation results, the server uses CG technology to generate a video. This video depicts in detail the state of the room at the time of the earthquake, the movement of furniture, and the scattering of objects. In addition, based on the analysis results of the emotion engine, messages and countermeasure information corresponding to the user's emotions are displayed in the video and within the app. For example, if the user is feeling anxious, a reassuring message and easy-to-implement countermeasure information are displayed.

[0268] The generated video is sent from the server to the user's device. The data is compressed and optimized before being sent, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[0269] When users receive a notification from the app that a simulation video has been prepared and watch the video, the system will customize and present specific earthquake preparedness measures to the user based on the analysis results of the emotion engine. If the user is feeling anxious, the system will suggest messages that will provide more reassurance and easy-to-implement measures. It will also suggest emergency supplies, making it easier for users to take measures immediately.

[0270] As a specific example, a user takes a 360-degree photo of their living room and enters the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine recognizes that the user is feeling anxious, the system displays specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[0271] An example of a prompt for the generative AI model is, "If it recognizes that the user is feeling anxious, please generate a message suggesting specific measures to reduce risks in the event of an earthquake." Using this prompt, the AI ​​model can generate appropriate countermeasure information according to the user's emotional state.

[0272] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0273] Step 1:

[0274] The user launches a dedicated app using a user device such as a smartphone or tablet. The device rotates the camera to capture the entire room, allowing the user to obtain 360-degree video data of the room within the app. The input in this step is the user's operation and video capture by the camera, and the output is 360-degree video data. The user then operates the device appropriately to rotate the camera correctly and obtain a 360-degree video image of the room.

[0275] Step 2:

[0276] The user inputs structural information about the building through a dedicated app. The device temporarily stores the information entered by the user, such as the building's materials (wood, reinforced concrete, etc.), age, number of floors, and seismic performance, in a database. The input in this step is detailed building information manually entered by the user, and the output is the temporarily stored building structural information.

[0277] Step 3:

[0278] The device uses a front camera and microphone to capture the user's facial expressions and tone of voice. The emotion engine analyzes this data in real time to generate the user's emotion data. The input in this step is the user's facial and voice data, and the output is the analyzed emotion data.

[0279] Step 4:

[0280] The device assembles the acquired 360-degree video data, building structure information, and emotion data into packets. These packets are then compressed and encrypted. The device then sends these packets to a server via the Internet. The input in this step is the individual data (360-degree video data, building structure information, emotion data), and the output is the encrypted data packet sent to the server.

[0281] Step 5:

[0282] The server receives the data packet sent from the terminal and checks the integrity of the content. If it is normal, it temporarily stores the data. The input in this step is the encrypted data packet, and the output is the temporarily stored data set.

[0283] Step 6:

[0284] The server uses an AI model to analyze the 360-degree video data and building structural information. Specifically, it uses image processing algorithms to identify room layouts and furniture placements and evaluate the building's earthquake resistance. The input for this step is the temporarily stored dataset (360-degree video data, building structural information), and the output is the analysis results.

[0285] Step 7:

[0286] The server runs an earthquake simulation based on the analysis results. The simulation calculates how the effects of a virtual earthquake will be transmitted to furniture and structures, and generates specific simulation data. The input in this step is the analysis results, and the output is simulation data.

[0287] Step 8:

[0288] The server uses computer graphics technology to generate a video that visualizes the indoor situation during an earthquake using the simulation data. Furthermore, based on the analysis results of the emotion engine, messages and countermeasure information for users are added to the video. The inputs for this step are the simulation data and emotion data, and the output is the generated video.

[0289] Step 9:

[0290] The server compresses the generated video and sends it to the user's device. By converting it into the appropriate format, the video can be played back without any problems on the user's device. The input of this step is the generated video, and the output is a compressed video file.

[0291] Step 10:

[0292] The device receives the video sent from the server and notifies the user. The input in this step is a compressed video file, and the output is a video file stored on the device and a notification to the user. The user opens the app and watches the generated simulation video.

[0293] Step 11:

[0294] The device provides a function in which the system suggests specific earthquake countermeasures based on the user's emotional data while watching the simulation video. If the user feels anxious, specific countermeasures are displayed. The input in this step is the emotional data and the generated video, and the output is a video display containing the countermeasures.

[0295] Through these steps, this system will encourage users to take earthquake countermeasures, thereby improving safety and reducing damage.

[0296] (Application example 2)

[0297] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0298] Conventional earthquake countermeasure proposal systems present uniform countermeasure proposals without considering the user's emotions, making it difficult to provide specific countermeasures that respond to the psychological state and needs of each individual user. Furthermore, when visualizing earthquake simulation results, the provision of countermeasure proposals to deepen users' understanding or information that gives them a sense of security is insufficient. This reduces users' motivation to actually implement countermeasures, making it difficult to improve safety.

[0299] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0300] In this invention, the server includes means for acquiring emotion data using an emotion recognition engine in the user terminal, means for generating information including countermeasures corresponding to the user's emotions based on a video generated by the server, and means for transmitting the generated video and countermeasures from the server to the user terminal. This makes it possible to provide individualized countermeasures corresponding to the user's emotions, allowing the user to take specific and feasible earthquake countermeasures with greater peace of mind.

[0301] A "user terminal" is an electronic device operated by a user, including portable devices such as smartphones and tablet PCs.

[0302] "Indoor 360-degree video data" refers to video data of an indoor environment captured by a user in all directions, visually recording the entire room.

[0303] "Building structural information" refers to information about the structure of a building, such as its materials, age, number of floors, and earthquake resistance.

[0304] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and tone of voice to identify the user's emotional state.

[0305] "Server" refers to a computer system for analyzing, processing, storing, transmitting and receiving data.

[0306] An "AI model" is a model that uses machine learning and deep learning techniques to analyze input data and has algorithms to perform specified tasks.

[0307] "Earthquake simulation" is the process of recreating on a computer what the interior of a building would look like when an earthquake occurs, based on building structural information and 360-degree video data.

[0308] "Videos that visualize simulation results" refers to animations and videos generated to visually show the results of earthquake simulations.

[0309] "Countermeasures" refers to information that includes specific actions, recommended items, and initiatives proposed to reduce earthquake risk.

[0310] "Means of transmission" refers to the software and hardware technology used to send data acquired or generated on the user's device to the server via the network.

[0311] "Means for receiving" refers to software and hardware technology for receiving, displaying, and storing data from the server to the user terminal.

[0312] When referring to "generated video and proposed countermeasures," "generated video" refers to the visual image generated based on the simulation results, and "proposed countermeasures" refers to information on earthquake countermeasures proposed based on the generated video.

[0313] "Means for displaying on the user's terminal" refers to the technology for displaying the received data on the terminal screen in a form that the user can view.

[0314] The present invention relates to an earthquake countermeasure proposal system that takes into account the user's emotions. Specifically, the system generates earthquake simulation results based on emotion recognition by the user's terminal and operates to present individualized countermeasure proposals.

[0315] User device operations

[0316] Users launch a dedicated application using their device, such as a smartphone or tablet PC. Within the app, users first rotate the camera to capture the entire room to obtain 360-degree indoor video data. Once the video is complete, it is temporarily saved. Next, users enter the building's structural information through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This procedure improves the accuracy of the earthquake simulation.

[0317] Emotion Engine Operation

[0318] While the user is recording video and inputting building structure information, an emotion recognition engine installed on the user's device analyzes the user's facial expressions and tone of voice to collect emotional data. This emotional data is temporarily stored and sent to the server along with the analysis results. This allows the system to accurately identify the user's current emotional state (e.g., relief, anxiety, fear).

[0319] Sending data

[0320] The user device sends the acquired 360-degree video data, building structure information, and emotion data to a server. The data is securely transferred over the Internet. Data compression and encryption technologies are used for this transmission to ensure communication security.

[0321] Processing on the server

[0322] The server receives and temporarily stores the data sent by the user. The server then uses an AI model (e.g., TensorFlow or PyTorch) to analyze the 360-degree video data and building structure information. This analysis identifies the room layout and furniture placement, and evaluates the earthquake resistance of the building based on its structure.

[0323] Earthquake Simulation

[0324] Once the analysis is complete, the server uses the data to run an earthquake simulation, which recreates how furniture and appliances move and fall, and how books, tableware, and other items will fly apart. This process provides a concrete visualization of the risks that may occur during an earthquake.

[0325] Video Creation and Delivery

[0326] Based on the simulation results, the server uses CG technology to generate a simulation video. This video shows in detail what the interior of the building will look like when an earthquake occurs, how furniture will move, and how objects will be scattered. Furthermore, based on the analysis results of the emotion engine, messages and countermeasure information corresponding to the user's emotions are added to the video.

[0327] Display on user device

[0328] The generated video is then sent back to the user's device. During transmission, the data is compressed and optimized, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[0329] Emotion-based countermeasure proposals

[0330] When watching a video, the system will present specific earthquake countermeasures to the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will display specific countermeasures such as "Here's how you can easily secure unnecessary furniture" along with the video. It will also suggest countermeasure products, making it easier for the user to take immediate action.

[0331] Specific examples

[0332] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine determines that the user is feeling anxious, the system will display specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[0333] Prompt Sentence Examples

[0334] Generate a video of the earthquake simulation results. If the emotion engine recognizes the user's anxiety, generate a CG that provides a specific message of reassurance and presents simple countermeasures.

[0335] In this way, the present invention aims to improve safety and reduce damage by providing personalized earthquake countermeasures based on the user's emotions.

[0336] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0337] Step 1:

[0338] Acquiring data from user devices

[0339] Users launch a dedicated application on their device, such as a smartphone or tablet PC, and capture 360-degree indoor video data. They rotate the device's camera to capture the entire room, and then enter building structural information (materials, age, number of floors, earthquake resistance, etc.) into the app. This data is temporarily saved.

[0340] Input: 360-degree video data, building structure information

[0341] Output: Temporarily saved 360-degree video data and building structure information

[0342] Step 2:

[0343] Acquiring emotion data

[0344] While the user is entering data, an emotion recognition engine installed on the user's device analyzes the user's facial expressions and tone of voice, and emotional data is collected and temporarily stored.

[0345] Input: User's facial expression data, tone of voice

[0346] Output: Temporarily saved emotion data

[0347] Step 3:

[0348] Sending data to the server

[0349] The user device transmits the captured 360-degree video data, building structure information, and emotion data to a server, where the data is securely transferred over the Internet and compressed and encrypted.

[0350] Input: 360-degree video data, building structure information, emotion data

[0351] Output: Data sent to the server

[0352] Step 4:

[0353] Data reception and storage on the server

[0354] The server receives the data sent by the user and temporarily stores it.

[0355] Input: 360-degree video data, building structure information, emotion data

[0356] Output: Temporarily saved data

[0357] Step 5:

[0358] Data analysis

[0359] The server uses AI models to analyze the received 360-degree video data and building structure information, which then evaluates the room layout, furniture placement, and earthquake resistance of the building based on its structure.

[0360] Input: 360-degree video data, building structure information

[0361] Output: Room layout, furniture placement, and earthquake resistance evaluation results

[0362] Step 6:

[0363] Running an earthquake simulation

[0364] The server then uses the analysis results to run an earthquake simulation, which recreates how furniture and appliances move and fall, and how books, tableware, and other items will fly apart.

[0365] Input: Room layout, furniture arrangement, earthquake resistance evaluation results

[0366] Output: Earthquake simulation results

[0367] Step 7:

[0368] Visualization of simulation results

[0369] Based on the simulation results, the server uses CG technology to generate a simulation video. The video shows in detail what the interior of the building will look like when an earthquake occurs, how furniture will move, and how objects will be scattered. Based on the analysis results of the emotion engine, countermeasures corresponding to the user's emotions are added to the video.

[0370] Input: Earthquake simulation results, emotion data

[0371] Output: Simulation video, video with countermeasures based on emotions

[0372] Step 8:

[0373] Video and countermeasures are sent from the server to the user's device

[0374] The server then sends the generated simulation video and countermeasures to the user's device, where data is compressed and optimized.

[0375] Input: Simulation video, video with emotion-based countermeasures

[0376] Output: Video and countermeasures sent to the user's device

[0377] Step 9:

[0378] Video playback on user devices and suggestions for solutions

[0379] The user device receives the video and proposed solutions sent from the server and notifies the user within the app. The user opens the app, watches the generated simulation video, and confirms the proposed solutions.

[0380] Input: Video and countermeasures sent from the server

[0381] Output: Videos watched by users and identified solutions

[0382] Through the above steps, the present invention can visualize specific risks during an earthquake for the user and provide customized countermeasures based on the user's emotions.

[0383] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0384] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0385] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0386] [Second embodiment]

[0387] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0388] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0389] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0390] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0391] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0392] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0393] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0394] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0395] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0396] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0397] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0398] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0399] This system allows users to collect 360-degree indoor video data, send it to a server along with building structural information, and then analyze the data and perform earthquake simulations on the server, generating the results as videos and providing them to users. To implement this system, a user terminal, a server, and software that connects them are required.

[0400] 1. User device operation

[0401] Users launch a dedicated app on their smartphone, tablet, or other device. In the app, the user first takes a picture of the entire room to obtain 360-degree video data of the room. At this time, the camera rotates to record video in all directions.

[0402] Next, users input building structural information through the app, including materials (e.g., wood, reinforced concrete), age, number of floors, seismic performance, etc. This improves the accuracy of the simulation.

[0403] 2. Data transmission

[0404] The user device sends the captured 360-degree video data and building structure information to a server, where the data is securely transferred over the Internet.

[0405] 3. Processing on the Server

[0406] The server receives the data sent by the user and analyzes it using an AI model. Specifically, it analyzes 360-degree video data to identify the room layout and furniture placement, and simulates shaking patterns during an earthquake based on building structural information.

[0407] 4. Earthquake Simulation

[0408] After the server analyzes the data, it runs an earthquake simulation based on the analysis results. The simulation recreates how furniture and appliances move and fall, and how books and tableware will fly apart. This process visualizes the risks in the event of an earthquake.

[0409] 5. Video Creation and Distribution

[0410] Based on the simulation results, the server uses CG technology to generate a video that shows in detail what the user's interior will be like when an earthquake occurs, and how furniture and objects will move.

[0411] The generated video is then sent back to the user's device, where it is displayed on the app and viewed by the user.

[0412] 6. User Notification and Recommendations

[0413] Users will receive a notification from the app that a simulation video has been prepared. By watching the video, they can gain a concrete understanding of the risks involved in an earthquake and recognize what countermeasures are necessary. The system also suggests countermeasure items, making it easier for users to immediately implement countermeasures.

[0414] Specific examples

[0415] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. By watching this video on the app, the user can consider specific measures to take, such as securing furniture and organizing unnecessary items.

[0416] In this way, the present invention encourages users to concretely understand the risks during earthquakes and take appropriate measures, thereby improving safety and reducing damage.

[0417] The processing flow will be explained below.

[0418] Step 1:

[0419] The user launches the dedicated app on their device. Following the app's instructions, the user rotates the camera to capture the entire room in order to obtain 360-degree video data of the indoor space. Once the capture is complete, the video data is temporarily saved.

[0420] Step 2:

[0421] On the user's device, the user enters building structural information into an input form within the app. Specifically, the user enters information such as the building's materials (e.g., wood, reinforced concrete), age, number of floors, and earthquake resistance. The entered information is temporarily saved along with the video data.

[0422] Step 3:

[0423] The user device sends the collected 360-degree video data and building structure information to a server. The data is securely transferred over the Internet. The data is compressed and encrypted as needed.

[0424] Step 4:

[0425] The server temporarily stores the received 360-degree video data and building structure information, then uses AI models to analyze the data, identify room layouts and furniture placements, and evaluate the building's earthquake resistance based on its structure.

[0426] Step 5:

[0427] The server then runs an earthquake simulation based on the analysis results. Specifically, it specifies the seismic intensity setting and calculates how shaking corresponding to that intensity will affect the space. This reproduces how furniture and objects will move, fall, and scatter.

[0428] Step 6:

[0429] The server generates a video based on the simulation results. The video includes the state of the interior of the building at the time of the earthquake, the movement of furniture, and the scattering of objects. The video is visualized intuitively to make it easy for users to understand.

[0430] Step 7:

[0431] The server then sends the generated video to the user's device, where it is compressed, optimized, and converted into a format that can be played on the user's device.

[0432] Step 8:

[0433] The user device receives the video sent from the server and notifies the user within the app, allowing the user to open the app and watch the generated simulation video.

[0434] Step 9:

[0435] After watching the video, the user device will present the user with specific earthquake countermeasures, including suggestions for securing furniture and emergency supplies, to help users take immediate action.

[0436] Through these steps, the present invention provides a system that allows users to concretely understand the risks during an earthquake and encourages them to take specific actions to take appropriate measures.

[0437] Example 1

[0438] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0439] Conventional earthquake countermeasures have had the problem that risk assessment taking into account the specific structure and furniture layout of a home is difficult, and general guidelines and countermeasures alone cannot ensure sufficient safety. In particular, in order for users to specifically understand the situation in their home and take appropriate countermeasures, simulations that reproduce the interior and furniture layout of the home are necessary. However, conducting such simulations individually requires high costs and specialized knowledge, making it difficult for many ordinary users to implement.

[0440] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0441] In this invention, the server includes means for acquiring indoor omnidirectional video data via a user terminal, means for inputting building structural information via the user terminal, means for transmitting the omnidirectional video data and the building structural information from the user terminal to the server, means for analyzing the received data using a generative machine learning model in the server, means for executing an earthquake simulation based on the analysis results in the server, means for generating a video that visualizes the simulation results in the server, means for transmitting the generated video from the server to the user terminal, and means for displaying the received video in the user terminal. This enables users to visually understand risks through earthquake simulation videos that recreate specific conditions in their homes and take appropriate measures.

[0442] A "user terminal" is an information device that can be operated by an individual, and typically takes the form of a smartphone or tablet.

[0443] "Omnidirectional video data" refers to video data captured from a 360-degree perspective, recording the entire area around the user.

[0444] "Building structure information" is data that includes detailed information about the building's structure, such as the building's materials, age, number of floors, and earthquake resistance.

[0445] "Transmission means" refers to the technical means used to send data from one point to another, typically referring to communications protocols over the Internet.

[0446] A "generative machine learning model" is a model that contains algorithms trained to perform advanced tasks such as data analysis and prediction.

[0447] "Earthquake simulation" refers to a virtual experiment that simulates the shaking and movement of buildings and furniture in the event of an earthquake.

[0448] "Means for generating animation" refers to technical means for creating animation-format data to visually represent analysis results or simulation results.

[0449] "Means of display" refers to the technical means by which users can visually confirm information, and is primarily a display or screen.

[0450] "Specific earthquake countermeasures" refers to information that includes suggestions for specific actions and countermeasures that users should take in the event of an earthquake.

[0451] "Disaster prevention supplies" refer to items designed to reduce risk, such as furniture fasteners and disaster prevention kits, which are used in the event of an earthquake.

[0452] This system allows users to acquire indoor omnidirectional video data, send it to a server along with building structural information, and then analyze the data and perform earthquake simulations on the server, generating the results as videos and providing them to users. To implement this system, a user terminal, a server, and software that links these together are required.

[0453] User device operations

[0454] The user launches the dedicated app on a user device such as a smartphone or tablet. When the app is launched for the first time, it obtains necessary permissions from the user, such as permission for the camera and internet connection. Next, the user uses the app's camera function to capture an image of the entire room to collect indoor omnidirectional video data. The user rotates the camera to record omnidirectional video. The user then enters building structural information into a form within the app. This structural information includes the building's material (e.g., wood or reinforced concrete), age, number of floors, and earthquake resistance.

[0455] Sending data

[0456] The user device combines the collected omnidirectional video data and the input building structure information into a single data package, which is then encrypted using the SSL / TLS protocol and sent to a server via the Internet.

[0457] Processing on the server

[0458] The server receives the data package sent from the user device and analyzes it using a generative machine learning model (e.g., TensorFlow or PyTorch). The analysis includes identifying the room layout and furniture placement based on the omnidirectional video data, and evaluating the building's seismic performance based on the building's structural information.

[0459] Earthquake Simulation

[0460] The server sets up a simulation using earthquake simulation software (e.g., OpenSees or FLAC) based on the analysis data. Simulation parameters include seismic intensity, seismic waveform, and ground conditions. The server runs the simulation under the set earthquake conditions and calculates the movement and fall of furniture and home appliances, as well as the scattering of books, tableware, etc.

[0461] Video Creation and Delivery

[0462] The server generates a video using CG software (e.g., Blender or Maya) based on the simulation results. The generated video includes scenes of furniture moving around and objects flying around. The server compresses the video file and sends it to the user's device. The video file is transferred using a secure protocol (e.g., HTTPS).

[0463] User notification and suggested solutions

[0464] The user device receives a notification that the video received from the server is ready. The app's notification function informs the user that the simulation results are available for viewing. The user plays the simulation video within the app to confirm specific earthquake risks. The system then recommends countermeasures (e.g., furniture fasteners, disaster prevention kits) and helps the user implement countermeasures immediately.

[0465] Specific examples

[0466] For example, if a user takes a photo of their living room with an omnidirectional camera and enters the building's structural information as "wooden construction" and "20 years old," the user's device will send this data to the server. The server will analyze the data and run a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. The user can watch this video in the app and consider specific measures, such as securing furniture and organizing unnecessary items.

[0467] Example prompt sentence:

[0468] "Take a 360-degree video of your living room and enter the building's structural information. For example, 'wooden structure' and '20 years old'. Then send the data to the server."

[0469] In this way, the present invention helps users to concretely understand the risks during earthquakes and take appropriate measures, thereby improving safety and reducing damage.

[0470] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0471] Step 1:

[0472] The user launches a dedicated app on a device such as a smartphone or tablet. When the app is launched for the first time, it obtains necessary permissions from the user, such as camera and internet access. Based on this, the device obtains camera access permission and internet access permission. This becomes the input.

[0473] What happens: The user launches the app and sees a popup asking for permission.

[0474] Step 2:

[0475] The user uses the camera function in the app to collect indoor omnidirectional video data. The user rotates the camera to capture the entire room, and the device saves this video as omnidirectional video data. This is the input, and the output is omnidirectional video data.

[0476] Specific operation: The user stands in the center of the room and rotates the camera horizontally to take a picture.

[0477] Step 3:

[0478] Users enter building structural information into a form within the app. This structural information includes materials (wood, reinforced concrete, etc.), age of the building, number of floors, and earthquake resistance. The device receives this input, organizes the data, and saves it. This is the input, and the output is the building structural information.

[0479] What happens: The user enters information using text boxes and drop-down menus and presses the "Submit" button.

[0480] Step 4:

[0481] The user terminal combines the collected omnidirectional video data and the input building structure information into a single data package, then encrypts the data package using the SSL / TLS protocol and sends it to the server. This is the input, and the output is the encrypted data package.

[0482] Specific operation: The app automatically packages the data, and when you press the "Send" button, the encrypted data is sent to the server.

[0483] Step 5:

[0484] The server receives the data package sent from the user terminal, decompresses it, and obtains the omnidirectional video data and building structure information. This is the input, and the output is the decompressed omnidirectional video data and building structure information.

[0485] Specific operation: The server receives the data package and automatically begins decompressing and analyzing it.

[0486] Step 6:

[0487] The server uses a generative machine learning model (e.g., TensorFlow or PyTorch) to analyze the omnidirectional video data. The server uses this analysis to determine the room layout and furniture placement. This is the input, and the output is the room layout information and furniture placement information.

[0488] Specific operation: The server inputs omnidirectional video data into the AI ​​model and generates layout information as the analysis result.

[0489] Step 7:

[0490] The server further evaluates the seismic performance based on the building structural information. From this evaluation, the server obtains an evaluation result of the building's susceptibility to shaking and seismic performance. This is the input, and the output is the result of the seismic performance evaluation.

[0491] Specific operation: The server processes building structural information and evaluates the structural earthquake resistance using numerical values ​​and indicators.

[0492] Step 8:

[0493] The server sets up and runs a simulation using earthquake simulation software (e.g., OpenSees or FLAC) based on the analysis data and seismic performance evaluation. The server calculates the movement and fall of furniture and appliances, and the scattering of books and tableware. This is the input, and the output is the results of the earthquake simulation.

[0494] Specific operation: The server runs the simulation software and performs a simulation on the 3D model.

[0495] Step 9:

[0496] The server generates a video using CG software (e.g., Blender or Maya) based on the simulation results. The server creates a video that visually represents the movement of furniture and objects. This is the input, and the output is the generated simulation video.

[0497] Specific operation: The server inputs the simulation data into the CG software and renders the animation.

[0498] Step 10:

[0499] The server compresses the generated video and sends it to the user's device. The video file is transferred using a secure protocol (e.g. HTTPS). This is the input, and the output is the compressed video transmission.

[0500] Specific operation: The server compresses the video file and sends it to the user's device.

[0501] Step 11:

[0502] The user device receives a notification from the server that the video is ready. The user plays the simulation video in the app. This is the input, and the output is a notification to the user and the video playback.

[0503] Specific behavior: The device displays a notification, and the user presses the "play" button in the app to play the video.

[0504] Step 12:

[0505] The system displays information during the video, including suggestions for countermeasures (e.g., furniture fixings, disaster prevention kits). This helps users to easily take specific countermeasures. This is the input, and the output is the display of countermeasure suggestion information.

[0506] How it works: The app displays a simulation video and recommended countermeasures products on the screen.

[0507] (Application example 1)

[0508] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0509] The present invention relates to a simulation system for specifically recognizing earthquake risks, and further aims to utilize this system to provide users with a means of effectively proposing disaster prevention products and insurance services, enabling them to take prompt and appropriate countermeasures. Another objective is to make it easier for users to ensure safety and security in their lives by obtaining useful suggestions directly from the earthquake simulation results.

[0510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0511] In this invention, the server includes a means for generating advertisements based on the analysis results, a means for inserting advertisements into videos, and a means for adding information including suggestions for disaster prevention goods and insurance services to the videos generated based on the analysis results. As a result, the user can receive specific suggestions for effective disaster prevention goods and insurance services while viewing the simulation results, enabling them to easily and quickly take earthquake countermeasures.

[0512] A "user terminal" is an information and communication device operated by a user, including smartphones and tablets.

[0513] "360-degree video data" refers to video data that records and displays the entirety of a specific location from every direction.

[0514] "Building structural information" refers to information about a building, including the building's materials, age, number of floors, earthquake resistance, etc.

[0515] A "generative model" refers to an artificial intelligence model that analyzes 360-degree video data and building structural information to generate the data that serves as the basis for simulations.

[0516] "Analysis" refers to the process of extracting and understanding specific information from received data.

[0517] "Earthquake simulation" refers to the process of predicting and virtually recreating the shaking and movement of objects that occur during an earthquake.

[0518] "Visualization video" refers to a video generated to display the results of analysis or simulation in a visually easy-to-understand format.

[0519] "Means for generating advertisements" refers to a mechanism that automatically creates advertisements for users, such as countermeasures products and insurance services, based on the analysis results.

[0520] "Emergency goods" refer to items used to reduce damage during earthquakes.

[0521] "Insurance services" refers to insurance products to cover damage caused by earthquakes.

[0522] "Insertion means" refers to the technical means used to add advertising or suggestion information to the generated video.

[0523] This invention is a system that allows users to collect 360-degree video data of their homes and rooms, as well as structural information about the building, using a dedicated app, and then sends that data to a server to perform earthquake simulations, visualize the risks, and insert advertisements into the video, including suggestions for disaster prevention products and insurance services.

[0524] User device operations

[0525] Users launch a dedicated app on their smartphone, tablet, or other device to first capture 360-degree indoor video data. At this time, the camera rotates to record video in all directions. Next, the user enters the building's structural information through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This improves the accuracy of the simulation.

[0526] Sending data

[0527] The user device sends the acquired 360-degree video data and building structure information to a server via the Internet, where the data is transferred securely.

[0528] Processing on the server

[0529] The server receives the data sent by the user and analyzes it using a generative model. Specifically, it analyzes the 360-degree video data to determine the room layout and furniture placement, and simulates the shaking patterns during an earthquake based on building structural information. Based on the simulation results, a video is generated using CG technology. The generated video shows in detail what state the user's indoors will be in when an earthquake occurs, and how furniture and objects will move.

[0530] Ad generation and insertion

[0531] Furthermore, the server generates advertisements for users based on the analysis results, including suggestions for countermeasures and insurance services. These advertisements are inserted into the simulation video. The generated final video is then sent back to the user's device.

[0532] Watching videos and suggesting solutions

[0533] The received video is displayed on the app on the user's device. By watching the simulation video, users can gain a concrete understanding of the risks involved in an earthquake and recognize what countermeasures are necessary. The system also suggests countermeasure products and insurance services, making it easier for users to take immediate action.

[0534] Hardware and software used

[0535] The hardware uses a smartphone and a 360-degree camera, and the software uses the requests library (for sending HTTP requests), the moviepy library (for video editing), and an AI model for earthquake simulation on the server side.

[0536] Examples of specific examples and prompts

[0537] Specific examples

[0538] Users launch a dedicated smartphone app and take a 360-degree photo of their living room. They then input the building's structural information, such as "wooden construction," "20 years old," "two stories," and "average earthquake resistance." This information is sent to a server, and a video containing advertisements for "earthquake-resistant mats," "earthquake insurance," and "furniture tip-prevention products" is displayed based on the simulation results.

[0539] Prompt Sentence Examples

[0540] Based on 360-degree video data taken by users and building structural information, simulate shaking during an earthquake and the movement of furniture and objects, and reflect the results in your advertising video. Specifically, simulate with high accuracy how furniture will fall and objects will fly off, and recommend products and services to users to mitigate the risk.

[0541] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0542] Step 1:

[0543] The user launches a dedicated app on a user device such as a smartphone or tablet. The user acquires 360-degree video data of the indoor space, rotating the camera to record video in all directions. The input is the video data acquired from the 360-degree camera, and the output is 360-degree video data stored on the device.

[0544] Step 2:

[0545] Users input building structural information through the app, including materials (e.g., wood, reinforced concrete), age, number of floors, earthquake resistance, etc. The input is the building structural information entered by the user, and the output is the building structural information stored on the device.

[0546] Step 3:

[0547] The user device transmits the acquired 360-degree video data and building structure information to a server via the Internet. The input is the 360-degree video data and building structure information stored on the device, and the output is the data transmitted to the server.

[0548] Step 4:

[0549] The server receives the data sent by the user and analyzes it using a generative AI model. Specifically, it analyzes the 360-degree video data to identify the room layout and furniture placement, and simulates the shaking patterns during an earthquake based on building structural information. The input is the sent 360-degree video data and building structural information, and the output is the analysis and simulation results.

[0550] Step 5:

[0551] The server uses CG technology to generate an earthquake simulation video based on the simulation results. The generated video shows in detail what state the user's interior will be in when an earthquake occurs, and how furniture and objects will move. The input is the simulation results, and the output is the generated simulation video.

[0552] Step 6:

[0553] Furthermore, based on the analysis results, the server generates advertisements for users, including suggestions for countermeasures and insurance services, and inserts these advertisements into the simulation video. The inputs are the simulation video and advertisement information, and the output is the final video with the advertisements inserted.

[0554] Step 7:

[0555] The server then sends the generated final video back to the user terminal. The input is the final video, and the output is the video sent to the user terminal.

[0556] Step 8:

[0557] The final video received is displayed on the app on the user's device and the user is allowed to watch it. By watching the simulation video, the user can concretely understand the risks in the event of an earthquake and recognize what countermeasures are necessary. The input is the final video received, and the output is the user's viewing and consideration of countermeasures.

[0558] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0559] This system uses an emotion engine installed in a user device to recognize the user's emotions and propose earthquake countermeasures based on those emotions. It operates by combining the user device, server, emotion engine, and software that links them together.

[0560] 1. User device operation

[0561] Users launch a dedicated app using their device, such as a smartphone or tablet. In the app, users first rotate the camera to capture the entire room to obtain 360-degree video data of the interior. Once the video is complete, the video data is temporarily saved. Next, users enter structural information about the building through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This improves the accuracy of the simulation.

[0562] 2. Operation of the Emotion Engine

[0563] While the user is recording video and inputting building structure information, the emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, and input. This data is analyzed to identify the user's current emotional state (e.g., relief, anxiety, fear). The emotion data is temporarily stored and sent to the server along with the analysis results.

[0564] 3. Data transmission

[0565] The user device transmits the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted as appropriate.

[0566] 4. Processing on the Server

[0567] The server receives and temporarily stores the data sent by the user, then uses an AI model to analyze the 360-degree video data and building structure information to determine room layout and furniture placement, and evaluate the earthquake resistance of the building based on its structure.

[0568] 5. Earthquake Simulation

[0569] After the server analyzes the data, it runs an earthquake simulation based on the analysis results. The simulation recreates how furniture and appliances move and fall, and how books and tableware will fly apart. This process visualizes the risks in the event of an earthquake.

[0570] 6. Video Creation and Distribution

[0571] Based on the simulation results, the server uses CG technology to generate a video. This video shows in detail what the interior of a building looks like when an earthquake occurs, how furniture moves, and how scattered objects are. Based on the analysis results of the emotion engine, messages and countermeasures information corresponding to the user's emotions are displayed in the video and within the app.

[0572] 7. Display on user device

[0573] The generated video is then sent back to the user's device. During transmission, the data is compressed and optimized, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[0574] 8. Emotion-based countermeasures

[0575] Users will receive a notification from the app that a simulation video has been prepared. As they watch the video, the system will customize and present specific earthquake preparedness measures to the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will suggest a more reassuring message and easy-to-implement measures. It will also suggest emergency supplies, making it easier for users to take measures immediately.

[0576] Specific examples

[0577] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine determines that the user is feeling anxious, the system will display specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[0578] In this way, the present invention helps users concretely understand the risks during earthquakes and implement appropriate countermeasures, thereby improving safety and reducing damage. A customized approach that takes into account the user's emotions can make the implementation of countermeasure actions even more effective.

[0579] The processing flow will be explained below.

[0580] Step 1:

[0581] The user launches a dedicated app on their smartphone or tablet. Following the app's instructions, the user rotates the camera to capture the entire room, capturing 360-degree video data of the indoor space. Once the video is complete, it is temporarily saved on the user's device.

[0582] Step 2:

[0583] Users input building structural information into the app, such as the building's materials (wood, reinforced concrete, etc.), age, number of floors, and earthquake resistance. The input building structural information is temporarily saved on the user's device along with the video data.

[0584] Step 3:

[0585] The emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, etc. This process is carried out while the user is entering information, and determines whether the user is feeling safe, anxious, or scared. The emotion data is analyzed and temporarily stored.

[0586] Step 4:

[0587] The user device sends the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted during transmission.

[0588] Step 5:

[0589] The server receives the 360-degree video data, building structure information, and emotion data sent by the user. The server temporarily stores this data and then begins the analysis process. First, it analyzes the 360-degree video data to identify the room layout and furniture placement.

[0590] Step 6:

[0591] The server evaluates the earthquake resistance of a building based on structural information, taking into account information such as the building's materials and age, and predicts the shaking pattern and extent of impact during an earthquake.

[0592] Step 7:

[0593] The server then runs an earthquake simulation based on the analysis results. The simulation reproduces how furniture and home appliances will move and fall, and how books, tableware, etc. will fly apart in response to a specified seismic intensity (for example, intensity 6). The simulation results are then generated.

[0594] Step 8:

[0595] The server uses CG technology to generate a video based on the simulation results, which shows in detail what the interior of a building looks like when an earthquake strikes, how furniture moves, and how objects are scattered.

[0596] Step 9:

[0597] Based on the analysis results of the emotion engine, the server inserts emotional messages and countermeasures information into videos and apps. For example, if a user is feeling anxious, the system will display a message designed to reassure them.

[0598] Step 10:

[0599] The server then sends the generated video to the user's device, where it is converted into a playable format and appropriately compressed.

[0600] Step 11:

[0601] The user's device receives the video sent from the server and notifies the user within the app. Upon receiving the notification, the user can open the app and watch the generated simulation video.

[0602] Step 12:

[0603] After watching the video, the user's device will present customized earthquake countermeasures based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will detail easy-to-implement countermeasures and display a message to provide reassurance. It will also suggest necessary countermeasure goods.

[0604] In this way, this system supports users in taking concrete and effective earthquake countermeasures while taking their emotions into consideration. The generated videos and customized countermeasures help users to concretely understand the risks of earthquakes and take appropriate measures.

[0605] Example 2

[0606] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0607] Conventional earthquake simulation systems propose earthquake countermeasures without considering the user's emotions or psychological state, which makes it difficult to alleviate users' anxiety and reduces the rate at which countermeasures are implemented. Furthermore, the system does not provide sufficient suggestions for specific countermeasures or the goods needed to implement them, so the proposals are not in a form that is easy for users to implement immediately.

[0608] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0609] A means for acquiring user emotion data by a user terminal;

[0610] A means for transmitting 360-degree video data, building structure information, and emotion data from a user terminal to a server;

[0611] A means for analyzing the received data by an AI model in a server;

[0612] This makes it possible to provide specific and feasible earthquake countermeasures that take into account the user's emotional state.

[0613] A "user terminal" is a portable information processing device operated by a user, such as a smartphone or tablet.

[0614] "360-degree video data" is data that digitally records visual information in all directions captured by a rotating camera.

[0615] "Building structural information" refers to data on the physical characteristics of a building, such as the building's materials, age, number of floors, and earthquake resistance.

[0616] "Emotional data" is data that indicates the user's psychological state, analyzed from the user's facial expressions, tone of voice, etc.

[0617] A "server" is a computer system that receives data from user terminals via a network and analyzes and processes it.

[0618] An "AI model" is a program that uses artificial intelligence algorithms to analyze data and make predictions and simulations.

[0619] "Earthquake simulation" is a computational process that recreates in a virtual environment the conditions inside a building during an earthquake.

[0620] "Movie" refers to dynamic video data generated to visually visualize the simulation results.

[0621] "Countermeasures proposals" are proposals that specifically outline the actions users should take and the preparations they should make in the event of an earthquake.

[0622] "Earthquake prevention goods" refer to equipment and devices necessary for earthquake prevention, which help users implement earthquake prevention measures.

[0623] This invention is a system that uses an emotion engine installed in a user terminal to recognize a user's emotions and proposes earthquake countermeasures based on those emotions. This system operates by combining a user terminal, a server, an emotion engine, and software that links these together.

[0624] First, the user launches a dedicated app on their device, such as a smartphone or tablet. To obtain 360-degree indoor video data within the app, the user rotates the camera to capture the entire room. Once the video is complete, it is temporarily saved. Next, the user enters the building's structural information. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and earthquake resistance.

[0625] Next, while the user records video and inputs building structure information, the emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, and input content. This data is analyzed to identify the user's current emotional state (e.g., relief, anxiety, fear). The emotion data is temporarily stored and sent to the server along with the analysis results.

[0626] The user device transmits the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted as appropriate.

[0627] The server receives the data sent by the user and temporarily stores it. The server then uses an AI model to analyze the 360-degree video data and building structural information to determine room layout and furniture placement, and evaluate the building's earthquake resistance based on its structure. Specific technologies used include image processing algorithms and machine learning models. This provides the basic data needed to accurately simulate risks in the event of an earthquake.

[0628] Once the analysis is complete, the server runs an earthquake simulation based on the results. The simulation recreates how furniture and appliances will move and fall, and how books, tableware, etc. will fly apart. This provides a visually easy-to-understand display of specific earthquake risks.

[0629] Based on the simulation results, the server uses CG technology to generate a video. This video depicts in detail the state of the room at the time of the earthquake, the movement of furniture, and the scattering of objects. In addition, based on the analysis results of the emotion engine, messages and countermeasure information corresponding to the user's emotions are displayed in the video and within the app. For example, if the user is feeling anxious, a reassuring message and easy-to-implement countermeasure information are displayed.

[0630] The generated video is sent from the server to the user's device. The data is compressed and optimized before being sent, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[0631] When users receive a notification from the app that a simulation video has been prepared and watch the video, the system will customize and present specific earthquake preparedness measures to the user based on the analysis results of the emotion engine. If the user is feeling anxious, the system will suggest messages that will provide more reassurance and easy-to-implement measures. It will also suggest emergency supplies, making it easier for users to take measures immediately.

[0632] As a specific example, a user takes a 360-degree photo of their living room and enters the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine recognizes that the user is feeling anxious, the system displays specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[0633] An example of a prompt for the generative AI model is, "If it recognizes that the user is feeling anxious, please generate a message suggesting specific measures to reduce risks in the event of an earthquake." Using this prompt, the AI ​​model can generate appropriate countermeasure information according to the user's emotional state.

[0634] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0635] Step 1:

[0636] The user launches a dedicated app using a user device such as a smartphone or tablet. The device rotates the camera to capture the entire room, allowing the user to obtain 360-degree video data of the room within the app. The input in this step is the user's operation and video capture by the camera, and the output is 360-degree video data. The user then operates the device appropriately to rotate the camera correctly and obtain a 360-degree video image of the room.

[0637] Step 2:

[0638] The user inputs structural information about the building through a dedicated app. The device temporarily stores the information entered by the user, such as the building's materials (wood, reinforced concrete, etc.), age, number of floors, and seismic performance, in a database. The input in this step is detailed building information manually entered by the user, and the output is the temporarily stored building structural information.

[0639] Step 3:

[0640] The device uses a front camera and microphone to capture the user's facial expressions and tone of voice. The emotion engine analyzes this data in real time to generate the user's emotion data. The input in this step is the user's facial and voice data, and the output is the analyzed emotion data.

[0641] Step 4:

[0642] The device assembles the acquired 360-degree video data, building structure information, and emotion data into packets. These packets are then compressed and encrypted. The device then sends these packets to a server via the Internet. The input in this step is the individual data (360-degree video data, building structure information, emotion data), and the output is the encrypted data packet sent to the server.

[0643] Step 5:

[0644] The server receives the data packet sent from the terminal and checks the integrity of the content. If it is normal, it temporarily stores the data. The input in this step is the encrypted data packet, and the output is the temporarily stored data set.

[0645] Step 6:

[0646] The server uses an AI model to analyze the 360-degree video data and building structural information. Specifically, it uses image processing algorithms to identify room layouts and furniture placements and evaluate the building's earthquake resistance. The input for this step is the temporarily stored dataset (360-degree video data, building structural information), and the output is the analysis results.

[0647] Step 7:

[0648] The server runs an earthquake simulation based on the analysis results. The simulation calculates how the effects of a virtual earthquake will be transmitted to furniture and structures, and generates specific simulation data. The input in this step is the analysis results, and the output is simulation data.

[0649] Step 8:

[0650] The server uses computer graphics technology to generate a video that visualizes the indoor situation during an earthquake using the simulation data. Furthermore, based on the analysis results of the emotion engine, messages and countermeasure information for users are added to the video. The inputs for this step are the simulation data and emotion data, and the output is the generated video.

[0651] Step 9:

[0652] The server compresses the generated video and sends it to the user's device. By converting it into the appropriate format, the video can be played back without any problems on the user's device. The input of this step is the generated video, and the output is a compressed video file.

[0653] Step 10:

[0654] The device receives the video sent from the server and notifies the user. The input in this step is a compressed video file, and the output is a video file stored on the device and a notification to the user. The user opens the app and watches the generated simulation video.

[0655] Step 11:

[0656] The device provides a function in which the system suggests specific earthquake countermeasures based on the user's emotional data while watching the simulation video. If the user feels anxious, specific countermeasures are displayed. The input in this step is the emotional data and the generated video, and the output is a video display containing the countermeasures.

[0657] Through these steps, this system will encourage users to take earthquake countermeasures, thereby improving safety and reducing damage.

[0658] (Application example 2)

[0659] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] Conventional earthquake countermeasure proposal systems present uniform countermeasure proposals without considering the user's emotions, making it difficult to provide specific countermeasures that respond to the psychological state and needs of each individual user. Furthermore, when visualizing earthquake simulation results, the provision of countermeasure proposals to deepen users' understanding or information that gives them a sense of security is insufficient. This reduces users' motivation to actually implement countermeasures, making it difficult to improve safety.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0662] In this invention, the server includes means for acquiring emotion data using an emotion recognition engine in the user terminal, means for generating information including countermeasures corresponding to the user's emotions based on a video generated by the server, and means for transmitting the generated video and countermeasures from the server to the user terminal. This makes it possible to provide individualized countermeasures corresponding to the user's emotions, allowing the user to take specific and feasible earthquake countermeasures with greater peace of mind.

[0663] A "user terminal" is an electronic device operated by a user, including portable devices such as smartphones and tablet PCs.

[0664] "Indoor 360-degree video data" refers to video data of an indoor environment captured by a user in all directions, visually recording the entire room.

[0665] "Building structural information" refers to information about the structure of a building, such as its materials, age, number of floors, and earthquake resistance.

[0666] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and tone of voice to identify the user's emotional state.

[0667] "Server" refers to a computer system for analyzing, processing, storing, transmitting and receiving data.

[0668] An "AI model" is a model that uses machine learning and deep learning techniques to analyze input data and has algorithms to perform specified tasks.

[0669] "Earthquake simulation" is the process of recreating on a computer what the interior of a building would look like when an earthquake occurs, based on building structural information and 360-degree video data.

[0670] "Videos that visualize simulation results" refers to animations and videos generated to visually show the results of earthquake simulations.

[0671] "Countermeasures" refers to information that includes specific actions, recommended items, and initiatives proposed to reduce earthquake risk.

[0672] "Means of transmission" refers to the software and hardware technology used to send data acquired or generated on the user's device to the server via the network.

[0673] "Means for receiving" refers to software and hardware technology for receiving, displaying, and storing data from the server to the user terminal.

[0674] When referring to "generated video and proposed countermeasures," "generated video" refers to the visual image generated based on the simulation results, and "proposed countermeasures" refers to information on earthquake countermeasures proposed based on the generated video.

[0675] "Means for displaying on the user's terminal" refers to the technology for displaying the received data on the terminal screen in a form that the user can view.

[0676] The present invention relates to an earthquake countermeasure proposal system that takes into account the user's emotions. Specifically, the system generates earthquake simulation results based on emotion recognition by the user's terminal and operates to present individualized countermeasure proposals.

[0677] User device operations

[0678] Users launch a dedicated application using their device, such as a smartphone or tablet PC. Within the app, users first rotate the camera to capture the entire room to obtain 360-degree indoor video data. Once the video is complete, it is temporarily saved. Next, users enter the building's structural information through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This procedure improves the accuracy of the earthquake simulation.

[0679] Emotion Engine Operation

[0680] While the user is recording video and inputting building structure information, an emotion recognition engine installed on the user's device analyzes the user's facial expressions and tone of voice to collect emotional data. This emotional data is temporarily stored and sent to the server along with the analysis results. This allows the system to accurately identify the user's current emotional state (e.g., relief, anxiety, fear).

[0681] Sending data

[0682] The user device sends the acquired 360-degree video data, building structure information, and emotion data to a server. The data is securely transferred over the Internet. Data compression and encryption technologies are used for this transmission to ensure communication security.

[0683] Processing on the server

[0684] The server receives and temporarily stores the data sent by the user. The server then uses an AI model (e.g., TensorFlow or PyTorch) to analyze the 360-degree video data and building structure information. This analysis identifies the room layout and furniture placement, and evaluates the earthquake resistance of the building based on its structure.

[0685] Earthquake Simulation

[0686] Once the analysis is complete, the server uses the data to run an earthquake simulation, which recreates how furniture and appliances move and fall, and how books, tableware, and other items will fly apart. This process provides a concrete visualization of the risks that may occur during an earthquake.

[0687] Video Creation and Delivery

[0688] Based on the simulation results, the server uses CG technology to generate a simulation video. This video shows in detail what the interior of the building will look like when an earthquake occurs, how furniture will move, and how objects will be scattered. Furthermore, based on the analysis results of the emotion engine, messages and countermeasure information corresponding to the user's emotions are added to the video.

[0689] Display on user device

[0690] The generated video is then sent back to the user's device. During transmission, the data is compressed and optimized, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[0691] Emotion-based countermeasure proposals

[0692] When watching a video, the system will present specific earthquake countermeasures to the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will display specific countermeasures such as "Here's how you can easily secure unnecessary furniture" along with the video. It will also suggest countermeasure products, making it easier for the user to take immediate action.

[0693] Specific examples

[0694] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine determines that the user is feeling anxious, the system will display specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[0695] Prompt Sentence Examples

[0696] Generate a video of the earthquake simulation results. If the emotion engine recognizes the user's anxiety, generate a CG that provides a specific message of reassurance and presents simple countermeasures.

[0697] In this way, the present invention aims to improve safety and reduce damage by providing personalized earthquake countermeasures based on the user's emotions.

[0698] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0699] Step 1:

[0700] Acquiring data from user devices

[0701] Users launch a dedicated application on their device, such as a smartphone or tablet PC, and capture 360-degree indoor video data. They rotate the device's camera to capture the entire room, and then enter building structural information (materials, age, number of floors, earthquake resistance, etc.) into the app. This data is temporarily saved.

[0702] Input: 360-degree video data, building structure information

[0703] Output: Temporarily saved 360-degree video data and building structure information

[0704] Step 2:

[0705] Acquiring emotion data

[0706] While the user is entering data, an emotion recognition engine installed on the user's device analyzes the user's facial expressions and tone of voice, and emotional data is collected and temporarily stored.

[0707] Input: User's facial expression data, tone of voice

[0708] Output: Temporarily saved emotion data

[0709] Step 3:

[0710] Sending data to the server

[0711] The user device transmits the captured 360-degree video data, building structure information, and emotion data to a server, where the data is securely transferred over the Internet and compressed and encrypted.

[0712] Input: 360-degree video data, building structure information, emotion data

[0713] Output: Data sent to the server

[0714] Step 4:

[0715] Data reception and storage on the server

[0716] The server receives the data sent by the user and temporarily stores it.

[0717] Input: 360-degree video data, building structure information, emotion data

[0718] Output: Temporarily saved data

[0719] Step 5:

[0720] Data analysis

[0721] The server uses AI models to analyze the received 360-degree video data and building structure information, which then evaluates the room layout, furniture placement, and earthquake resistance of the building based on its structure.

[0722] Input: 360-degree video data, building structure information

[0723] Output: Room layout, furniture placement, and earthquake resistance evaluation results

[0724] Step 6:

[0725] Running an earthquake simulation

[0726] The server then uses the analysis results to run an earthquake simulation, which recreates how furniture and appliances move and fall, and how books, tableware, and other items will fly apart.

[0727] Input: Room layout, furniture arrangement, earthquake resistance evaluation results

[0728] Output: Earthquake simulation results

[0729] Step 7:

[0730] Visualization of simulation results

[0731] Based on the simulation results, the server uses CG technology to generate a simulation video. The video shows in detail what the interior of the building will look like when an earthquake occurs, how furniture will move, and how objects will be scattered. Based on the analysis results of the emotion engine, countermeasures corresponding to the user's emotions are added to the video.

[0732] Input: Earthquake simulation results, emotion data

[0733] Output: Simulation video, video with countermeasures based on emotions

[0734] Step 8:

[0735] Video and countermeasures are sent from the server to the user's device

[0736] The server then sends the generated simulation video and countermeasures to the user's device, where data is compressed and optimized.

[0737] Input: Simulation video, video with emotion-based countermeasures

[0738] Output: Video and countermeasures sent to the user's device

[0739] Step 9:

[0740] Video playback on user devices and suggestions for solutions

[0741] The user device receives the video and proposed solutions sent from the server and notifies the user within the app. The user opens the app, watches the generated simulation video, and confirms the proposed solutions.

[0742] Input: Video and countermeasures sent from the server

[0743] Output: Videos watched by users and identified solutions

[0744] Through the above steps, the present invention can visualize specific risks during an earthquake for the user and provide customized countermeasures based on the user's emotions.

[0745] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0746] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0747] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0748] [Third embodiment]

[0749] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0750] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0751] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0752] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0753] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0754] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0755] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0756] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0757] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0758] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0759] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0760] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0761] This system allows users to collect 360-degree indoor video data, send it to a server along with building structural information, and then analyze the data and perform earthquake simulations on the server, generating the results as videos and providing them to users. To implement this system, a user terminal, a server, and software that connects them are required.

[0762] 1. User device operation

[0763] Users launch a dedicated app on their smartphone, tablet, or other device. In the app, the user first takes a picture of the entire room to obtain 360-degree video data of the room. At this time, the camera rotates to record video in all directions.

[0764] Next, users input building structural information through the app, including materials (e.g., wood, reinforced concrete), age, number of floors, seismic performance, etc. This improves the accuracy of the simulation.

[0765] 2. Data transmission

[0766] The user device sends the captured 360-degree video data and building structure information to a server, where the data is securely transferred over the Internet.

[0767] 3. Processing on the Server

[0768] The server receives the data sent by the user and analyzes it using an AI model. Specifically, it analyzes 360-degree video data to identify the room layout and furniture placement, and simulates shaking patterns during an earthquake based on building structural information.

[0769] 4. Earthquake Simulation

[0770] After the server analyzes the data, it runs an earthquake simulation based on the analysis results. The simulation recreates how furniture and appliances move and fall, and how books and tableware will fly apart. This process visualizes the risks in the event of an earthquake.

[0771] 5. Video Creation and Distribution

[0772] Based on the simulation results, the server uses CG technology to generate a video that shows in detail what the user's interior will be like when an earthquake occurs, and how furniture and objects will move.

[0773] The generated video is then sent back to the user's device, where it is displayed on the app and viewed by the user.

[0774] 6. User Notification and Recommendations

[0775] Users will receive a notification from the app that a simulation video has been prepared. By watching the video, they can gain a concrete understanding of the risks involved in an earthquake and recognize what countermeasures are necessary. The system also suggests countermeasure items, making it easier for users to immediately implement countermeasures.

[0776] Specific examples

[0777] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. By watching this video on the app, the user can consider specific measures to take, such as securing furniture and organizing unnecessary items.

[0778] In this way, the present invention encourages users to concretely understand the risks during earthquakes and take appropriate measures, thereby improving safety and reducing damage.

[0779] The processing flow will be explained below.

[0780] Step 1:

[0781] The user launches the dedicated app on their device. Following the app's instructions, the user rotates the camera to capture the entire room in order to obtain 360-degree video data of the indoor space. Once the capture is complete, the video data is temporarily saved.

[0782] Step 2:

[0783] On the user's device, the user enters building structural information into an input form within the app. Specifically, the user enters information such as the building's materials (e.g., wood, reinforced concrete), age, number of floors, and earthquake resistance. The entered information is temporarily saved along with the video data.

[0784] Step 3:

[0785] The user device sends the collected 360-degree video data and building structure information to a server. The data is securely transferred over the Internet. The data is compressed and encrypted as needed.

[0786] Step 4:

[0787] The server temporarily stores the received 360-degree video data and building structure information, then uses AI models to analyze the data, identify room layouts and furniture placements, and evaluate the building's earthquake resistance based on its structure.

[0788] Step 5:

[0789] The server then runs an earthquake simulation based on the analysis results. Specifically, it specifies the seismic intensity setting and calculates how shaking corresponding to that intensity will affect the space. This reproduces how furniture and objects will move, fall, and scatter.

[0790] Step 6:

[0791] The server generates a video based on the simulation results. The video includes the state of the interior of the building at the time of the earthquake, the movement of furniture, and the scattering of objects. The video is visualized intuitively to make it easy for users to understand.

[0792] Step 7:

[0793] The server then sends the generated video to the user's device, where it is compressed, optimized, and converted into a format that can be played on the user's device.

[0794] Step 8:

[0795] The user device receives the video sent from the server and notifies the user within the app, allowing the user to open the app and watch the generated simulation video.

[0796] Step 9:

[0797] After watching the video, the user device will present the user with specific earthquake countermeasures, including suggestions for securing furniture and emergency supplies, to help users take immediate action.

[0798] Through these steps, the present invention provides a system that allows users to concretely understand the risks during an earthquake and encourages them to take specific actions to take appropriate measures.

[0799] Example 1

[0800] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0801] Conventional earthquake countermeasures have had the problem that risk assessment taking into account the specific structure and furniture layout of a home is difficult, and general guidelines and countermeasures alone cannot ensure sufficient safety. In particular, in order for users to specifically understand the situation in their home and take appropriate countermeasures, simulations that reproduce the interior and furniture layout of the home are necessary. However, conducting such simulations individually requires high costs and specialized knowledge, making it difficult for many ordinary users to implement.

[0802] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0803] In this invention, the server includes means for acquiring indoor omnidirectional video data via a user terminal, means for inputting building structural information via the user terminal, means for transmitting the omnidirectional video data and the building structural information from the user terminal to the server, means for analyzing the received data using a generative machine learning model in the server, means for executing an earthquake simulation based on the analysis results in the server, means for generating a video that visualizes the simulation results in the server, means for transmitting the generated video from the server to the user terminal, and means for displaying the received video in the user terminal. This enables users to visually understand risks through earthquake simulation videos that recreate specific conditions in their homes and take appropriate measures.

[0804] A "user terminal" is an information device that can be operated by an individual, and typically takes the form of a smartphone or tablet.

[0805] "Omnidirectional video data" refers to video data captured from a 360-degree perspective, recording the entire area around the user.

[0806] "Building structure information" is data that includes detailed information about the building's structure, such as the building's materials, age, number of floors, and earthquake resistance.

[0807] "Transmission means" refers to the technical means used to send data from one point to another, typically referring to communications protocols over the Internet.

[0808] A "generative machine learning model" is a model that contains algorithms trained to perform advanced tasks such as data analysis and prediction.

[0809] "Earthquake simulation" refers to a virtual experiment that simulates the shaking and movement of buildings and furniture in the event of an earthquake.

[0810] "Means for generating animation" refers to technical means for creating animation-format data to visually represent analysis results or simulation results.

[0811] "Means of display" refers to the technical means by which users can visually confirm information, and is primarily a display or screen.

[0812] "Specific earthquake countermeasures" refers to information that includes suggestions for specific actions and countermeasures that users should take in the event of an earthquake.

[0813] "Disaster prevention supplies" refer to items designed to reduce risk, such as furniture fasteners and disaster prevention kits, which are used in the event of an earthquake.

[0814] This system allows users to acquire indoor omnidirectional video data, send it to a server along with building structural information, and then analyze the data and perform earthquake simulations on the server, generating the results as videos and providing them to users. To implement this system, a user terminal, a server, and software that links these together are required.

[0815] User device operations

[0816] The user launches the dedicated app on a user device such as a smartphone or tablet. When the app is launched for the first time, it obtains necessary permissions from the user, such as permission for the camera and internet connection. Next, the user uses the app's camera function to capture an image of the entire room to collect indoor omnidirectional video data. The user rotates the camera to record omnidirectional video. The user then enters building structural information into a form within the app. This structural information includes the building's material (e.g., wood or reinforced concrete), age, number of floors, and earthquake resistance.

[0817] Sending data

[0818] The user device combines the collected omnidirectional video data and the input building structure information into a single data package, which is then encrypted using the SSL / TLS protocol and sent to a server via the Internet.

[0819] Processing on the server

[0820] The server receives the data package sent from the user device and analyzes it using a generative machine learning model (e.g., TensorFlow or PyTorch). The analysis includes identifying the room layout and furniture placement based on the omnidirectional video data, and evaluating the building's seismic performance based on the building's structural information.

[0821] Earthquake Simulation

[0822] The server sets up a simulation using earthquake simulation software (e.g., OpenSees or FLAC) based on the analysis data. Simulation parameters include seismic intensity, seismic waveform, and ground conditions. The server runs the simulation under the set earthquake conditions and calculates the movement and fall of furniture and home appliances, as well as the scattering of books, tableware, etc.

[0823] Video Creation and Delivery

[0824] The server generates a video using CG software (e.g., Blender or Maya) based on the simulation results. The generated video includes scenes of furniture moving around and objects flying around. The server compresses the video file and sends it to the user's device. The video file is transferred using a secure protocol (e.g., HTTPS).

[0825] User notification and suggested solutions

[0826] The user device receives a notification that the video received from the server is ready. The app's notification function informs the user that the simulation results are available for viewing. The user plays the simulation video within the app to confirm specific earthquake risks. The system then recommends countermeasures (e.g., furniture fasteners, disaster prevention kits) and helps the user implement countermeasures immediately.

[0827] Specific examples

[0828] For example, if a user takes a photo of their living room with an omnidirectional camera and enters the building's structural information as "wooden construction" and "20 years old," the user's device will send this data to the server. The server will analyze the data and run a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. The user can watch this video in the app and consider specific measures, such as securing furniture and organizing unnecessary items.

[0829] Example prompt sentence:

[0830] "Take a 360-degree video of your living room and enter the building's structural information. For example, 'wooden structure' and '20 years old'. Then send the data to the server."

[0831] In this way, the present invention helps users to concretely understand the risks during earthquakes and take appropriate measures, thereby improving safety and reducing damage.

[0832] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0833] Step 1:

[0834] The user launches a dedicated app on a device such as a smartphone or tablet. When the app is launched for the first time, it obtains necessary permissions from the user, such as camera and internet access. Based on this, the device obtains camera access permission and internet access permission. This becomes the input.

[0835] What happens: The user launches the app and sees a popup asking for permission.

[0836] Step 2:

[0837] The user uses the camera function in the app to collect indoor omnidirectional video data. The user rotates the camera to capture the entire room, and the device saves this video as omnidirectional video data. This is the input, and the output is omnidirectional video data.

[0838] Specific operation: The user stands in the center of the room and rotates the camera horizontally to take a picture.

[0839] Step 3:

[0840] Users enter building structural information into a form within the app. This structural information includes materials (wood, reinforced concrete, etc.), age of the building, number of floors, and earthquake resistance. The device receives this input, organizes the data, and saves it. This is the input, and the output is the building structural information.

[0841] What happens: The user enters information using text boxes and drop-down menus and presses the "Submit" button.

[0842] Step 4:

[0843] The user terminal combines the collected omnidirectional video data and the input building structure information into a single data package, then encrypts the data package using the SSL / TLS protocol and sends it to the server. This is the input, and the output is the encrypted data package.

[0844] Specific operation: The app automatically packages the data, and when you press the "Send" button, the encrypted data is sent to the server.

[0845] Step 5:

[0846] The server receives the data package sent from the user terminal, decompresses it, and obtains the omnidirectional video data and building structure information. This is the input, and the output is the decompressed omnidirectional video data and building structure information.

[0847] Specific operation: The server receives the data package and automatically begins decompressing and analyzing it.

[0848] Step 6:

[0849] The server uses a generative machine learning model (e.g., TensorFlow or PyTorch) to analyze the omnidirectional video data. The server uses this analysis to determine the room layout and furniture placement. This is the input, and the output is the room layout information and furniture placement information.

[0850] Specific operation: The server inputs omnidirectional video data into the AI ​​model and generates layout information as the analysis result.

[0851] Step 7:

[0852] The server further evaluates the seismic performance based on the building structural information. From this evaluation, the server obtains an evaluation result of the building's susceptibility to shaking and seismic performance. This is the input, and the output is the result of the seismic performance evaluation.

[0853] Specific operation: The server processes building structural information and evaluates the structural earthquake resistance using numerical values ​​and indicators.

[0854] Step 8:

[0855] The server sets up and runs a simulation using earthquake simulation software (e.g., OpenSees or FLAC) based on the analysis data and seismic performance evaluation. The server calculates the movement and fall of furniture and appliances, and the scattering of books and tableware. This is the input, and the output is the results of the earthquake simulation.

[0856] Specific operation: The server runs the simulation software and performs a simulation on the 3D model.

[0857] Step 9:

[0858] The server generates a video using CG software (e.g., Blender or Maya) based on the simulation results. The server creates a video that visually represents the movement of furniture and objects. This is the input, and the output is the generated simulation video.

[0859] Specific operation: The server inputs the simulation data into the CG software and renders the animation.

[0860] Step 10:

[0861] The server compresses the generated video and sends it to the user's device. The video file is transferred using a secure protocol (e.g. HTTPS). This is the input, and the output is the compressed video transmission.

[0862] Specific operation: The server compresses the video file and sends it to the user's device.

[0863] Step 11:

[0864] The user device receives a notification from the server that the video is ready. The user plays the simulation video in the app. This is the input, and the output is a notification to the user and the video playback.

[0865] Specific behavior: The device displays a notification, and the user presses the "play" button in the app to play the video.

[0866] Step 12:

[0867] The system displays information during the video, including suggestions for countermeasures (e.g., furniture fixings, disaster prevention kits). This helps users to easily take specific countermeasures. This is the input, and the output is the display of countermeasure suggestion information.

[0868] How it works: The app displays a simulation video and recommended countermeasures products on the screen.

[0869] (Application example 1)

[0870] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0871] The present invention relates to a simulation system for specifically recognizing earthquake risks, and further aims to utilize this system to provide users with a means of effectively proposing disaster prevention products and insurance services, enabling them to take prompt and appropriate countermeasures. Another objective is to make it easier for users to ensure safety and security in their lives by obtaining useful suggestions directly from the earthquake simulation results.

[0872] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0873] In this invention, the server includes a means for generating advertisements based on the analysis results, a means for inserting advertisements into videos, and a means for adding information including suggestions for disaster prevention goods and insurance services to the videos generated based on the analysis results. As a result, the user can receive specific suggestions for effective disaster prevention goods and insurance services while viewing the simulation results, enabling them to easily and quickly take earthquake countermeasures.

[0874] A "user terminal" is an information and communication device operated by a user, including smartphones and tablets.

[0875] "360-degree video data" refers to video data that records and displays the entirety of a specific location from every direction.

[0876] "Building structural information" refers to information about a building, including the building's materials, age, number of floors, earthquake resistance, etc.

[0877] A "generative model" refers to an artificial intelligence model that analyzes 360-degree video data and building structural information to generate the data that serves as the basis for simulations.

[0878] "Analysis" refers to the process of extracting and understanding specific information from received data.

[0879] "Earthquake simulation" refers to the process of predicting and virtually recreating the shaking and movement of objects that occur during an earthquake.

[0880] "Visualization video" refers to a video generated to display the results of analysis or simulation in a visually easy-to-understand format.

[0881] "Means for generating advertisements" refers to a mechanism that automatically creates advertisements for users, such as countermeasures products and insurance services, based on the analysis results.

[0882] "Emergency goods" refer to items used to reduce damage during earthquakes.

[0883] "Insurance services" refers to insurance products to cover damage caused by earthquakes.

[0884] "Insertion means" refers to the technical means used to add advertising or suggestion information to the generated video.

[0885] This invention is a system that allows users to collect 360-degree video data of their homes and rooms, as well as structural information about the building, using a dedicated app, and then sends that data to a server to perform earthquake simulations, visualize the risks, and insert advertisements into the video, including suggestions for disaster prevention products and insurance services.

[0886] User device operations

[0887] Users launch a dedicated app on their smartphone, tablet, or other device to first capture 360-degree indoor video data. At this time, the camera rotates to record video in all directions. Next, the user enters the building's structural information through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This improves the accuracy of the simulation.

[0888] Sending data

[0889] The user device sends the acquired 360-degree video data and building structure information to a server via the Internet, where the data is transferred securely.

[0890] Processing on the server

[0891] The server receives the data sent by the user and analyzes it using a generative model. Specifically, it analyzes the 360-degree video data to determine the room layout and furniture placement, and simulates the shaking patterns during an earthquake based on building structural information. Based on the simulation results, a video is generated using CG technology. The generated video shows in detail what state the user's indoors will be in when an earthquake occurs, and how furniture and objects will move.

[0892] Ad generation and insertion

[0893] Furthermore, the server generates advertisements for users based on the analysis results, including suggestions for countermeasures and insurance services. These advertisements are inserted into the simulation video. The generated final video is then sent back to the user's device.

[0894] Watching videos and suggesting solutions

[0895] The received video is displayed on the app on the user's device. By watching the simulation video, users can gain a concrete understanding of the risks involved in an earthquake and recognize what countermeasures are necessary. The system also suggests countermeasure products and insurance services, making it easier for users to take immediate action.

[0896] Hardware and software used

[0897] The hardware uses a smartphone and a 360-degree camera, and the software uses the requests library (for sending HTTP requests), the moviepy library (for video editing), and an AI model for earthquake simulation on the server side.

[0898] Examples of specific examples and prompts

[0899] Specific examples

[0900] Users launch a dedicated smartphone app and take a 360-degree photo of their living room. They then input the building's structural information, such as "wooden construction," "20 years old," "two stories," and "average earthquake resistance." This information is sent to a server, and a video containing advertisements for "earthquake-resistant mats," "earthquake insurance," and "furniture tip-prevention products" is displayed based on the simulation results.

[0901] Prompt Sentence Examples

[0902] Based on 360-degree video data taken by users and building structural information, simulate shaking during an earthquake and the movement of furniture and objects, and reflect the results in your advertising video. Specifically, simulate with high accuracy how furniture will fall and objects will fly off, and recommend products and services to users to mitigate the risk.

[0903] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0904] Step 1:

[0905] The user launches a dedicated app on a user device such as a smartphone or tablet. The user acquires 360-degree video data of the indoor space, rotating the camera to record video in all directions. The input is the video data acquired from the 360-degree camera, and the output is 360-degree video data stored on the device.

[0906] Step 2:

[0907] Users input building structural information through the app, including materials (e.g., wood, reinforced concrete), age, number of floors, earthquake resistance, etc. The input is the building structural information entered by the user, and the output is the building structural information stored on the device.

[0908] Step 3:

[0909] The user device transmits the acquired 360-degree video data and building structure information to a server via the Internet. The input is the 360-degree video data and building structure information stored on the device, and the output is the data transmitted to the server.

[0910] Step 4:

[0911] The server receives the data sent by the user and analyzes it using a generative AI model. Specifically, it analyzes the 360-degree video data to identify the room layout and furniture placement, and simulates the shaking patterns during an earthquake based on building structural information. The input is the sent 360-degree video data and building structural information, and the output is the analysis and simulation results.

[0912] Step 5:

[0913] The server uses CG technology to generate an earthquake simulation video based on the simulation results. The generated video shows in detail what state the user's interior will be in when an earthquake occurs, and how furniture and objects will move. The input is the simulation results, and the output is the generated simulation video.

[0914] Step 6:

[0915] Furthermore, based on the analysis results, the server generates advertisements for users, including suggestions for countermeasures and insurance services, and inserts these advertisements into the simulation video. The inputs are the simulation video and advertisement information, and the output is the final video with the advertisements inserted.

[0916] Step 7:

[0917] The server then sends the generated final video back to the user terminal. The input is the final video, and the output is the video sent to the user terminal.

[0918] Step 8:

[0919] The final video received is displayed on the app on the user's device and the user is allowed to watch it. By watching the simulation video, the user can concretely understand the risks in the event of an earthquake and recognize what countermeasures are necessary. The input is the final video received, and the output is the user's viewing and consideration of countermeasures.

[0920] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0921] This system uses an emotion engine installed in a user device to recognize the user's emotions and propose earthquake countermeasures based on those emotions. It operates by combining the user device, server, emotion engine, and software that links them together.

[0922] 1. User device operation

[0923] Users launch a dedicated app using their device, such as a smartphone or tablet. In the app, users first rotate the camera to capture the entire room to obtain 360-degree video data of the interior. Once the video is complete, the video data is temporarily saved. Next, users enter structural information about the building through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This improves the accuracy of the simulation.

[0924] 2. Operation of the Emotion Engine

[0925] While the user is recording video and inputting building structure information, the emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, and input. This data is analyzed to identify the user's current emotional state (e.g., relief, anxiety, fear). The emotion data is temporarily stored and sent to the server along with the analysis results.

[0926] 3. Data transmission

[0927] The user device transmits the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted as appropriate.

[0928] 4. Processing on the Server

[0929] The server receives and temporarily stores the data sent by the user, then uses an AI model to analyze the 360-degree video data and building structure information to determine room layout and furniture placement, and evaluate the earthquake resistance of the building based on its structure.

[0930] 5. Earthquake Simulation

[0931] After the server analyzes the data, it runs an earthquake simulation based on the analysis results. The simulation recreates how furniture and appliances move and fall, and how books and tableware will fly apart. This process visualizes the risks in the event of an earthquake.

[0932] 6. Video Creation and Distribution

[0933] Based on the simulation results, the server uses CG technology to generate a video. This video shows in detail what the interior of a building looks like when an earthquake occurs, how furniture moves, and how scattered objects are. Based on the analysis results of the emotion engine, messages and countermeasures information corresponding to the user's emotions are displayed in the video and within the app.

[0934] 7. Display on user device

[0935] The generated video is then sent back to the user's device. During transmission, the data is compressed and optimized, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[0936] 8. Emotion-based countermeasures

[0937] Users will receive a notification from the app that a simulation video has been prepared. As they watch the video, the system will customize and present specific earthquake preparedness measures to the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will suggest a more reassuring message and easy-to-implement measures. It will also suggest emergency supplies, making it easier for users to take measures immediately.

[0938] Specific examples

[0939] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine determines that the user is feeling anxious, the system will display specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[0940] In this way, the present invention helps users concretely understand the risks during earthquakes and implement appropriate countermeasures, thereby improving safety and reducing damage. A customized approach that takes into account the user's emotions can make the implementation of countermeasure actions even more effective.

[0941] The processing flow will be explained below.

[0942] Step 1:

[0943] The user launches a dedicated app on their smartphone or tablet. Following the app's instructions, the user rotates the camera to capture the entire room, capturing 360-degree video data of the indoor space. Once the video is complete, it is temporarily saved on the user's device.

[0944] Step 2:

[0945] Users input building structural information into the app, such as the building's materials (wood, reinforced concrete, etc.), age, number of floors, and earthquake resistance. The input building structural information is temporarily saved on the user's device along with the video data.

[0946] Step 3:

[0947] The emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, etc. This process is carried out while the user is entering information, and determines whether the user is feeling safe, anxious, or scared. The emotion data is analyzed and temporarily stored.

[0948] Step 4:

[0949] The user device sends the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted during transmission.

[0950] Step 5:

[0951] The server receives the 360-degree video data, building structure information, and emotion data sent by the user. The server temporarily stores this data and then begins the analysis process. First, it analyzes the 360-degree video data to identify the room layout and furniture placement.

[0952] Step 6:

[0953] The server evaluates the earthquake resistance of a building based on structural information, taking into account information such as the building's materials and age, and predicts the shaking pattern and extent of impact during an earthquake.

[0954] Step 7:

[0955] The server then runs an earthquake simulation based on the analysis results. The simulation reproduces how furniture and home appliances will move and fall, and how books, tableware, etc. will fly apart in response to a specified seismic intensity (for example, intensity 6). The simulation results are then generated.

[0956] Step 8:

[0957] The server uses CG technology to generate a video based on the simulation results, which shows in detail what the interior of a building looks like when an earthquake strikes, how furniture moves, and how objects are scattered.

[0958] Step 9:

[0959] Based on the analysis results of the emotion engine, the server inserts emotional messages and countermeasures information into videos and apps. For example, if a user is feeling anxious, the system will display a message designed to reassure them.

[0960] Step 10:

[0961] The server then sends the generated video to the user's device, where it is converted into a playable format and appropriately compressed.

[0962] Step 11:

[0963] The user's device receives the video sent from the server and notifies the user within the app. Upon receiving the notification, the user can open the app and watch the generated simulation video.

[0964] Step 12:

[0965] After watching the video, the user's device will present customized earthquake countermeasures based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will detail easy-to-implement countermeasures and display a message to provide reassurance. It will also suggest necessary countermeasure goods.

[0966] In this way, this system supports users in taking concrete and effective earthquake countermeasures while taking their emotions into consideration. The generated videos and customized countermeasures help users to concretely understand the risks of earthquakes and take appropriate measures.

[0967] Example 2

[0968] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0969] Conventional earthquake simulation systems propose earthquake countermeasures without considering the user's emotions or psychological state, which makes it difficult to alleviate users' anxiety and reduces the rate at which countermeasures are implemented. Furthermore, the system does not provide sufficient suggestions for specific countermeasures or the goods needed to implement them, so the proposals are not in a form that is easy for users to implement immediately.

[0970] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0971] A means for acquiring user emotion data by a user terminal;

[0972] A means for transmitting 360-degree video data, building structure information, and emotion data from a user terminal to a server;

[0973] A means for analyzing the received data by an AI model in a server;

[0974] This makes it possible to provide specific and feasible earthquake countermeasures that take into account the user's emotional state.

[0975] A "user terminal" is a portable information processing device operated by a user, such as a smartphone or tablet.

[0976] "360-degree video data" is data that digitally records visual information in all directions captured by a rotating camera.

[0977] "Building structural information" refers to data on the physical characteristics of a building, such as the building's materials, age, number of floors, and earthquake resistance.

[0978] "Emotional data" is data that indicates the user's psychological state, analyzed from the user's facial expressions, tone of voice, etc.

[0979] A "server" is a computer system that receives data from user terminals via a network and analyzes and processes it.

[0980] An "AI model" is a program that uses artificial intelligence algorithms to analyze data and make predictions and simulations.

[0981] "Earthquake simulation" is a computational process that recreates in a virtual environment the conditions inside a building during an earthquake.

[0982] "Movie" refers to dynamic video data generated to visually visualize the simulation results.

[0983] "Countermeasures proposals" are proposals that specifically outline the actions users should take and the preparations they should make in the event of an earthquake.

[0984] "Earthquake prevention goods" refer to equipment and devices necessary for earthquake prevention, which help users implement earthquake prevention measures.

[0985] This invention is a system that uses an emotion engine installed in a user terminal to recognize a user's emotions and proposes earthquake countermeasures based on those emotions. This system operates by combining a user terminal, a server, an emotion engine, and software that links these together.

[0986] First, the user launches a dedicated app on their device, such as a smartphone or tablet. To obtain 360-degree indoor video data within the app, the user rotates the camera to capture the entire room. Once the video is complete, it is temporarily saved. Next, the user enters the building's structural information. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and earthquake resistance.

[0987] Next, while the user records video and inputs building structure information, the emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, and input content. This data is analyzed to identify the user's current emotional state (e.g., relief, anxiety, fear). The emotion data is temporarily stored and sent to the server along with the analysis results.

[0988] The user device transmits the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted as appropriate.

[0989] The server receives the data sent by the user and temporarily stores it. The server then uses an AI model to analyze the 360-degree video data and building structural information to determine room layout and furniture placement, and evaluate the building's earthquake resistance based on its structure. Specific technologies used include image processing algorithms and machine learning models. This provides the basic data needed to accurately simulate risks in the event of an earthquake.

[0990] Once the analysis is complete, the server runs an earthquake simulation based on the results. The simulation recreates how furniture and appliances will move and fall, and how books, tableware, etc. will fly apart. This provides a visually easy-to-understand display of specific earthquake risks.

[0991] Based on the simulation results, the server uses CG technology to generate a video. This video depicts in detail the state of the room at the time of the earthquake, the movement of furniture, and the scattering of objects. In addition, based on the analysis results of the emotion engine, messages and countermeasure information corresponding to the user's emotions are displayed in the video and within the app. For example, if the user is feeling anxious, a reassuring message and easy-to-implement countermeasure information are displayed.

[0992] The generated video is sent from the server to the user's device. The data is compressed and optimized before being sent, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[0993] When users receive a notification from the app that a simulation video has been prepared and watch the video, the system will customize and present specific earthquake preparedness measures to the user based on the analysis results of the emotion engine. If the user is feeling anxious, the system will suggest messages that will provide more reassurance and easy-to-implement measures. It will also suggest emergency supplies, making it easier for users to take measures immediately.

[0994] As a specific example, a user takes a 360-degree photo of their living room and enters the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine recognizes that the user is feeling anxious, the system displays specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[0995] An example of a prompt for the generative AI model is, "If it recognizes that the user is feeling anxious, please generate a message suggesting specific measures to reduce risks in the event of an earthquake." Using this prompt, the AI ​​model can generate appropriate countermeasure information according to the user's emotional state.

[0996] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0997] Step 1:

[0998] The user launches a dedicated app using a user device such as a smartphone or tablet. The device rotates the camera to capture the entire room, allowing the user to obtain 360-degree video data of the room within the app. The input in this step is the user's operation and video capture by the camera, and the output is 360-degree video data. The user then operates the device appropriately to rotate the camera correctly and obtain a 360-degree video image of the room.

[0999] Step 2:

[1000] The user inputs structural information about the building through a dedicated app. The device temporarily stores the information entered by the user, such as the building's materials (wood, reinforced concrete, etc.), age, number of floors, and seismic performance, in a database. The input in this step is detailed building information manually entered by the user, and the output is the temporarily stored building structural information.

[1001] Step 3:

[1002] The device uses a front camera and microphone to capture the user's facial expressions and tone of voice. The emotion engine analyzes this data in real time to generate the user's emotion data. The input in this step is the user's facial and voice data, and the output is the analyzed emotion data.

[1003] Step 4:

[1004] The device assembles the acquired 360-degree video data, building structure information, and emotion data into packets. These packets are then compressed and encrypted. The device then sends these packets to a server via the Internet. The input in this step is the individual data (360-degree video data, building structure information, emotion data), and the output is the encrypted data packet sent to the server.

[1005] Step 5:

[1006] The server receives the data packet sent from the terminal and checks the integrity of the content. If it is normal, it temporarily stores the data. The input in this step is the encrypted data packet, and the output is the temporarily stored data set.

[1007] Step 6:

[1008] The server uses an AI model to analyze the 360-degree video data and building structural information. Specifically, it uses image processing algorithms to identify room layouts and furniture placements and evaluate the building's earthquake resistance. The input for this step is the temporarily stored dataset (360-degree video data, building structural information), and the output is the analysis results.

[1009] Step 7:

[1010] The server runs an earthquake simulation based on the analysis results. The simulation calculates how the effects of a virtual earthquake will be transmitted to furniture and structures, and generates specific simulation data. The input in this step is the analysis results, and the output is simulation data.

[1011] Step 8:

[1012] The server uses computer graphics technology to generate a video that visualizes the indoor situation during an earthquake using the simulation data. Furthermore, based on the analysis results of the emotion engine, messages and countermeasure information for users are added to the video. The inputs for this step are the simulation data and emotion data, and the output is the generated video.

[1013] Step 9:

[1014] The server compresses the generated video and sends it to the user's device. By converting it into the appropriate format, the video can be played back without any problems on the user's device. The input of this step is the generated video, and the output is a compressed video file.

[1015] Step 10:

[1016] The device receives the video sent from the server and notifies the user. The input in this step is a compressed video file, and the output is a video file stored on the device and a notification to the user. The user opens the app and watches the generated simulation video.

[1017] Step 11:

[1018] The device provides a function in which the system suggests specific earthquake countermeasures based on the user's emotional data while watching the simulation video. If the user feels anxious, specific countermeasures are displayed. The input in this step is the emotional data and the generated video, and the output is a video display containing the countermeasures.

[1019] Through these steps, this system will encourage users to take earthquake countermeasures, thereby improving safety and reducing damage.

[1020] (Application example 2)

[1021] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1022] Conventional earthquake countermeasure proposal systems present uniform countermeasure proposals without considering the user's emotions, making it difficult to provide specific countermeasures that respond to the psychological state and needs of each individual user. Furthermore, when visualizing earthquake simulation results, the provision of countermeasure proposals to deepen users' understanding or information that gives them a sense of security is insufficient. This reduces users' motivation to actually implement countermeasures, making it difficult to improve safety.

[1023] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1024] In this invention, the server includes means for acquiring emotion data using an emotion recognition engine in the user terminal, means for generating information including countermeasures corresponding to the user's emotions based on a video generated by the server, and means for transmitting the generated video and countermeasures from the server to the user terminal. This makes it possible to provide individualized countermeasures corresponding to the user's emotions, allowing the user to take specific and feasible earthquake countermeasures with greater peace of mind.

[1025] A "user terminal" is an electronic device operated by a user, including portable devices such as smartphones and tablet PCs.

[1026] "Indoor 360-degree video data" refers to video data of an indoor environment captured by a user in all directions, visually recording the entire room.

[1027] "Building structural information" refers to information about the structure of a building, such as its materials, age, number of floors, and earthquake resistance.

[1028] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and tone of voice to identify the user's emotional state.

[1029] "Server" refers to a computer system for analyzing, processing, storing, transmitting and receiving data.

[1030] An "AI model" is a model that uses machine learning and deep learning techniques to analyze input data and has algorithms to perform specified tasks.

[1031] "Earthquake simulation" is the process of recreating on a computer what the interior of a building would look like when an earthquake occurs, based on building structural information and 360-degree video data.

[1032] "Videos that visualize simulation results" refers to animations and videos generated to visually show the results of earthquake simulations.

[1033] "Countermeasures" refers to information that includes specific actions, recommended items, and initiatives proposed to reduce earthquake risk.

[1034] "Means of transmission" refers to the software and hardware technology used to send data acquired or generated on the user's device to the server via the network.

[1035] "Means for receiving" refers to software and hardware technology for receiving, displaying, and storing data from the server to the user terminal.

[1036] When referring to "generated video and proposed countermeasures," "generated video" refers to the visual image generated based on the simulation results, and "proposed countermeasures" refers to information on earthquake countermeasures proposed based on the generated video.

[1037] "Means for displaying on the user's terminal" refers to the technology for displaying the received data on the terminal screen in a form that the user can view.

[1038] The present invention relates to an earthquake countermeasure proposal system that takes into account the user's emotions. Specifically, the system generates earthquake simulation results based on emotion recognition by the user's terminal and operates to present individualized countermeasure proposals.

[1039] User device operations

[1040] Users launch a dedicated application using their device, such as a smartphone or tablet PC. Within the app, users first rotate the camera to capture the entire room to obtain 360-degree indoor video data. Once the video is complete, it is temporarily saved. Next, users enter the building's structural information through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This procedure improves the accuracy of the earthquake simulation.

[1041] Emotion Engine Operation

[1042] While the user is recording video and inputting building structure information, an emotion recognition engine installed on the user's device analyzes the user's facial expressions and tone of voice to collect emotional data. This emotional data is temporarily stored and sent to the server along with the analysis results. This allows the system to accurately identify the user's current emotional state (e.g., relief, anxiety, fear).

[1043] Sending data

[1044] The user device sends the acquired 360-degree video data, building structure information, and emotion data to a server. The data is securely transferred over the Internet. Data compression and encryption technologies are used for this transmission to ensure communication security.

[1045] Processing on the server

[1046] The server receives and temporarily stores the data sent by the user. The server then uses an AI model (e.g., TensorFlow or PyTorch) to analyze the 360-degree video data and building structure information. This analysis identifies the room layout and furniture placement, and evaluates the earthquake resistance of the building based on its structure.

[1047] Earthquake Simulation

[1048] Once the analysis is complete, the server uses the data to run an earthquake simulation, which recreates how furniture and appliances move and fall, and how books, tableware, and other items will fly apart. This process provides a concrete visualization of the risks that may occur during an earthquake.

[1049] Video Creation and Delivery

[1050] Based on the simulation results, the server uses CG technology to generate a simulation video. This video shows in detail what the interior of the building will look like when an earthquake occurs, how furniture will move, and how objects will be scattered. Furthermore, based on the analysis results of the emotion engine, messages and countermeasure information corresponding to the user's emotions are added to the video.

[1051] Display on user device

[1052] The generated video is then sent back to the user's device. During transmission, the data is compressed and optimized, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[1053] Emotion-based countermeasure proposals

[1054] When watching a video, the system will present specific earthquake countermeasures to the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will display specific countermeasures such as "Here's how you can easily secure unnecessary furniture" along with the video. It will also suggest countermeasure products, making it easier for the user to take immediate action.

[1055] Specific examples

[1056] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine determines that the user is feeling anxious, the system will display specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[1057] Prompt Sentence Examples

[1058] Generate a video of the earthquake simulation results. If the emotion engine recognizes the user's anxiety, generate a CG that provides a specific message of reassurance and presents simple countermeasures.

[1059] In this way, the present invention aims to improve safety and reduce damage by providing personalized earthquake countermeasures based on the user's emotions.

[1060] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1061] Step 1:

[1062] Acquiring data from user devices

[1063] Users launch a dedicated application on their device, such as a smartphone or tablet PC, and capture 360-degree indoor video data. They rotate the device's camera to capture the entire room, and then enter building structural information (materials, age, number of floors, earthquake resistance, etc.) into the app. This data is temporarily saved.

[1064] Input: 360-degree video data, building structure information

[1065] Output: Temporarily saved 360-degree video data and building structure information

[1066] Step 2:

[1067] Acquiring emotion data

[1068] While the user is entering data, an emotion recognition engine installed on the user's device analyzes the user's facial expressions and tone of voice, and emotional data is collected and temporarily stored.

[1069] Input: User's facial expression data, tone of voice

[1070] Output: Temporarily saved emotion data

[1071] Step 3:

[1072] Sending data to the server

[1073] The user device transmits the captured 360-degree video data, building structure information, and emotion data to a server, where the data is securely transferred over the Internet and compressed and encrypted.

[1074] Input: 360-degree video data, building structure information, emotion data

[1075] Output: Data sent to the server

[1076] Step 4:

[1077] Data reception and storage on the server

[1078] The server receives the data sent by the user and temporarily stores it.

[1079] Input: 360-degree video data, building structure information, emotion data

[1080] Output: Temporarily saved data

[1081] Step 5:

[1082] Data analysis

[1083] The server uses AI models to analyze the received 360-degree video data and building structure information, which then evaluates the room layout, furniture placement, and earthquake resistance of the building based on its structure.

[1084] Input: 360-degree video data, building structure information

[1085] Output: Room layout, furniture placement, and earthquake resistance evaluation results

[1086] Step 6:

[1087] Running an earthquake simulation

[1088] The server then uses the analysis results to run an earthquake simulation, which recreates how furniture and appliances move and fall, and how books, tableware, and other items will fly apart.

[1089] Input: Room layout, furniture arrangement, earthquake resistance evaluation results

[1090] Output: Earthquake simulation results

[1091] Step 7:

[1092] Visualization of simulation results

[1093] Based on the simulation results, the server uses CG technology to generate a simulation video. The video shows in detail what the interior of the building will look like when an earthquake occurs, how furniture will move, and how objects will be scattered. Based on the analysis results of the emotion engine, countermeasures corresponding to the user's emotions are added to the video.

[1094] Input: Earthquake simulation results, emotion data

[1095] Output: Simulation video, video with countermeasures based on emotions

[1096] Step 8:

[1097] Video and countermeasures are sent from the server to the user's device

[1098] The server then sends the generated simulation video and countermeasures to the user's device, where data is compressed and optimized.

[1099] Input: Simulation video, video with emotion-based countermeasures

[1100] Output: Video and countermeasures sent to the user's device

[1101] Step 9:

[1102] Video playback on user devices and suggestions for solutions

[1103] The user device receives the video and proposed solutions sent from the server and notifies the user within the app. The user opens the app, watches the generated simulation video, and confirms the proposed solutions.

[1104] Input: Video and countermeasures sent from the server

[1105] Output: Videos watched by users and identified solutions

[1106] Through the above steps, the present invention can visualize specific risks during an earthquake for the user and provide customized countermeasures based on the user's emotions.

[1107] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1108] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1109] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1110] [Fourth embodiment]

[1111] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1112] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1113] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1114] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1115] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1116] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1117] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1118] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1119] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1120] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1121] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1122] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1123] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1124] This system allows users to collect 360-degree indoor video data, send it to a server along with building structural information, and then analyze the data and perform earthquake simulations on the server, generating the results as videos and providing them to users. To implement this system, a user terminal, a server, and software that connects them are required.

[1125] 1. User device operation

[1126] Users launch a dedicated app on their smartphone, tablet, or other device. In the app, the user first takes a picture of the entire room to obtain 360-degree video data of the room. At this time, the camera rotates to record video in all directions.

[1127] Next, users input building structural information through the app, including materials (e.g., wood, reinforced concrete), age, number of floors, seismic performance, etc. This improves the accuracy of the simulation.

[1128] 2. Data transmission

[1129] The user device sends the captured 360-degree video data and building structure information to a server, where the data is securely transferred over the Internet.

[1130] 3. Processing on the Server

[1131] The server receives the data sent by the user and analyzes it using an AI model. Specifically, it analyzes 360-degree video data to identify the room layout and furniture placement, and simulates shaking patterns during an earthquake based on building structural information.

[1132] 4. Earthquake Simulation

[1133] After the server analyzes the data, it runs an earthquake simulation based on the analysis results. The simulation recreates how furniture and appliances move and fall, and how books and tableware will fly apart. This process visualizes the risks in the event of an earthquake.

[1134] 5. Video Creation and Distribution

[1135] Based on the simulation results, the server uses CG technology to generate a video that shows in detail what the user's interior will be like when an earthquake occurs, and how furniture and objects will move.

[1136] The generated video is then sent back to the user's device, where it is displayed on the app and viewed by the user.

[1137] 6. User Notification and Recommendations

[1138] Users will receive a notification from the app that a simulation video has been prepared. By watching the video, they can gain a concrete understanding of the risks involved in an earthquake and recognize what countermeasures are necessary. The system also suggests countermeasure items, making it easier for users to immediately implement countermeasures.

[1139] Specific examples

[1140] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. By watching this video on the app, the user can consider specific measures to take, such as securing furniture and organizing unnecessary items.

[1141] In this way, the present invention encourages users to concretely understand the risks during earthquakes and take appropriate measures, thereby improving safety and reducing damage.

[1142] The processing flow will be explained below.

[1143] Step 1:

[1144] The user launches the dedicated app on their device. Following the app's instructions, the user rotates the camera to capture the entire room in order to obtain 360-degree video data of the indoor space. Once the capture is complete, the video data is temporarily saved.

[1145] Step 2:

[1146] On the user's device, the user enters building structural information into an input form within the app. Specifically, the user enters information such as the building's materials (e.g., wood, reinforced concrete), age, number of floors, and earthquake resistance. The entered information is temporarily saved along with the video data.

[1147] Step 3:

[1148] The user device sends the collected 360-degree video data and building structure information to a server. The data is securely transferred over the Internet. The data is compressed and encrypted as needed.

[1149] Step 4:

[1150] The server temporarily stores the received 360-degree video data and building structure information, then uses AI models to analyze the data, identify room layouts and furniture placements, and evaluate the building's earthquake resistance based on its structure.

[1151] Step 5:

[1152] The server then runs an earthquake simulation based on the analysis results. Specifically, it specifies the seismic intensity setting and calculates how shaking corresponding to that intensity will affect the space. This reproduces how furniture and objects will move, fall, and scatter.

[1153] Step 6:

[1154] The server generates a video based on the simulation results. The video includes the state of the interior of the building at the time of the earthquake, the movement of furniture, and the scattering of objects. The video is visualized intuitively to make it easy for users to understand.

[1155] Step 7:

[1156] The server then sends the generated video to the user's device, where it is compressed, optimized, and converted into a format that can be played on the user's device.

[1157] Step 8:

[1158] The user device receives the video sent from the server and notifies the user within the app, allowing the user to open the app and watch the generated simulation video.

[1159] Step 9:

[1160] After watching the video, the user device will present the user with specific earthquake countermeasures, including suggestions for securing furniture and emergency supplies, to help users take immediate action.

[1161] Through these steps, the present invention provides a system that allows users to concretely understand the risks during an earthquake and encourages them to take specific actions to take appropriate measures.

[1162] Example 1

[1163] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1164] Conventional earthquake countermeasures have had the problem that risk assessment taking into account the specific structure and furniture layout of a home is difficult, and general guidelines and countermeasures alone cannot ensure sufficient safety. In particular, in order for users to specifically understand the situation in their home and take appropriate countermeasures, simulations that reproduce the interior and furniture layout of the home are necessary. However, conducting such simulations individually requires high costs and specialized knowledge, making it difficult for many ordinary users to implement.

[1165] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1166] In this invention, the server includes means for acquiring indoor omnidirectional video data via a user terminal, means for inputting building structural information via the user terminal, means for transmitting the omnidirectional video data and the building structural information from the user terminal to the server, means for analyzing the received data using a generative machine learning model in the server, means for executing an earthquake simulation based on the analysis results in the server, means for generating a video that visualizes the simulation results in the server, means for transmitting the generated video from the server to the user terminal, and means for displaying the received video in the user terminal. This enables users to visually understand risks through earthquake simulation videos that recreate specific conditions in their homes and take appropriate measures.

[1167] A "user terminal" is an information device that can be operated by an individual, and typically takes the form of a smartphone or tablet.

[1168] "Omnidirectional video data" refers to video data captured from a 360-degree perspective, recording the entire area around the user.

[1169] "Building structure information" is data that includes detailed information about the building's structure, such as the building's materials, age, number of floors, and earthquake resistance.

[1170] "Transmission means" refers to the technical means used to send data from one point to another, typically referring to communications protocols over the Internet.

[1171] A "generative machine learning model" is a model that contains algorithms trained to perform advanced tasks such as data analysis and prediction.

[1172] "Earthquake simulation" refers to a virtual experiment that simulates the shaking and movement of buildings and furniture in the event of an earthquake.

[1173] "Means for generating animation" refers to technical means for creating animation-format data to visually represent analysis results or simulation results.

[1174] "Means of display" refers to the technical means by which users can visually confirm information, and is primarily a display or screen.

[1175] "Specific earthquake countermeasures" refers to information that includes suggestions for specific actions and countermeasures that users should take in the event of an earthquake.

[1176] "Disaster prevention supplies" refer to items designed to reduce risk, such as furniture fasteners and disaster prevention kits, which are used in the event of an earthquake.

[1177] This system allows users to acquire indoor omnidirectional video data, send it to a server along with building structural information, and then analyze the data and perform earthquake simulations on the server, generating the results as videos and providing them to users. To implement this system, a user terminal, a server, and software that links these together are required.

[1178] User device operations

[1179] The user launches the dedicated app on a user device such as a smartphone or tablet. When the app is launched for the first time, it obtains necessary permissions from the user, such as permission for the camera and internet connection. Next, the user uses the app's camera function to capture an image of the entire room to collect indoor omnidirectional video data. The user rotates the camera to record omnidirectional video. The user then enters building structural information into a form within the app. This structural information includes the building's material (e.g., wood or reinforced concrete), age, number of floors, and earthquake resistance.

[1180] Sending data

[1181] The user device combines the collected omnidirectional video data and the input building structure information into a single data package, which is then encrypted using the SSL / TLS protocol and sent to a server via the Internet.

[1182] Processing on the server

[1183] The server receives the data package sent from the user device and analyzes it using a generative machine learning model (e.g., TensorFlow or PyTorch). The analysis includes identifying the room layout and furniture placement based on the omnidirectional video data, and evaluating the building's seismic performance based on the building's structural information.

[1184] Earthquake Simulation

[1185] The server sets up a simulation using earthquake simulation software (e.g., OpenSees or FLAC) based on the analysis data. Simulation parameters include seismic intensity, seismic waveform, and ground conditions. The server runs the simulation under the set earthquake conditions and calculates the movement and fall of furniture and home appliances, as well as the scattering of books, tableware, etc.

[1186] Video Creation and Delivery

[1187] The server generates a video using CG software (e.g., Blender or Maya) based on the simulation results. The generated video includes scenes of furniture moving around and objects flying around. The server compresses the video file and sends it to the user's device. The video file is transferred using a secure protocol (e.g., HTTPS).

[1188] User notification and suggested solutions

[1189] The user device receives a notification that the video received from the server is ready. The app's notification function informs the user that the simulation results are available for viewing. The user plays the simulation video within the app to confirm specific earthquake risks. The system then recommends countermeasures (e.g., furniture fasteners, disaster prevention kits) and helps the user implement countermeasures immediately.

[1190] Specific examples

[1191] For example, if a user takes a photo of their living room with an omnidirectional camera and enters the building's structural information as "wooden construction" and "20 years old," the user's device will send this data to the server. The server will analyze the data and run a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. The user can watch this video in the app and consider specific measures, such as securing furniture and organizing unnecessary items.

[1192] Example prompt sentence:

[1193] "Take a 360-degree video of your living room and enter the building's structural information. For example, 'wooden structure' and '20 years old'. Then send the data to the server."

[1194] In this way, the present invention helps users to concretely understand the risks during earthquakes and take appropriate measures, thereby improving safety and reducing damage.

[1195] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1196] Step 1:

[1197] The user launches a dedicated app on a device such as a smartphone or tablet. When the app is launched for the first time, it obtains necessary permissions from the user, such as camera and internet access. Based on this, the device obtains camera access permission and internet access permission. This becomes the input.

[1198] What happens: The user launches the app and sees a popup asking for permission.

[1199] Step 2:

[1200] The user uses the camera function in the app to collect indoor omnidirectional video data. The user rotates the camera to capture the entire room, and the device saves this video as omnidirectional video data. This is the input, and the output is omnidirectional video data.

[1201] Specific operation: The user stands in the center of the room and rotates the camera horizontally to take a picture.

[1202] Step 3:

[1203] Users enter building structural information into a form within the app. This structural information includes materials (wood, reinforced concrete, etc.), age of the building, number of floors, and earthquake resistance. The device receives this input, organizes the data, and saves it. This is the input, and the output is the building structural information.

[1204] What happens: The user enters information using text boxes and drop-down menus and presses the "Submit" button.

[1205] Step 4:

[1206] The user terminal combines the collected omnidirectional video data and the input building structure information into a single data package, then encrypts the data package using the SSL / TLS protocol and sends it to the server. This is the input, and the output is the encrypted data package.

[1207] Specific operation: The app automatically packages the data, and when you press the "Send" button, the encrypted data is sent to the server.

[1208] Step 5:

[1209] The server receives the data package sent from the user terminal, decompresses it, and obtains the omnidirectional video data and building structure information. This is the input, and the output is the decompressed omnidirectional video data and building structure information.

[1210] Specific operation: The server receives the data package and automatically begins decompressing and analyzing it.

[1211] Step 6:

[1212] The server uses a generative machine learning model (e.g., TensorFlow or PyTorch) to analyze the omnidirectional video data. The server uses this analysis to determine the room layout and furniture placement. This is the input, and the output is the room layout information and furniture placement information.

[1213] Specific operation: The server inputs omnidirectional video data into the AI ​​model and generates layout information as the analysis result.

[1214] Step 7:

[1215] The server further evaluates the seismic performance based on the building structural information. From this evaluation, the server obtains an evaluation result of the building's susceptibility to shaking and seismic performance. This is the input, and the output is the result of the seismic performance evaluation.

[1216] Specific operation: The server processes building structural information and evaluates the structural earthquake resistance using numerical values ​​and indicators.

[1217] Step 8:

[1218] The server sets up and runs a simulation using earthquake simulation software (e.g., OpenSees or FLAC) based on the analysis data and seismic performance evaluation. The server calculates the movement and fall of furniture and appliances, and the scattering of books and tableware. This is the input, and the output is the results of the earthquake simulation.

[1219] Specific operation: The server runs the simulation software and performs a simulation on the 3D model.

[1220] Step 9:

[1221] The server generates a video using CG software (e.g., Blender or Maya) based on the simulation results. The server creates a video that visually represents the movement of furniture and objects. This is the input, and the output is the generated simulation video.

[1222] Specific operation: The server inputs the simulation data into the CG software and renders the animation.

[1223] Step 10:

[1224] The server compresses the generated video and sends it to the user's device. The video file is transferred using a secure protocol (e.g. HTTPS). This is the input, and the output is the compressed video transmission.

[1225] Specific operation: The server compresses the video file and sends it to the user's device.

[1226] Step 11:

[1227] The user device receives a notification from the server that the video is ready. The user plays the simulation video in the app. This is the input, and the output is a notification to the user and the video playback.

[1228] Specific behavior: The device displays a notification, and the user presses the "play" button in the app to play the video.

[1229] Step 12:

[1230] The system displays information during the video, including suggestions for countermeasures (e.g., furniture fixings, disaster prevention kits). This helps users to easily take specific countermeasures. This is the input, and the output is the display of countermeasure suggestion information.

[1231] How it works: The app displays a simulation video and recommended countermeasures products on the screen.

[1232] (Application example 1)

[1233] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1234] The present invention relates to a simulation system for specifically recognizing earthquake risks, and further aims to utilize this system to provide users with a means of effectively proposing disaster prevention products and insurance services, enabling them to take prompt and appropriate countermeasures. Another objective is to make it easier for users to ensure safety and security in their lives by obtaining useful suggestions directly from the earthquake simulation results.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1236] In this invention, the server includes a means for generating advertisements based on the analysis results, a means for inserting advertisements into videos, and a means for adding information including suggestions for disaster prevention goods and insurance services to the videos generated based on the analysis results. As a result, the user can receive specific suggestions for effective disaster prevention goods and insurance services while viewing the simulation results, enabling them to easily and quickly take earthquake countermeasures.

[1237] A "user terminal" is an information and communication device operated by a user, including smartphones and tablets.

[1238] "360-degree video data" refers to video data that records and displays the entirety of a specific location from every direction.

[1239] "Building structural information" refers to information about a building, including the building's materials, age, number of floors, earthquake resistance, etc.

[1240] A "generative model" refers to an artificial intelligence model that analyzes 360-degree video data and building structural information to generate the data that serves as the basis for simulations.

[1241] "Analysis" refers to the process of extracting and understanding specific information from received data.

[1242] "Earthquake simulation" refers to the process of predicting and virtually recreating the shaking and movement of objects that occur during an earthquake.

[1243] "Visualization video" refers to a video generated to display the results of analysis or simulation in a visually easy-to-understand format.

[1244] "Means for generating advertisements" refers to a mechanism that automatically creates advertisements for users, such as countermeasures products and insurance services, based on the analysis results.

[1245] "Emergency goods" refer to items used to reduce damage during earthquakes.

[1246] "Insurance services" refers to insurance products to cover damage caused by earthquakes.

[1247] "Insertion means" refers to the technical means used to add advertising or suggestion information to the generated video.

[1248] This invention is a system that allows users to collect 360-degree video data of their homes and rooms, as well as structural information about the building, using a dedicated app, and then sends that data to a server to perform earthquake simulations, visualize the risks, and insert advertisements into the video, including suggestions for disaster prevention products and insurance services.

[1249] User device operations

[1250] Users launch a dedicated app on their smartphone, tablet, or other device to first capture 360-degree indoor video data. At this time, the camera rotates to record video in all directions. Next, the user enters the building's structural information through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This improves the accuracy of the simulation.

[1251] Sending data

[1252] The user device sends the acquired 360-degree video data and building structure information to a server via the Internet, where the data is transferred securely.

[1253] Processing on the server

[1254] The server receives the data sent by the user and analyzes it using a generative model. Specifically, it analyzes the 360-degree video data to determine the room layout and furniture placement, and simulates the shaking patterns during an earthquake based on building structural information. Based on the simulation results, a video is generated using CG technology. The generated video shows in detail what state the user's indoors will be in when an earthquake occurs, and how furniture and objects will move.

[1255] Ad generation and insertion

[1256] Furthermore, the server generates advertisements for users based on the analysis results, including suggestions for countermeasures and insurance services. These advertisements are inserted into the simulation video. The generated final video is then sent back to the user's device.

[1257] Watching videos and suggesting solutions

[1258] The received video is displayed on the app on the user's device. By watching the simulation video, users can gain a concrete understanding of the risks involved in an earthquake and recognize what countermeasures are necessary. The system also suggests countermeasure products and insurance services, making it easier for users to take immediate action.

[1259] Hardware and software used

[1260] The hardware uses a smartphone and a 360-degree camera, and the software uses the requests library (for sending HTTP requests), the moviepy library (for video editing), and an AI model for earthquake simulation on the server side.

[1261] Examples of specific examples and prompts

[1262] Specific examples

[1263] Users launch a dedicated smartphone app and take a 360-degree photo of their living room. They then input the building's structural information, such as "wooden construction," "20 years old," "two stories," and "average earthquake resistance." This information is sent to a server, and a video containing advertisements for "earthquake-resistant mats," "earthquake insurance," and "furniture tip-prevention products" is displayed based on the simulation results.

[1264] Prompt Sentence Examples

[1265] Based on 360-degree video data taken by users and building structural information, simulate shaking during an earthquake and the movement of furniture and objects, and reflect the results in your advertising video. Specifically, simulate with high accuracy how furniture will fall and objects will fly off, and recommend products and services to users to mitigate the risk.

[1266] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1267] Step 1:

[1268] The user launches a dedicated app on a user device such as a smartphone or tablet. The user acquires 360-degree video data of the indoor space, rotating the camera to record video in all directions. The input is the video data acquired from the 360-degree camera, and the output is 360-degree video data stored on the device.

[1269] Step 2:

[1270] Users input building structural information through the app, including materials (e.g., wood, reinforced concrete), age, number of floors, earthquake resistance, etc. The input is the building structural information entered by the user, and the output is the building structural information stored on the device.

[1271] Step 3:

[1272] The user device transmits the acquired 360-degree video data and building structure information to a server via the Internet. The input is the 360-degree video data and building structure information stored on the device, and the output is the data transmitted to the server.

[1273] Step 4:

[1274] The server receives the data sent by the user and analyzes it using a generative AI model. Specifically, it analyzes the 360-degree video data to identify the room layout and furniture placement, and simulates the shaking patterns during an earthquake based on building structural information. The input is the sent 360-degree video data and building structural information, and the output is the analysis and simulation results.

[1275] Step 5:

[1276] The server uses CG technology to generate an earthquake simulation video based on the simulation results. The generated video shows in detail what state the user's interior will be in when an earthquake occurs, and how furniture and objects will move. The input is the simulation results, and the output is the generated simulation video.

[1277] Step 6:

[1278] Furthermore, based on the analysis results, the server generates advertisements for users, including suggestions for countermeasures and insurance services, and inserts these advertisements into the simulation video. The inputs are the simulation video and advertisement information, and the output is the final video with the advertisements inserted.

[1279] Step 7:

[1280] The server then sends the generated final video back to the user terminal. The input is the final video, and the output is the video sent to the user terminal.

[1281] Step 8:

[1282] The final video received is displayed on the app on the user's device and the user is allowed to watch it. By watching the simulation video, the user can concretely understand the risks in the event of an earthquake and recognize what countermeasures are necessary. The input is the final video received, and the output is the user's viewing and consideration of countermeasures.

[1283] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1284] This system uses an emotion engine installed in a user device to recognize the user's emotions and propose earthquake countermeasures based on those emotions. It operates by combining the user device, server, emotion engine, and software that links them together.

[1285] 1. User device operation

[1286] Users launch a dedicated app using their device, such as a smartphone or tablet. In the app, users first rotate the camera to capture the entire room to obtain 360-degree video data of the interior. Once the video is complete, the video data is temporarily saved. Next, users enter structural information about the building through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This improves the accuracy of the simulation.

[1287] 2. Operation of the Emotion Engine

[1288] While the user is recording video and inputting building structure information, the emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, and input. This data is analyzed to identify the user's current emotional state (e.g., relief, anxiety, fear). The emotion data is temporarily stored and sent to the server along with the analysis results.

[1289] 3. Data transmission

[1290] The user device transmits the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted as appropriate.

[1291] 4. Processing on the Server

[1292] The server receives and temporarily stores the data sent by the user, then uses an AI model to analyze the 360-degree video data and building structure information to determine room layout and furniture placement, and evaluate the earthquake resistance of the building based on its structure.

[1293] 5. Earthquake Simulation

[1294] After the server analyzes the data, it runs an earthquake simulation based on the analysis results. The simulation recreates how furniture and appliances move and fall, and how books and tableware will fly apart. This process visualizes the risks in the event of an earthquake.

[1295] 6. Video Creation and Distribution

[1296] Based on the simulation results, the server uses CG technology to generate a video. This video shows in detail what the interior of a building looks like when an earthquake occurs, how furniture moves, and how scattered objects are. Based on the analysis results of the emotion engine, messages and countermeasures information corresponding to the user's emotions are displayed in the video and within the app.

[1297] 7. Display on user device

[1298] The generated video is then sent back to the user's device. During transmission, the data is compressed and optimized, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[1299] 8. Emotion-based countermeasures

[1300] Users will receive a notification from the app that a simulation video has been prepared. As they watch the video, the system will customize and present specific earthquake preparedness measures to the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will suggest a more reassuring message and easy-to-implement measures. It will also suggest emergency supplies, making it easier for users to take measures immediately.

[1301] Specific examples

[1302] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine determines that the user is feeling anxious, the system will display specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[1303] In this way, the present invention helps users concretely understand the risks during earthquakes and implement appropriate countermeasures, thereby improving safety and reducing damage. A customized approach that takes into account the user's emotions can make the implementation of countermeasure actions even more effective.

[1304] The processing flow will be explained below.

[1305] Step 1:

[1306] The user launches a dedicated app on their smartphone or tablet. Following the app's instructions, the user rotates the camera to capture the entire room, capturing 360-degree video data of the indoor space. Once the video is complete, it is temporarily saved on the user's device.

[1307] Step 2:

[1308] Users input building structural information into the app, such as the building's materials (wood, reinforced concrete, etc.), age, number of floors, and earthquake resistance. The input building structural information is temporarily saved on the user's device along with the video data.

[1309] Step 3:

[1310] The emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, etc. This process is carried out while the user is entering information, and determines whether the user is feeling safe, anxious, or scared. The emotion data is analyzed and temporarily stored.

[1311] Step 4:

[1312] The user device sends the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted during transmission.

[1313] Step 5:

[1314] The server receives the 360-degree video data, building structure information, and emotion data sent by the user. The server temporarily stores this data and then begins the analysis process. First, it analyzes the 360-degree video data to identify the room layout and furniture placement.

[1315] Step 6:

[1316] The server evaluates the earthquake resistance of a building based on structural information, taking into account information such as the building's materials and age, and predicts the shaking pattern and extent of impact during an earthquake.

[1317] Step 7:

[1318] The server then runs an earthquake simulation based on the analysis results. The simulation reproduces how furniture and home appliances will move and fall, and how books, tableware, etc. will fly apart in response to a specified seismic intensity (for example, intensity 6). The simulation results are then generated.

[1319] Step 8:

[1320] The server uses CG technology to generate a video based on the simulation results, which shows in detail what the interior of a building looks like when an earthquake strikes, how furniture moves, and how objects are scattered.

[1321] Step 9:

[1322] Based on the analysis results of the emotion engine, the server inserts emotional messages and countermeasures information into videos and apps. For example, if a user is feeling anxious, the system will display a message designed to reassure them.

[1323] Step 10:

[1324] The server then sends the generated video to the user's device, where it is converted into a playable format and appropriately compressed.

[1325] Step 11:

[1326] The user's device receives the video sent from the server and notifies the user within the app. Upon receiving the notification, the user can open the app and watch the generated simulation video.

[1327] Step 12:

[1328] After watching the video, the user's device will present customized earthquake countermeasures based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will detail easy-to-implement countermeasures and display a message to provide reassurance. It will also suggest necessary countermeasure goods.

[1329] In this way, this system supports users in taking concrete and effective earthquake countermeasures while taking their emotions into consideration. The generated videos and customized countermeasures help users to concretely understand the risks of earthquakes and take appropriate measures.

[1330] Example 2

[1331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1332] Conventional earthquake simulation systems propose earthquake countermeasures without considering the user's emotions or psychological state, which makes it difficult to alleviate users' anxiety and reduces the rate at which countermeasures are implemented. Furthermore, the system does not provide sufficient suggestions for specific countermeasures or the goods needed to implement them, so the proposals are not in a form that is easy for users to implement immediately.

[1333] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1334] A means for acquiring user emotion data by a user terminal;

[1335] A means for transmitting 360-degree video data, building structure information, and emotion data from a user terminal to a server;

[1336] A means for analyzing the received data by an AI model in a server;

[1337] This makes it possible to provide specific and feasible earthquake countermeasures that take into account the user's emotional state.

[1338] A "user terminal" is a portable information processing device operated by a user, such as a smartphone or tablet.

[1339] "360-degree video data" is data that digitally records visual information in all directions captured by a rotating camera.

[1340] "Building structural information" refers to data on the physical characteristics of a building, such as the building's materials, age, number of floors, and earthquake resistance.

[1341] "Emotional data" is data that indicates the user's psychological state, analyzed from the user's facial expressions, tone of voice, etc.

[1342] A "server" is a computer system that receives data from user terminals via a network and analyzes and processes it.

[1343] An "AI model" is a program that uses artificial intelligence algorithms to analyze data and make predictions and simulations.

[1344] "Earthquake simulation" is a computational process that recreates in a virtual environment the conditions inside a building during an earthquake.

[1345] "Movie" refers to dynamic video data generated to visually visualize the simulation results.

[1346] "Countermeasures proposals" are proposals that specifically outline the actions users should take and the preparations they should make in the event of an earthquake.

[1347] "Earthquake prevention goods" refer to equipment and devices necessary for earthquake prevention, which help users implement earthquake prevention measures.

[1348] This invention is a system that uses an emotion engine installed in a user terminal to recognize a user's emotions and proposes earthquake countermeasures based on those emotions. This system operates by combining a user terminal, a server, an emotion engine, and software that links these together.

[1349] First, the user launches a dedicated app on their device, such as a smartphone or tablet. To obtain 360-degree indoor video data within the app, the user rotates the camera to capture the entire room. Once the video is complete, it is temporarily saved. Next, the user enters the building's structural information. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and earthquake resistance.

[1350] Next, while the user records video and inputs building structure information, the emotion engine on the user's device recognizes emotions from the user's facial expressions, tone of voice, and input content. This data is analyzed to identify the user's current emotional state (e.g., relief, anxiety, fear). The emotion data is temporarily stored and sent to the server along with the analysis results.

[1351] The user device transmits the acquired 360-degree video data, building structure information, and emotion data to the server. The data is securely transferred over the Internet. The data is compressed and encrypted as appropriate.

[1352] The server receives the data sent by the user and temporarily stores it. The server then uses an AI model to analyze the 360-degree video data and building structural information to determine room layout and furniture placement, and evaluate the building's earthquake resistance based on its structure. Specific technologies used include image processing algorithms and machine learning models. This provides the basic data needed to accurately simulate risks in the event of an earthquake.

[1353] Once the analysis is complete, the server runs an earthquake simulation based on the results. The simulation recreates how furniture and appliances will move and fall, and how books, tableware, etc. will fly apart. This provides a visually easy-to-understand display of specific earthquake risks.

[1354] Based on the simulation results, the server uses CG technology to generate a video. This video depicts in detail the state of the room at the time of the earthquake, the movement of furniture, and the scattering of objects. In addition, based on the analysis results of the emotion engine, messages and countermeasure information corresponding to the user's emotions are displayed in the video and within the app. For example, if the user is feeling anxious, a reassuring message and easy-to-implement countermeasure information are displayed.

[1355] The generated video is sent from the server to the user's device. The data is compressed and optimized before being sent, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[1356] When users receive a notification from the app that a simulation video has been prepared and watch the video, the system will customize and present specific earthquake preparedness measures to the user based on the analysis results of the emotion engine. If the user is feeling anxious, the system will suggest messages that will provide more reassurance and easy-to-implement measures. It will also suggest emergency supplies, making it easier for users to take measures immediately.

[1357] As a specific example, a user takes a 360-degree photo of their living room and enters the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine recognizes that the user is feeling anxious, the system displays specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[1358] An example of a prompt for the generative AI model is, "If it recognizes that the user is feeling anxious, please generate a message suggesting specific measures to reduce risks in the event of an earthquake." Using this prompt, the AI ​​model can generate appropriate countermeasure information according to the user's emotional state.

[1359] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1360] Step 1:

[1361] The user launches a dedicated app using a user device such as a smartphone or tablet. The device rotates the camera to capture the entire room, allowing the user to obtain 360-degree video data of the room within the app. The input in this step is the user's operation and video capture by the camera, and the output is 360-degree video data. The user then operates the device appropriately to rotate the camera correctly and obtain a 360-degree video image of the room.

[1362] Step 2:

[1363] The user inputs structural information about the building through a dedicated app. The device temporarily stores the information entered by the user, such as the building's materials (wood, reinforced concrete, etc.), age, number of floors, and seismic performance, in a database. The input in this step is detailed building information manually entered by the user, and the output is the temporarily stored building structural information.

[1364] Step 3:

[1365] The device uses a front camera and microphone to capture the user's facial expressions and tone of voice. The emotion engine analyzes this data in real time to generate the user's emotion data. The input in this step is the user's facial and voice data, and the output is the analyzed emotion data.

[1366] Step 4:

[1367] The device assembles the acquired 360-degree video data, building structure information, and emotion data into packets. These packets are then compressed and encrypted. The device then sends these packets to a server via the Internet. The input in this step is the individual data (360-degree video data, building structure information, emotion data), and the output is the encrypted data packet sent to the server.

[1368] Step 5:

[1369] The server receives the data packet sent from the terminal and checks the integrity of the content. If it is normal, it temporarily stores the data. The input in this step is the encrypted data packet, and the output is the temporarily stored data set.

[1370] Step 6:

[1371] The server uses an AI model to analyze the 360-degree video data and building structural information. Specifically, it uses image processing algorithms to identify room layouts and furniture placements and evaluate the building's earthquake resistance. The input for this step is the temporarily stored dataset (360-degree video data, building structural information), and the output is the analysis results.

[1372] Step 7:

[1373] The server runs an earthquake simulation based on the analysis results. The simulation calculates how the effects of a virtual earthquake will be transmitted to furniture and structures, and generates specific simulation data. The input in this step is the analysis results, and the output is simulation data.

[1374] Step 8:

[1375] The server uses computer graphics technology to generate a video that visualizes the indoor situation during an earthquake using the simulation data. Furthermore, based on the analysis results of the emotion engine, messages and countermeasure information for users are added to the video. The inputs for this step are the simulation data and emotion data, and the output is the generated video.

[1376] Step 9:

[1377] The server compresses the generated video and sends it to the user's device. By converting it into the appropriate format, the video can be played back without any problems on the user's device. The input of this step is the generated video, and the output is a compressed video file.

[1378] Step 10:

[1379] The device receives the video sent from the server and notifies the user. The input in this step is a compressed video file, and the output is a video file stored on the device and a notification to the user. The user opens the app and watches the generated simulation video.

[1380] Step 11:

[1381] The device provides a function in which the system suggests specific earthquake countermeasures based on the user's emotional data while watching the simulation video. If the user feels anxious, specific countermeasures are displayed. The input in this step is the emotional data and the generated video, and the output is a video display containing the countermeasures.

[1382] Through these steps, this system will encourage users to take earthquake countermeasures, thereby improving safety and reducing damage.

[1383] (Application example 2)

[1384] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1385] Conventional earthquake countermeasure proposal systems present uniform countermeasure proposals without considering the user's emotions, making it difficult to provide specific countermeasures that respond to the psychological state and needs of each individual user. Furthermore, when visualizing earthquake simulation results, the provision of countermeasure proposals to deepen users' understanding or information that gives them a sense of security is insufficient. This reduces users' motivation to actually implement countermeasures, making it difficult to improve safety.

[1386] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1387] In this invention, the server includes means for acquiring emotion data using an emotion recognition engine in the user terminal, means for generating information including countermeasures corresponding to the user's emotions based on a video generated by the server, and means for transmitting the generated video and countermeasures from the server to the user terminal. This makes it possible to provide individualized countermeasures corresponding to the user's emotions, allowing the user to take specific and feasible earthquake countermeasures with greater peace of mind.

[1388] A "user terminal" is an electronic device operated by a user, including portable devices such as smartphones and tablet PCs.

[1389] "Indoor 360-degree video data" refers to video data of an indoor environment captured by a user in all directions, visually recording the entire room.

[1390] "Building structural information" refers to information about the structure of a building, such as its materials, age, number of floors, and earthquake resistance.

[1391] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and tone of voice to identify the user's emotional state.

[1392] "Server" refers to a computer system for analyzing, processing, storing, transmitting and receiving data.

[1393] An "AI model" is a model that uses machine learning and deep learning techniques to analyze input data and has algorithms to perform specified tasks.

[1394] "Earthquake simulation" is the process of recreating on a computer what the interior of a building would look like when an earthquake occurs, based on building structural information and 360-degree video data.

[1395] "Videos that visualize simulation results" refers to animations and videos generated to visually show the results of earthquake simulations.

[1396] "Countermeasures" refers to information that includes specific actions, recommended items, and initiatives proposed to reduce earthquake risk.

[1397] "Means of transmission" refers to the software and hardware technology used to send data acquired or generated on the user's device to the server via the network.

[1398] "Means for receiving" refers to software and hardware technology for receiving, displaying, and storing data from the server to the user terminal.

[1399] When referring to "generated video and proposed countermeasures," "generated video" refers to the visual image generated based on the simulation results, and "proposed countermeasures" refers to information on earthquake countermeasures proposed based on the generated video.

[1400] "Means for displaying on the user's terminal" refers to the technology for displaying the received data on the terminal screen in a form that the user can view.

[1401] The present invention relates to an earthquake countermeasure proposal system that takes into account the user's emotions. Specifically, the system generates earthquake simulation results based on emotion recognition by the user's terminal and operates to present individualized countermeasure proposals.

[1402] User device operations

[1403] Users launch a dedicated application using their device, such as a smartphone or tablet PC. Within the app, users first rotate the camera to capture the entire room to obtain 360-degree indoor video data. Once the video is complete, it is temporarily saved. Next, users enter the building's structural information through the app. This information includes the building's materials (e.g., wood, reinforced concrete), age, number of floors, and seismic performance. This procedure improves the accuracy of the earthquake simulation.

[1404] Emotion Engine Operation

[1405] While the user is recording video and inputting building structure information, an emotion recognition engine installed on the user's device analyzes the user's facial expressions and tone of voice to collect emotional data. This emotional data is temporarily stored and sent to the server along with the analysis results. This allows the system to accurately identify the user's current emotional state (e.g., relief, anxiety, fear).

[1406] Sending data

[1407] The user device sends the acquired 360-degree video data, building structure information, and emotion data to a server. The data is securely transferred over the Internet. Data compression and encryption technologies are used for this transmission to ensure communication security.

[1408] Processing on the server

[1409] The server receives and temporarily stores the data sent by the user. The server then uses an AI model (e.g., TensorFlow or PyTorch) to analyze the 360-degree video data and building structure information. This analysis identifies the room layout and furniture placement, and evaluates the earthquake resistance of the building based on its structure.

[1410] Earthquake Simulation

[1411] Once the analysis is complete, the server uses the data to run an earthquake simulation, which recreates how furniture and appliances move and fall, and how books, tableware, and other items will fly apart. This process provides a concrete visualization of the risks that may occur during an earthquake.

[1412] Video Creation and Delivery

[1413] Based on the simulation results, the server uses CG technology to generate a simulation video. This video shows in detail what the interior of the building will look like when an earthquake occurs, how furniture will move, and how objects will be scattered. Furthermore, based on the analysis results of the emotion engine, messages and countermeasure information corresponding to the user's emotions are added to the video.

[1414] Display on user device

[1415] The generated video is then sent back to the user's device. During transmission, the data is compressed and optimized, and converted into a format that can be played on the user's device. The user's device receives the video sent from the server and notifies the user within the app. The user can then open the app and watch the generated simulation video.

[1416] Emotion-based countermeasure proposals

[1417] When watching a video, the system will present specific earthquake countermeasures to the user based on the analysis results of the emotion engine. For example, if the user is feeling anxious, the system will display specific countermeasures such as "Here's how you can easily secure unnecessary furniture" along with the video. It will also suggest countermeasure products, making it easier for the user to take immediate action.

[1418] Specific examples

[1419] For example, a user can take a 360-degree photo of their living room and input the building's structural information, such as "wooden construction" and "20 years old." The user's device sends this data to a server, which analyzes the data and runs a simulation of an earthquake with a seismic intensity of 6. Based on the simulation results, a video is generated that recreates furniture falling over and books scattering, and is sent from the server to the user. If the emotion engine determines that the user is feeling anxious, the system will display specific countermeasures, such as "Here's how you can easily secure unnecessary furniture," along with the video.

[1420] Prompt Sentence Examples

[1421] Generate a video of the earthquake simulation results. If the emotion engine recognizes the user's anxiety, generate a CG that provides a specific message of reassurance and presents simple countermeasures.

[1422] In this way, the present invention aims to improve safety and reduce damage by providing personalized earthquake countermeasures based on the user's emotions.

[1423] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1424] Step 1:

[1425] Acquiring data from user devices

[1426] Users launch a dedicated application on their device, such as a smartphone or tablet PC, and capture 360-degree indoor video data. They rotate the device's camera to capture the entire room, and then enter building structural information (materials, age, number of floors, earthquake resistance, etc.) into the app. This data is temporarily saved.

[1427] Input: 360-degree video data, building structure information

[1428] Output: Temporarily saved 360-degree video data and building structure information

[1429] Step 2:

[1430] Acquiring emotion data

[1431] While the user is entering data, an emotion recognition engine installed on the user's device analyzes the user's facial expressions and tone of voice, and emotional data is collected and temporarily stored.

[1432] Input: User's facial expression data, tone of voice

[1433] Output: Temporarily saved emotion data

[1434] Step 3:

[1435] Sending data to the server

[1436] The user device transmits the captured 360-degree video data, building structure information, and emotion data to a server, where the data is securely transferred over the Internet and compressed and encrypted.

[1437] Input: 360-degree video data, building structure information, emotion data

[1438] Output: Data sent to the server

[1439] Step 4:

[1440] Data reception and storage on the server

[1441] The server receives the data sent by the user and temporarily stores it.

[1442] Input: 360-degree video data, building structure information, emotion data

[1443] Output: Temporarily saved data

[1444] Step 5:

[1445] Data analysis

[1446] The server uses AI models to analyze the received 360-degree video data and building structure information, which then evaluates the room layout, furniture placement, and earthquake resistance of the building based on its structure.

[1447] Input: 360-degree video data, building structure information

[1448] Output: Room layout, furniture placement, and earthquake resistance evaluation results

[1449] Step 6:

[1450] Running an earthquake simulation

[1451] The server then uses the analysis results to run an earthquake simulation, which recreates how furniture and appliances move and fall, and how books, tableware, and other items will fly apart.

[1452] Input: Room layout, furniture arrangement, earthquake resistance evaluation results

[1453] Output: Earthquake simulation results

[1454] Step 7:

[1455] Visualization of simulation results

[1456] Based on the simulation results, the server uses CG technology to generate a simulation video. The video shows in detail what the interior of the building will look like when an earthquake occurs, how furniture will move, and how objects will be scattered. Based on the analysis results of the emotion engine, countermeasures corresponding to the user's emotions are added to the video.

[1457] Input: Earthquake simulation results, emotion data

[1458] Output: Simulation video, video with countermeasures based on emotions

[1459] Step 8:

[1460] Video and countermeasures are sent from the server to the user's device

[1461] The server then sends the generated simulation video and countermeasures to the user's device, where data is compressed and optimized.

[1462] Input: Simulation video, video with emotion-based countermeasures

[1463] Output: Video and countermeasures sent to the user's device

[1464] Step 9:

[1465] Video playback on user devices and suggestions for solutions

[1466] The user device receives the video and proposed solutions sent from the server and notifies the user within the app. The user opens the app, watches the generated simulation video, and confirms the proposed solutions.

[1467] Input: Video and countermeasures sent from the server

[1468] Output: Videos watched by users and identified solutions

[1469] Through the above steps, the present invention can visualize specific risks during an earthquake for the user and provide customized countermeasures based on the user's emotions.

[1470] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1471] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1472] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1473] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1474] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1475] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1476] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1477] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1478] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1479] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1480] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1481] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1482] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1483] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1484] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1485] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1486] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1487] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1488] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1489] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, func...

Claims

1. A means for acquiring indoor 360-degree video data using a user terminal; A means for inputting structural information of a building via a user terminal; A means for transmitting 360-degree video data and building structure information from a user terminal to a server; A means for analyzing the received data by an AI model in a server; A means for executing an earthquake simulation based on the analysis results in the server; a means for generating a video that visualizes the simulation results in the server; A means for transmitting the generated video from the server to the user terminal; means for displaying the received video at the user terminal; A system including:

2. 2. The system according to claim 1, further comprising means for presenting specific earthquake countermeasures to the user in the user terminal.

3. The system according to claim 1, further comprising means for adding information including suggestions for countermeasures goods to the simulation video generated based on the 360-degree video data and building structure information in the server.

Images