System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting data of a physical space; means for analyzing the collected data to generate a virtual reality space; and means for providing an interface for a user to manipulate the virtual reality space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional on-site investigations and troubleshooting have been subject to significant constraints, such as physical travel and time, making it difficult to respond quickly. Furthermore, collecting detailed environmental data and accurately analyzing it requires highly specialized knowledge and a great deal of effort. This makes efficient data collection and analysis difficult, resulting in a high likelihood of response delays and errors. This invention solves these problems and provides a system that enables faster and more accurate data collection and analysis. [Means for solving the problem]

[0005] The present invention provides a system including a means for collecting data on a physical space, a means for analyzing the collected data and generating a virtual reality space, and a means for providing an interface for a user to operate the virtual reality space. Specifically, the means for collecting data on the physical space uses a laser sensor and a camera to collect the shape and object arrangement of the physical space, and the means for generating the virtual reality space complements missing data in cooperation with a generative artificial intelligence and a database. The system also provides an interface for a user to operate the virtual reality space, allowing the user to easily operate objects in the virtual reality space using gloves and audio glasses. This significantly improves the efficiency of on-site investigations and troubleshooting.

[0006] "Physical space" refers to a real space or area that actually exists.

[0007] "Data" means facts, concepts, instructions, or other representations of information used directly or indirectly to communicate or express knowledge.

[0008] "Collect" refers to the act of gathering information or data needed for a specific purpose.

[0009] "Means" refers to the methods, techniques, or devices used to achieve an end.

[0010] "Analysis" refers to the act of breaking down and analyzing given data or information to understand its content and meaning.

[0011] "Virtual reality space" refers to a three-dimensional virtual environment generated using computer technology, which can be experienced visually and tactilely by the user.

[0012] "User" refers to a person who operates or uses a system or device.

[0013] An "interface" refers to the point of contact or means by which two different systems or devices exchange information and communicate with each other.

[0014] A "laser sensor" refers to a device that uses laser light to measure the distance to an object and its shape.

[0015] "Camera" refers to an optical device for capturing images or video.

[0016] "Shape" refers to the appearance and contours of an object, including its position in three-dimensional space.

[0017] An "object" refers to a physical or virtual object or item that exists in space.

[0018] "Generative artificial intelligence (AI)" refers to artificial intelligence that has the ability to create new information and value from data.

[0019] A "database" refers to a collection of information that is structured to efficiently store large amounts of data and enable quick retrieval when needed.

[0020] "To complete" refers to the act of adding missing parts to make something complete.

[0021] A "glove" refers to a device worn on the hand that senses the user's movements and transmits those movements to the system.

[0022] "Audio glasses" are devices that allow a user to receive audio information, and are usually attached to eyeglasses. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] The present invention is a system that collects data from a physical space, analyzes the data, generates a virtual reality space (hereinafter referred to as a VR space), and provides an interface for a user to operate the VR space. An embodiment of the present invention will be described in detail below.

[0045] System Configuration

[0046] 1. Data collection terminal

[0047] The device is equipped with a laser sensor and a camera, which allows it to collect detailed geometric and image data of the physical space, as well as environmental data using temperature sensors and other sensors.

[0048] 2. Server

[0049] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space.

[0050] 3. VR System

[0051] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[0052] Program processing

[0053] 1. Physical space data collection:

[0054] The device uses a laser sensor to scan the shape of the physical space and obtain the location of objects, a camera to capture color images and perform object recognition, and a temperature sensor to collect environmental data, which is then compressed, packaged, and sent to a server in real time.

[0055] 2. Data processing and VR space generation:

[0056] The server receives the collected data and begins analysis. It generates a 3D point cloud from the laser scan data and fills in any missing data using generative AI and a database. This creates a complete 3D model. The generated 3D model is then converted into a data format optimized for VR systems.

[0057] 3. VR space operation:

[0058] The user wears VR goggles and uses gloves to navigate within the VR space. The VR system receives optimization data sent from the server and displays it to the user. The user can receive real-time audio guidance and explanations through the audio glasses, and can display and operate information using voice commands.

[0059] Specific examples

[0060] 1. Data Collection Example:

[0061] The device scans an office room. It uses a laser sensor to capture the shape of the walls, floor, and ceiling, and a camera to capture the position and shape of each piece of furniture. It uses a temperature sensor to measure the temperature at each point in the room, and sends all the data to a server.

[0062] 2. Data processing examples:

[0063] The server analyzes the received data and generates a 3D model of the office, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[0064] 3. Examples of on-site investigations and troubleshooting:

[0065] Users put on VR goggles to access a virtual office and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature sensor data. They receive audio guidance through the audio glasses, identify the cause, and share steps to take with other teams.

[0066] In this way, users can remotely capture and manipulate detailed data about physical spaces, enabling highly efficient and accurate on-site inspections and troubleshooting.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] The device initializes the sensors: it activates the laser sensor, camera, and temperature sensor and checks that they are working properly.

[0070] Step 2:

[0071] The device begins scanning the physical space, using laser sensors to capture 3D positional data of the entire room and cameras to capture color images, while temperature sensors gather environmental data.

[0072] Step 3:

[0073] The terminal compresses and packages the collected data in real time, where it is formatted and ready to be sent to the server.

[0074] Step 4:

[0075] The device sends the compressed data packets over the network to the server, which monitors the communication to ensure the data transfer is successful.

[0076] Step 5:

[0077] The server receives data sent from the device, such as 3D position data, color images, and temperature data, and stores them in a waiting state for processing.

[0078] Step 6:

[0079] The server analyzes the received data and generates a 3D point cloud, building a complete 3D model of the physical space based on the laser scan data.

[0080] Step 7:

[0081] The server complements missing parts of the generated 3D model by working with generative artificial intelligence (AI) and a database. Missing objects and texture information are generated by AI and retrieved from the database.

[0082] Step 8:

[0083] The server optimizes the interpolated 3D model for the VR system, reducing the data size and converting it into a format compatible with VR goggles.

[0084] Step 9:

[0085] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins.

[0086] Step 10:

[0087] The user manipulates objects in the VR space using the glove, and receives real-time audio guidance and explanations through the audio glasses to operate the system.

[0088] Step 11:

[0089] Users can conduct real-time on-site investigations and troubleshoot problems within the VR space. If an abnormality is discovered, they can check the details and plan the necessary countermeasures.

[0090] Step 12:

[0091] The server reflects user operation information in real time and updates the data for the entire system. It also saves necessary data based on user instructions and generates reports to manage work history.

[0092] As described above, the system of the present invention includes a series of processes from collecting data on the physical space to generating and operating the VR space, enabling efficient and accurate on-site investigations and troubleshooting.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] There is a demand for technology that can quickly and accurately collect data from physical spaces and efficiently reproduce and manipulate it in virtual reality spaces. In particular, advanced processing power and collaboration are required to complement missing data and optimize 3D models, but current systems are unable to meet these requirements. Therefore, the challenge is to generate highly accurate 3D models and provide an environment in which users can intuitively operate them.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes a means for receiving collected data, a means for analyzing the received data and generating a 3D point cloud, a means for generating a complete 3D model by utilizing generative artificial intelligence and a database to supplement missing data, and a means for providing the optimized 3D model to the user's virtual reality space, thereby enabling detailed data of a physical space to be collected quickly and accurately and efficiently reproduced and manipulated in a virtual space.

[0098] "Physical space" refers to the space in which a user acts in the real world, an environment with a concrete shape and layout.

[0099] "Data collection means" refers to a combination of hardware and software for collecting shape data, image data, environmental data, etc. of a physical space.

[0100] A "server" is a high-performance computer system that receives, analyzes, and processes data, and has the ability to link with generative artificial intelligence and databases.

[0101] A "laser sensor" is a sensor that uses laser light to measure the distance and shape of an object.

[0102] A "camera" is an optical device for capturing image data of physical space.

[0103] A "temperature sensor" is a sensor for measuring the ambient temperature of a physical space.

[0104] A "3D point cloud" is a three-dimensional data set composed of point cloud data acquired from laser sensors and other measurement devices.

[0105] "Generative AI" is AI that has the ability to use existing data to fill in missing data or generate new data.

[0106] A "database" is a system that can efficiently store and search large amounts of data.

[0107] "User" refers to a person or organization that uses the system to manipulate data in physical space.

[0108] "Virtual reality space" refers to a virtual three-dimensional space generated by computer technology that provides an environment in which users can engage in activities within that space.

[0109] "VR goggles" are devices that allow users to visually experience virtual reality spaces.

[0110] A "glove" is an input device that a user uses to manipulate objects in a virtual reality space.

[0111] An "audio device" is an audio-related device that allows a user to use voice guidance and commands.

[0112] This invention is a system that collects detailed data of a physical space, analyzes it to generate a virtual reality space (hereinafter referred to as VR space), and allows users to operate the VR space in real time.

[0113] Data collection terminal

[0114] The device is equipped with a laser sensor, camera, and temperature sensor to collect geometric data, image data, and environmental data of the physical space. Specifically, the laser sensor collects detailed geometric data of the space, and the camera takes high-resolution image data. The temperature sensor measures the environmental temperature at multiple points, compresses and packages all the data, and sends it to the server in real time.

[0115] Server data analysis

[0116] The server is equipped with a high-performance processor and storage to analyze the received data. The server first generates a 3D point cloud from the received laser scan data and integrates it with image and temperature data acquired from the camera. Missing data is then supplemented using generative artificial intelligence (generative AI models) and a database to generate a complete 3D model. The generated 3D model is then converted into a data format optimized for the VR system.

[0117] VR system operation

[0118] Users manipulate the VR space using VR goggles, gloves, and an audio device. The VR system receives optimization data sent from the server and provides the user with a highly accurate virtual space. Users can manipulate objects in the virtual space using the gloves and receive real-time audio guidance and explanations through the audio device. They can also use voice commands to obtain specific information or perform operations.

[0119] Specific examples

[0120] 1. Data Collection Example:

[0121] The device uses a laser sensor to scan an office room, collecting geometric data on the walls, floors, ceilings, and furniture. It also uses a camera to capture a panoramic view of the office and a temperature sensor to create a temperature map of the room. This data is then sent to a server in real time.

[0122] 2. Data processing examples:

[0123] The server analyzes the received data and uses generative AI models to fill in any incomplete data, resulting in a complete, optimized 3D model of the office.

[0124] 3. Examples of on-site investigations and troubleshooting:

[0125] Users access the virtual office by wearing VR goggles, manipulate objects with gloves, and receive real-time voice guidance about abnormal temperature rises through an audio device. They can use voice commands to obtain additional information and identify the cause of the temperature anomaly.

[0126] Prompt Sentence Examples

[0127] "Check for temperature abnormalities in the VR space and explain the corrective steps using audio."

[0128] "Please explain in detail the process from collecting data from the physical space to generating the VR space."

[0129] In this way, the present invention allows detailed data of a physical space to be quickly and accurately collected and efficiently reproduced and manipulated in a virtual space.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] The device collects data on the physical space. First, it uses a laser sensor to scan the shape of the physical space and collects layout data on walls, floors, ceilings, furniture, etc. Next, it uses a camera to capture high-resolution color images. It also uses a temperature sensor to measure the ambient temperature at multiple points. This data is then combined to create a detailed dataset of the physical space. This dataset is then compressed and sent to a server in real time.

[0133] Input: Physical space

[0134] Output: Compressed package of geometry, image data, and temperature data

[0135] Step 2:

[0136] The server receives data sent from the device. It checks the data for consistency and requests the device to resend it if it contains incomplete data. It then analyzes the received laser scan data, image data, and temperature data to generate a 3D point cloud. It then builds a detailed 3D model of the physical space based on the point cloud data.

[0137] Input: Compressed data sent from the terminal

[0138] Output: 3D point cloud data, initial 3D model

[0139] Step 3:

[0140] The server uses generative artificial intelligence (generative AI model) and a database to fill in missing or incomplete data. The generative AI model naturally fills in missing parts based on existing data. A complete 3D model is constructed based on the analyzed data.

[0141] Input: initial 3D model, generative AI model, database

[0142] Output: Full 3D model

[0143] Step 4:

[0144] The server processes the generated 3D model to optimize it for the VR system. It performs optimization processes such as simplifying the mesh, optimizing the texture, and adjusting the frame rate to ensure smooth display in the VR space. The optimized data is then sent to the user's VR system.

[0145] Input: A complete 3D model

[0146] Output: Optimized VR data

[0147] Step 5:

[0148] Users interact with the VR space using VR goggles, gloves, and an audio device. The VR system receives optimized data from the server and displays a highly accurate virtual space. Users use the gloves to manipulate objects in the virtual space and receive real-time audio guidance and explanations through the audio device. Voice commands can also be used to obtain specific information or perform operations.

[0149] Input: Optimized VR data

[0150] Output: High-precision virtual space displayed to the user, audio guidance, and operation results

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] In modern factories, monitoring and maintenance of complex equipment is important, but it is necessary to dispatch specialized engineers to the site, which is time-consuming and costly. Furthermore, when an abnormality occurs, a rapid response is required, but this response can be delayed if the engineer is located in a remote location. Furthermore, when an abnormality is detected, it can be difficult to obtain detailed environmental and physical information, making troubleshooting difficult. There is a need to solve these issues and achieve more efficient monitoring and maintenance of equipment within factories and a faster response to abnormalities.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes a means for collecting data on the physical space, a means for analyzing the collected data and generating a virtual reality space, a means for providing an interface for a user to operate the virtual reality space, and a means for detecting abnormalities and notifying them in real time. This makes it possible to monitor and operate detailed equipment status in a factory in real time even from a remote location, and to respond quickly even if an abnormality is detected.

[0156] "Physical space" refers to a space that exists in the real world and has a specific size and shape.

[0157] "Data collection means" refers to devices and technologies used to acquire geometric and environmental data from physical spaces.

[0158] "Analysis" refers to examining collected data in detail to understand and process its structure and characteristics.

[0159] A "virtual reality space" is a virtual three-dimensional space generated by a computer that users can experience through their senses of sight and touch.

[0160] "Means for providing an interface" refers to devices and software that allow a user to operate a virtual reality space.

[0161] "Means for detecting anomalies" refers to devices or technologies for detecting unusual behavior or values ​​from collected data.

[0162] "Means for real-time notification" refers to devices or technologies that immediately convey information to users when an abnormality is detected.

[0163] A "laser sensor" is a device that uses light to scan the shape of physical space and obtain accurate three-dimensional data.

[0164] A "camera" is a device that takes pictures or videos and stores them as digital data.

[0165] "Generative AI" is an AI technology that automatically generates missing information based on collected data.

[0166] A "database" is a system that organizes and stores large amounts of data and allows fast access when needed.

[0167] A specific embodiment of the present invention will be described below. This system collects and analyzes data from a physical space to generate a virtual reality (VR) space, allowing users to manipulate the VR space.

[0168] System Configuration

[0169] 1. Data collection terminal

[0170] The device is equipped with a laser sensor and a camera to collect detailed geometric and image data of the physical space, and also collects environmental data using a temperature sensor, allowing the shape, layout, and environmental data of the physical space to be obtained in real time.

[0171] 2. Server

[0172] The server has a high-performance processor and storage for receiving and analyzing data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space. It also has the ability to detect abnormalities and send notifications in real time.

[0173] 3. VR System

[0174] The VR system used by users consists of VR goggles, gloves, and an audio device. This allows users to perform visual and tactile manipulation within the VR space. Users can also receive real-time audio guidance and explanations through the audio device.

[0175] Program processing

[0176] Data collection and transmission

[0177] The device uses a laser sensor to scan the physical space and generate a 3D point cloud, while a camera captures color images for object identification. A temperature sensor also collects environmental data, which is then compressed, packaged, and sent to a server in real time.

[0178] Data processing and VR space generation

[0179] The server analyzes the received data, generates a 3D point cloud from the laser scan data, and fills in any missing data using generative AI and a database. This generates a complete 3D model, which is then converted into a data format optimized for VR systems.

[0180] Interface with VR space

[0181] The user wears VR goggles and uses gloves to navigate within the VR space. The VR system receives optimized data sent from the server and displays it to the user. The user receives real-time audio guidance and explanations through the audio device, and uses voice commands to display and operate information.

[0182] Specific examples

[0183] 1. Data Collection Example:

[0184] The terminal scans a section of the factory, captures the shape of the equipment with a laser sensor, photographs the layout of each piece of equipment with a camera, measures the temperature of each piece of equipment with a temperature sensor, and sends all the data to a server.

[0185] 2. Data processing examples:

[0186] The server analyzes the received data and generates a 3D model of the factory, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[0187] 3. Examples of anomaly detection and response:

[0188] Users put on VR goggles to access a virtual factory and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature data. They receive audio guidance through an audio device, identify the cause, and share the steps to take to address the issue with other team members.

[0189] Examples of prompt statements

[0190] "Generate a 3D point cloud from the factory's lidar and camera data and fill in any gaps. Model all the equipment in the factory in detail and convert it into a data format optimized for VR systems and output it."

[0191] In this way, users can remotely acquire and manipulate detailed data about the physical space, enabling highly efficient and accurate monitoring of factory equipment and response to anomalies.

[0192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0193] Step 1:

[0194] Data collection

[0195] The device collects data from the physical space. Specifically, it uses a laser sensor to obtain shape data of the object, a camera to capture color images, and a temperature sensor to obtain environmental data. This data is then sent to a server in real time.

[0196] Input: Physical space geometry, color image, temperature data

[0197] Output: Package of collected shape data, image data, and temperature data

[0198] Step 2:

[0199] Data analysis

[0200] The server receives the data sent from the device and begins analysis. It generates a 3D point cloud based on the laser scan data, identifies objects from camera images, and analyzes data from the temperature sensor to calculate the temperature distribution.

[0201] Input: Collected shape data, image data, temperature data

[0202] Output: 3D point cloud, identified object information, temperature distribution data

[0203] Step 3:

[0204] Data Completion

[0205] The server uses generative artificial intelligence (AI) to fill in any missing data based on the analyzed data. It also connects with the database to fill in any missing data. It also applies anomaly detection algorithms to detect abnormalities.

[0206] Input: 3D point cloud, identified object information, temperature distribution data

[0207] Output: Interpolated 3D point cloud, anomaly detection results

[0208] Step 4:

[0209] VR data generation

[0210] The server converts the data from the interpolated 3D point cloud into a format that can be used by the VR system, and optimizes the generated 3D model into a format suitable for the VR device.

[0211] Input: Interpolated 3D point cloud

[0212] Output: Data format optimized for VR systems

[0213] Step 5:

[0214] Providing a VR environment

[0215] The user wears VR goggles and accesses the VR space using gloves and an audio device. The user operates the VR space visually and tactilely, and is notified via the audio device if an abnormality is detected.

[0216] Input: Data format optimized for VR systems, anomaly detection results

[0217] Output: VR space operated by the user, abnormality notification

[0218] Step 6:

[0219] On-site simulation and response

[0220] Users can check the status of equipment in the VR space and operate any abnormalities. Specifically, they can access the abnormal area with gloves, check temperature data, and receive audio guidance through an audio device. They can then identify the cause of the abnormality and take countermeasures.

[0221] Input: VR space, audio guide, temperature data

[0222] Output: User operation results, identification of the cause of the abnormality, and countermeasures

[0223] In this way, users can efficiently monitor equipment in factories and respond to abnormalities from a remote location, eliminating the need to go to the site in person.

[0224] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0225] The present invention combines a system that collects data from a physical space, analyzes it to generate a virtual reality space (hereinafter referred to as a VR space), and provides an interface for a user to operate the VR space with an emotion engine that recognizes the user's emotions. An embodiment of the present invention will be described in detail below.

[0226] System Configuration

[0227] 1. Data collection terminal

[0228] The device is equipped with a laser sensor and a camera to collect geometric and image data of the physical space, and also collects environmental data using temperature sensors, etc.

[0229] 2. Server

[0230] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space.

[0231] The server is equipped with an emotion engine that recognizes the user's emotions by analyzing the user's facial expressions and voice data.

[0232] 3. VR System

[0233] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[0234] Program processing

[0235] 1. Physical space data collection:

[0236] The device uses a laser sensor to scan the shape of the physical space and obtain the location of objects, a camera to capture color images and perform object recognition, and a temperature sensor to collect environmental data, which is then compressed, packaged, and sent to a server in real time.

[0237] 2. Data processing and VR space generation:

[0238] The server receives the collected data and begins analysis. It generates a 3D point cloud from the laser scan data and fills in any missing data using generative AI and a database. This creates a complete 3D model. The generated 3D model is then converted into a data format optimized for VR systems.

[0239] 3. Emotion Recognition with Emotion Engine:

[0240] The emotion engine installed on the server analyzes facial expression and voice data acquired from the user to recognize the user's emotions. For example, it scans the user's face with a camera and captures the user's voice with a voice input device.

[0241] 4. VR space manipulation and emotional response:

[0242] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins. The user uses the gloves to manipulate objects in the VR space.

[0243] The server's emotion engine monitors the user's emotions in real time and dynamically adjusts the VR environment and interface based on the user's emotions. For example, if the user feels anxious, the system will change the environment to use brighter, warmer colors.

[0244] Specific examples

[0245] 1. Data Collection Example:

[0246] The device scans an office room. It uses a laser sensor to capture the shape of the walls, floor, and ceiling, and a camera to capture the position and shape of each piece of furniture. It uses a temperature sensor to measure the temperature at each point in the room, and sends all the data to a server.

[0247] 2. Data processing examples:

[0248] The server analyzes the received data and generates a 3D model of the office, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[0249] 3. Example of emotion recognition and VR space adjustment:

[0250] Users put on VR goggles to access a virtual office and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature sensor data. They receive audio guidance through the audio glasses, identify the cause, and share steps to take with other teams.

[0251] If the server's emotion engine detects the user's anxiety, it will adjust the lighting in the VR space and play relaxing music to reduce the user's stress.

[0252] In this way, users can remotely acquire and manipulate detailed data of physical spaces, enabling them to carry out on-site investigations and troubleshoot problems with great efficiency and accuracy. The emotion engine also enables flexible responses that adapt to the user's emotions.

[0253] The processing flow will be explained below.

[0254] Step 1:

[0255] The device initializes the sensors: it activates the laser sensor, camera, and temperature sensor and checks that they are working properly.

[0256] Step 2:

[0257] The device begins scanning the physical space, using laser sensors to capture 3D positional data of the entire room and cameras to capture color images, while temperature sensors gather environmental data.

[0258] Step 3:

[0259] The terminal compresses and packages the collected data in real time, where it is formatted and ready to be sent to the server.

[0260] Step 4:

[0261] The device sends the compressed data packets over the network to the server, which monitors the communication to ensure the data transfer is successful.

[0262] Step 5:

[0263] The server receives data sent from the device, such as 3D position data, color images, and temperature data, and stores them in a waiting state for processing.

[0264] Step 6:

[0265] The server analyzes the received data and generates a 3D point cloud, building a complete 3D model of the physical space based on the laser scan data.

[0266] Step 7:

[0267] The server complements missing parts of the generated 3D model by working with generative artificial intelligence (AI) and a database. Missing objects and texture information are generated by AI and retrieved from the database.

[0268] Step 8:

[0269] The server optimizes the interpolated 3D model for the VR system, reducing the data size and converting it into a format compatible with VR goggles.

[0270] Step 9:

[0271] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins.

[0272] Step 10:

[0273] The user manipulates objects in the VR space using the glove, and receives real-time audio guidance and explanations through the audio glasses to operate the system.

[0274] Step 11:

[0275] The server's emotion engine analyzes the user's facial expression and voice data in real time, thereby recognizing the user's emotions.

[0276] Step 12:

[0277] The server dynamically adjusts the environment and interface of the VR space based on the emotions recognized by the emotion engine. For example, if the user feels anxious, the system will change the environment to use brighter, warmer colors.

[0278] Step 13:

[0279] Users can conduct real-time on-site investigations and troubleshoot problems within the VR space. If an abnormality is discovered, they can check the details and plan the necessary countermeasures.

[0280] Step 14:

[0281] The server reflects user operation information in real time and updates the data for the entire system. It also saves necessary data based on user instructions and generates reports to manage work history.

[0282] As described above, the system of the present invention includes a series of processes from collecting data from the physical space to generating and operating the VR space, enabling efficient and accurate on-site investigations and troubleshooting. Furthermore, the emotion engine enables flexible responses that adapt to the user's emotions.

[0283] Example 2

[0284] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0285] Current virtual reality (VR) systems are capable of collecting and analyzing highly accurate physical space data, but lack the ability to recognize the user's emotional state and dynamically adjust the VR environment accordingly. This can lead to stressful or unpleasant experiences for users. Furthermore, when there are gaps or missing data in the process of collecting and analyzing physical space data, it is difficult to easily fill in the gaps.

[0286] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting data of a physical space, means for analyzing the collected data and generating a virtual reality space, means for providing an interface for a user to operate the virtual reality space, means for recognizing the user's emotions in real time, and means for dynamically adjusting the environment of the virtual reality space based on the user's emotions. This makes it possible to adapt to the user's emotional state and provide a comfortable, stress-free VR experience. Furthermore, by utilizing generative artificial intelligence and a database, it is possible to effectively compensate for insufficient or missing collected data.

[0287] "Physical space" refers to a three-dimensional space that exists in the real world.

[0288] "Data collection means" refers to devices and technologies that measure and acquire information about the shape and environment of physical spaces.

[0289] A "laser sensor" refers to a device that uses a light laser to scan the shape data of physical space with high precision.

[0290] "Camera" refers to a photographing device for acquiring image data.

[0291] "Virtual reality space" refers to a three-dimensional virtual reality space generated on a computer.

[0292] "Analysis tools" refers to the technology and devices that process collected data and convert it into meaningful information.

[0293] "Generative AI" refers to artificial intelligence techniques used to fill in gaps in collected data.

[0294] A "database" refers to a collection of information that is systematically organized and stored so that it can be accessed and used as needed.

[0295] "Interface" refers to the devices and software that allow a user to interact with and manipulate a system or data.

[0296] "Means for recognizing emotions in real time" refers to technology or devices that analyze a user's facial expressions and voice data to identify their current emotional state.

[0297] "Means for dynamic adjustment" refers to technology or devices that instantly change environmental settings to suit the user's state based on information obtained in real time.

[0298] "Server" refers to a high-performance computing device responsible for data processing, analysis, storage, and interfacing with other systems.

[0299] "User" refers to a person who uses this system to operate the VR space.

[0300] This invention relates to a system that generates a virtual reality (VR) space by collecting and analyzing data from a physical space, and dynamically adjusts the VR space by recognizing the user's emotions. This system is mainly composed of three main components: a terminal, a server, and a VR system used by the user.

[0301] 1. Data collection by device

[0302] The device uses a variety of sensors to collect detailed data about the physical space. The device uses the following hardware:

[0303] Laser sensor: Scans the shape of physical space with high precision.

[0304] Camera: Photographs objects and their arrangement in physical space and acquires color image data.

[0305] Temperature sensor: Collects ambient environmental data.

[0306] The data acquired by these sensors is compressed, packaged, and sent to a server in real time.

[0307] Examples:

[0308] When scanning an office room, a laser sensor scans the shape of the walls, floor, and ceiling, and a camera captures the position and shape of the furniture. A temperature sensor measures the temperature of the room, and all this data is sent to a server. An example prompt is, "Which sensor should be used to collect office shape and temperature data?"

[0309] 2. Data processing and VR space generation by the server

[0310] The server receives the data sent from the terminal and performs a series of processes. The main software and technologies used are as follows:

[0311] Generative artificial intelligence (AI): Complementing gaps in collected data.

[0312] Database: Provides necessary information to complete the data.

[0313] 3D point cloud generation system: Generates 3D point clouds based on laser scan data.

[0314] VR data optimization system: Converts the generated 3D model into a data format optimized for the VR system.

[0315] Examples:

[0316] The server receives the office data and generates a 3D point cloud. It uses generative AI to fill in any incomplete data and create a complete 3D model. It then converts this 3D model into a data format optimized for the VR system. An example prompt might be, "Please tell me the steps to convert the data received by the server into a 3D point cloud and fill in any missing data using generative AI."

[0317] 3. Emotion Recognition by Emotion Engine

[0318] The server is equipped with an emotion engine that recognizes the user's emotions in real time. This engine uses the following data:

[0319] Facial expression data: The camera scans the user's face and analyzes their facial expressions.

[0320] Voice Data: We capture and analyze your voice using your voice input device.

[0321] Examples:

[0322] While the user is wearing the VR goggles, the server's camera scans the user's face and captures their facial expression data. The user's voice is also captured by a voice input device, and the emotion engine analyzes this data in real time. An example of a prompt sentence is, "Please tell me how to analyze the user's emotions in a VR environment in real time and adjust the environment based on those emotions."

[0323] 4. User manipulation of VR space and emotional response

[0324] Users operate the VR space by wearing the following devices.

[0325] VR goggles: Displays visual information.

[0326] Gloves: Allows for tactile control.

[0327] Audio Glasses: Receive audio guides and explanations in real time.

[0328] The server monitors the user's emotional state and dynamically adjusts the VR environment based on the user's emotions.

[0329] Examples:

[0330] When a user puts on VR goggles and accesses a virtual office, they discover an abnormal temperature rise. The server's emotion engine detects the user's anxiety and adjusts the lighting in the VR space and plays relaxing music to reduce stress.

[0331] In this way, our invention collects and analyzes detailed data on the physical space, creates a VR space that is easy for the user to operate, and recognizes and adapts to the user's emotions in real time, thereby realizing a comfortable and efficient VR experience.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1:

[0334] The device collects data on the physical space. Specifically, it uses a laser sensor to scan the shape of the physical space and generate 3D point cloud data. It also uses a camera to capture color images of the physical space and collect object placement data. In addition, it uses a temperature sensor to capture environmental data. This data is compressed and packaged in real time and sent to the server.

[0335] Input: Physical space information

[0336] Output: Compressed and packaged shape data, color image data, and environmental data

[0337] Step 2:

[0338] The server receives the data sent from the terminal. To analyze the received data, it first extracts the data and performs preprocessing according to the format.

[0339] Input: Compressed data sent from the terminal

[0340] Output: Extracted shape data, color image data, environmental data

[0341] Step 3:

[0342] The server generates a 3D point cloud based on the received laser scan data. Missing data is then filled in using the generation AI and database. Incomplete scan data is input, and the generation AI predicts and fills in the missing parts to output 3D model data.

[0343] Input: extracted shape data

[0344] Output: Interpolated 3D point cloud data

[0345] Step 4:

[0346] The server converts the generated 3D model into a data format optimized for the VR system, performing optimization processing to reduce data size and rendering time. The server takes the 3D point cloud as input and outputs optimized data suitable for the VR system.

[0347] Input: Interpolated 3D point cloud data

[0348] Output: Data optimized for VR systems

[0349] Step 5:

[0350] The server acquires the user's facial expression and voice data and analyzes it using an emotion engine. By analyzing facial features captured by the camera and recording and analyzing the user's voice, the server identifies the user's emotional state in real time. The acquired facial expression and voice data are used as input to output the user's emotional state.

[0351] Input: facial expression data, voice data

[0352] Output: Data on the user's emotional state

[0353] Step 6:

[0354] The user puts on VR goggles, gloves, and audio glasses to access the VR environment. Optimized data is received from the server, and the experience in the VR space begins. The user uses the gloves to manipulate objects in the VR space, and receives audio guidance and explanations through the audio glasses.

[0355] Input: Data optimized for VR systems

[0356] Output: VR experience according to user operations

[0357] Step 7:

[0358] The server's emotion engine analyzes and monitors the user's emotional state in real time. If the user feels anxious or stressed, the server dynamically adjusts the VR environment, for example, by brightening the colors or playing relaxing music.

[0359] Input: User's emotional state data

[0360] Output: Calibrated VR environment

[0361] Through these processing steps, our system can efficiently collect and analyze detailed data from the physical space and generate a VR space. Furthermore, by dynamically adapting to the user's emotional state, it can provide an optimal VR experience.

[0362] (Application example 2)

[0363] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] Conventional virtual reality (VR) systems have been successful in collecting and reproducing data from physical spaces, but they have not been able to dynamically adjust to suit the user's emotions. As a result, they are unable to respond adequately to situations that cause stress or that require increased satisfaction, limiting their ability to improve the quality of the user's experience. Furthermore, in brick-and-mortar stores, there is a need for technology that can optimize the shopping experience by changing the store environment in response to the user's emotions.

[0365] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0366] In this invention, the server includes a means for collecting data on the physical space, a means for analyzing the collected data and generating a virtual reality space, a means for providing an interface for the user to operate the virtual reality space, and a means for recognizing the user's emotions and dynamically adjusting the environment of the virtual reality space. This allows the user to remotely obtain and manipulate detailed data on the physical space and further enables flexible responses that adapt to the user's emotions. Furthermore, in physical stores, lighting and music can be adjusted in real time according to the user's emotions, providing a comfortable shopping experience.

[0367] "Physical space" refers to the space that exists in the real world and the objects placed within it.

[0368] "Data collection means" refers to methods and devices that use equipment such as laser sensors and cameras to acquire information about the shape and environment of a physical space.

[0369] "Analysis means" refers to the methods and devices that process collected data and convert it into meaningful information.

[0370] "Virtual reality space" refers to a realistic three-dimensional virtual space generated using computer technology.

[0371] "Generation means" refers to the method or device that creates a virtual reality space based on collected data.

[0372] "Interface" refers to the input devices and software that allow users to interact with the virtual reality space.

[0373] "Emotion recognition means" refers to a method or device that analyzes a user's facial expressions and voice and identifies their emotions.

[0374] "Environment adjustment means" refers to a method or device that dynamically changes lighting, music, and other environmental elements in a virtual reality space based on the user's emotions.

[0375] "Server" refers to a computer system for collecting, analyzing, and storing data, and generating virtual reality spaces.

[0376] This invention combines a system that collects data from physical spaces, analyzes it to generate a virtual reality space (hereinafter referred to as VR space), and provides an interface for users to operate the VR space with an emotion engine that recognizes the user's emotions.

[0377] System Configuration

[0378] 1. Data collection terminal

[0379] The device is equipped with a laser sensor and a camera to collect shape and image data of the physical space. It also collects environmental data using temperature sensors, etc. This allows detailed data of the physical space to be obtained.

[0380] 2. Server

[0381] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space. The server also has an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions by analyzing the user's facial expressions and voice data.

[0382] 3. VR System

[0383] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[0384] Program processing description

[0385] Physical space data collection

[0386] The device uses a laser sensor to scan and capture geometric data of the physical space. It also uses a camera to identify the placement of objects and collects this data as image data. Environmental data is also collected using a temperature sensor. This data is compressed and sent to a server in real time.

[0387] Data processing and VR space generation

[0388] The server analyzes the data received from the device and generates a 3D point cloud, and missing data is filled in using generative AI and a database, resulting in a complete 3D model that is then converted into a format optimized for the VR system.

[0389] Emotion recognition by emotion engine

[0390] The server's emotion engine analyzes facial expression and voice data acquired from the user and recognizes the user's emotions in real time. For example, it can grasp emotions by scanning the user's face with a camera and capturing the user's voice with a voice input device.

[0391] Manipulating VR space and responding to emotions

[0392] The user puts on VR goggles to access the VR space. They start the experience using optimization data sent from the server and use the gloves to manipulate objects in the VR space. The server's emotion engine monitors the user's emotions in real time and dynamically adjusts the environment and interface of the VR space based on their emotions. For example, if the user is feeling stressed, the system will change the environment using brighter, warmer colors.

[0393] Specific examples

[0394] Data Collection Example

[0395] The device scans an office room, capturing the room's shape with a laser sensor, photographing the furniture layout with a camera, collecting temperature data with a temperature sensor, and sending all data to a server.

[0396] Data Processing Examples

[0397] The server analyzes the received data and generates a 3D model of the office. Imperfections are filled in by generative AI, and optimized VR data is created and exported to the VR system.

[0398] Example of emotion recognition and VR space adjustment

[0399] The user puts on VR goggles to access the virtual office and discovers an abnormal temperature rise. They manipulate objects with the gloves to check temperature sensor data and listen to audio guidance through the audio glasses. If the server's emotion engine detects the user's anxiety, the system adjusts the lighting and plays relaxing music.

[0400] Prompt Sentence Examples

[0401] "Please explain how we can use VR space and emotion engines to test the most effective placement of new in-store promotional placements."

[0402] In this way, users can remotely obtain and manipulate detailed data of physical spaces, enabling highly efficient and accurate on-site investigations and troubleshooting. The emotion engine also enables flexible responses that adapt to the user's emotions.

[0403] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0404] Step 1:

[0405] The device uses a laser sensor to scan and capture geometric data of the physical space, which is then processed into an initial 3D point cloud and combined with image data captured by the camera, which is then compressed and sent to a server in real time.

[0406] (Input: Shape of physical space, image data, Output: 3D point cloud data)

[0407] Step 2:

[0408] The server analyzes the 3D point cloud data and image data received from the device, uses generative artificial intelligence (AI) to fill in any incomplete data, and generates a complete 3D model. It then converts this 3D model into a data format optimized for the VR system.

[0409] (Input: 3D point cloud data, image data, Output: complete 3D model)

[0410] Step 3:

[0411] The user puts on the VR goggles and accesses the VR space. The server sends optimized 3D model data to the VR goggles, allowing the user to begin operating within the VR space. The user uses the gloves to manipulate objects within the VR space and receives audio guidance and explanations through the audio glasses.

[0412] (Input: VR goggles, gloves, audio glasses, Output: Visual and tactile manipulation of VR space)

[0413] Step 4:

[0414] The server's emotion engine analyzes the user's facial expressions and voice data. It recognizes emotions by scanning the user's face with a camera and capturing the user's voice with a voice input device. This allows the system to grasp the user's emotional state, such as stress level and excitement level, in real time.

[0415] (Input: facial expression data, voice data, output: user's emotional state)

[0416] Step 5:

[0417] The server dynamically adjusts the VR environment based on the user's emotional state. For example, if the user is feeling stressed, the server softens the lighting and plays relaxing music. If the user is excited, the server brightens the lighting and plays upbeat music. This provides the optimal VR experience according to the user's emotions.

[0418] (Input: user's emotional state, Output: adjusted VR environment)

[0419] Step 6:

[0420] After the user finishes their work in the VR space, the server saves the collected data for future use. It also analyzes the user's emotional data and operation logs to gather feedback for further optimization of the experience. Based on this, the server makes improvements to ensure the next VR experience is more comfortable and efficient.

[0421] (Input: Operation and emotion data during VR experience, Output: Saved data, feedback results)

[0422] In this way, the system obtains detailed data of the physical space to generate a VR space, and dynamically adjusts it according to the user's emotions to provide an optimal VR experience.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0424] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0426] [Second embodiment]

[0427] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0428] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0429] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0431] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0433] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0434] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0435] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0438] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0439] The present invention is a system that collects data from a physical space, analyzes the data, generates a virtual reality space (hereinafter referred to as a VR space), and provides an interface for a user to operate the VR space. An embodiment of the present invention will be described in detail below.

[0440] System Configuration

[0441] 1. Data collection terminal

[0442] The device is equipped with a laser sensor and a camera, which allows it to collect detailed geometric and image data of the physical space, as well as environmental data using temperature sensors and other sensors.

[0443] 2. Server

[0444] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space.

[0445] 3. VR System

[0446] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[0447] Program processing

[0448] 1. Physical space data collection:

[0449] The device uses a laser sensor to scan the shape of the physical space and obtain the location of objects, a camera to capture color images and perform object recognition, and a temperature sensor to collect environmental data, which is then compressed, packaged, and sent to a server in real time.

[0450] 2. Data processing and VR space generation:

[0451] The server receives the collected data and begins analysis. It generates a 3D point cloud from the laser scan data and fills in any missing data using generative AI and a database. This creates a complete 3D model. The generated 3D model is then converted into a data format optimized for VR systems.

[0452] 3. VR space operation:

[0453] The user wears VR goggles and uses gloves to navigate within the VR space. The VR system receives optimization data sent from the server and displays it to the user. The user can receive real-time audio guidance and explanations through the audio glasses, and can display and operate information using voice commands.

[0454] Specific examples

[0455] 1. Data collection example:

[0456] The device scans an office room. It uses a laser sensor to capture the shape of the walls, floor, and ceiling, and a camera to capture the position and shape of each piece of furniture. It uses a temperature sensor to measure the temperature at each point in the room, and sends all the data to a server.

[0457] 2. Data processing examples:

[0458] The server analyzes the received data and generates a 3D model of the office, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[0459] 3. Examples of on-site investigations and troubleshooting:

[0460] Users put on VR goggles to access a virtual office and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature sensor data. They receive audio guidance through the audio glasses, identify the cause, and share steps to take with other teams.

[0461] In this way, users can remotely capture and manipulate detailed data about physical spaces, enabling highly efficient and accurate on-site inspections and troubleshooting.

[0462] The processing flow will be explained below.

[0463] Step 1:

[0464] The device initializes the sensors: it activates the laser sensor, camera, and temperature sensor and checks that they are working properly.

[0465] Step 2:

[0466] The device begins scanning the physical space, using laser sensors to capture 3D positional data of the entire room and cameras to capture color images, while temperature sensors gather environmental data.

[0467] Step 3:

[0468] The terminal compresses and packages the collected data in real time, where it is formatted and ready to be sent to the server.

[0469] Step 4:

[0470] The device sends the compressed data packets over the network to the server, which monitors the communication to ensure the data transfer is successful.

[0471] Step 5:

[0472] The server receives data sent from the device, such as 3D position data, color images, and temperature data, and stores them in a waiting state for processing.

[0473] Step 6:

[0474] The server analyzes the received data and generates a 3D point cloud, building a complete 3D model of the physical space based on the laser scan data.

[0475] Step 7:

[0476] The server complements missing parts of the generated 3D model by working with generative artificial intelligence (AI) and a database. Missing object and texture information is generated by AI and retrieved from the database.

[0477] Step 8:

[0478] The server optimizes the interpolated 3D model for the VR system, reducing the data size and converting it into a format compatible with VR goggles.

[0479] Step 9:

[0480] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins.

[0481] Step 10:

[0482] The user uses the glove to manipulate objects in the VR space, and receives real-time audio guidance and explanations through the audio glasses to operate within the system.

[0483] Step 11:

[0484] Users can conduct real-time on-site investigations and troubleshoot problems within the VR space. If an abnormality is discovered, they can check the details and plan the necessary countermeasures.

[0485] Step 12:

[0486] The server reflects user operation information in real time and updates the data for the entire system. It also saves necessary data based on user instructions and generates reports to manage work history.

[0487] As described above, the system of the present invention includes a series of processes from collecting data on the physical space to generating and operating the VR space, enabling efficient and accurate on-site investigations and troubleshooting.

[0488] Example 1

[0489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] There is a demand for technology that can quickly and accurately collect data from physical spaces and efficiently reproduce and manipulate it in virtual reality spaces. In particular, advanced processing power and collaboration are required to complement missing data and optimize 3D models, but current systems are unable to meet these requirements. Therefore, the challenge is to generate highly accurate 3D models and provide an environment in which users can intuitively operate them.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0492] In this invention, the server includes a means for receiving collected data, a means for analyzing the received data and generating a 3D point cloud, a means for generating a complete 3D model by utilizing generative artificial intelligence and a database to supplement missing data, and a means for providing the optimized 3D model to the user's virtual reality space, thereby enabling detailed data of a physical space to be collected quickly and accurately and efficiently reproduced and manipulated in a virtual space.

[0493] "Physical space" refers to the space in which a user acts in the real world, an environment with a concrete shape and layout.

[0494] "Data collection means" refers to a combination of hardware and software for collecting shape data, image data, environmental data, etc. of a physical space.

[0495] A "server" is a high-performance computer system that receives, analyzes, and processes data, and has the ability to link with generative artificial intelligence and databases.

[0496] A "laser sensor" is a sensor that uses laser light to measure the distance and shape of an object.

[0497] A "camera" is an optical device for capturing image data of physical space.

[0498] A "temperature sensor" is a sensor for measuring the ambient temperature of a physical space.

[0499] A "3D point cloud" is a three-dimensional data set composed of point cloud data acquired from laser sensors and other measurement devices.

[0500] "Generative AI" is AI that has the ability to use existing data to fill in missing data or generate new data.

[0501] A "database" is a system that can efficiently store and search large amounts of data.

[0502] "User" refers to a person or organization that uses the system to manipulate data in physical space.

[0503] "Virtual reality space" refers to a virtual three-dimensional space generated by computer technology that provides an environment in which users can engage in activities within that space.

[0504] "VR goggles" are devices that allow users to visually experience virtual reality spaces.

[0505] A "glove" is an input device that a user uses to manipulate objects in a virtual reality space.

[0506] An "audio device" is an audio-related device that allows a user to use voice guidance and commands.

[0507] This invention is a system that collects detailed data of a physical space, analyzes it to generate a virtual reality space (hereinafter referred to as VR space), and allows users to operate the VR space in real time.

[0508] Data collection terminal

[0509] The device is equipped with a laser sensor, camera, and temperature sensor to collect geometric data, image data, and environmental data of the physical space. Specifically, the laser sensor collects detailed geometric data of the space, and the camera takes high-resolution image data. The temperature sensor measures the environmental temperature at multiple points, compresses and packages all the data, and sends it to the server in real time.

[0510] Server data analysis

[0511] The server is equipped with a high-performance processor and storage to analyze the received data. The server first generates a 3D point cloud from the received laser scan data and integrates it with image and temperature data acquired from the camera. Missing data is then supplemented using generative artificial intelligence (generative AI models) and a database to generate a complete 3D model. The generated 3D model is then converted into a data format optimized for the VR system.

[0512] VR system operation

[0513] Users manipulate the VR space using VR goggles, gloves, and an audio device. The VR system receives optimization data sent from the server and provides the user with a highly accurate virtual space. Users can manipulate objects in the virtual space using the gloves and receive real-time audio guidance and explanations through the audio device. They can also use voice commands to obtain specific information or perform operations.

[0514] Specific examples

[0515] 1. Data collection example:

[0516] The device uses a laser sensor to scan an office room, collecting geometric data on the walls, floors, ceilings, and furniture. It also uses a camera to capture a panoramic view of the office and a temperature sensor to create a temperature map of the room. This data is then sent to a server in real time.

[0517] 2. Data processing examples:

[0518] The server analyzes the received data and uses generative AI models to fill in any incomplete data, resulting in a complete, optimized 3D model of the office.

[0519] 3. Examples of on-site investigations and troubleshooting:

[0520] Users access the virtual office by wearing VR goggles, manipulate objects with gloves, and receive real-time voice guidance about abnormal temperature rises through an audio device. They can obtain additional information through voice commands and identify the cause of the temperature anomaly.

[0521] Prompt Sentence Examples

[0522] "Check for temperature abnormalities in the VR space and explain the corrective steps using audio."

[0523] "Please explain in detail the process from collecting data from the physical space to generating the VR space."

[0524] In this way, the present invention allows detailed data of a physical space to be quickly and accurately collected and efficiently reproduced and manipulated in a virtual space.

[0525] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0526] Step 1:

[0527] The device collects data on the physical space. First, it uses a laser sensor to scan the shape of the physical space and collects layout data on walls, floors, ceilings, furniture, etc. Next, it uses a camera to capture high-resolution color images. It also uses a temperature sensor to measure the ambient temperature at multiple points. This data is then combined to create a detailed dataset of the physical space. This dataset is then compressed and sent to a server in real time.

[0528] Input: Physical space

[0529] Output: Compressed package of geometry, image data, and temperature data

[0530] Step 2:

[0531] The server receives data sent from the device. It checks the data for consistency and requests the device to resend it if it contains incomplete data. It then analyzes the received laser scan data, image data, and temperature data to generate a 3D point cloud. It then builds a detailed 3D model of the physical space based on the point cloud data.

[0532] Input: Compressed data sent from the terminal

[0533] Output: 3D point cloud data, initial 3D model

[0534] Step 3:

[0535] The server uses generative artificial intelligence (generative AI model) and a database to fill in missing or incomplete data. The generative AI model naturally fills in missing parts based on existing data. A complete 3D model is constructed based on the analyzed data.

[0536] Input: initial 3D model, generative AI model, database

[0537] Output: Full 3D model

[0538] Step 4:

[0539] The server processes the generated 3D model to optimize it for the VR system. It performs optimization processes such as simplifying the mesh, optimizing the texture, and adjusting the frame rate to ensure smooth display in the VR space. The optimized data is then sent to the user's VR system.

[0540] Input: A complete 3D model

[0541] Output: Optimized VR data

[0542] Step 5:

[0543] Users interact with the VR space using VR goggles, gloves, and an audio device. The VR system receives optimized data from the server and displays a highly accurate virtual space. Users use the gloves to manipulate objects in the virtual space and receive real-time audio guidance and explanations through the audio device. Voice commands can also be used to obtain specific information or perform operations.

[0544] Input: Optimized VR data

[0545] Output: High-precision virtual space displayed to the user, audio guidance, and operation results

[0546] (Application example 1)

[0547] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] In modern factories, monitoring and maintenance of complex equipment is important, but it is necessary to dispatch specialized engineers to the site, which is time-consuming and costly. Furthermore, when an abnormality occurs, a rapid response is required, but this response can be delayed if the engineer is located in a remote location. Furthermore, when an abnormality is detected, it can be difficult to obtain detailed environmental and physical information, making troubleshooting difficult. There is a need to solve these issues and achieve more efficient monitoring and maintenance of equipment within factories and a faster response to abnormalities.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0550] In this invention, the server includes a means for collecting data on the physical space, a means for analyzing the collected data and generating a virtual reality space, a means for providing an interface for a user to operate the virtual reality space, and a means for detecting abnormalities and notifying them in real time. This makes it possible to monitor and operate detailed equipment status in a factory in real time even from a remote location, and to respond quickly even if an abnormality is detected.

[0551] "Physical space" refers to a space that exists in the real world and has a specific size and shape.

[0552] "Data collection means" refers to devices and technologies used to acquire geometric and environmental data from physical spaces.

[0553] "Analysis" refers to examining collected data in detail to understand and process its structure and characteristics.

[0554] A "virtual reality space" is a virtual three-dimensional space generated by a computer that users can experience through their senses of sight and touch.

[0555] "Means for providing an interface" refers to devices and software that allow a user to operate a virtual reality space.

[0556] "Means for detecting anomalies" refers to devices or technologies for detecting unusual behavior or values ​​from collected data.

[0557] "Means for real-time notification" refers to devices or technologies that immediately convey information to users when an abnormality is detected.

[0558] A "laser sensor" is a device that uses light to scan the shape of physical space and obtain accurate three-dimensional data.

[0559] A "camera" is a device that takes pictures or videos and stores them as digital data.

[0560] "Generative AI" is an AI technology that automatically generates missing information based on collected data.

[0561] A "database" is a system that organizes and stores large amounts of data and allows fast access when needed.

[0562] A specific embodiment of the present invention will be described below. This system collects and analyzes data from a physical space to generate a virtual reality (VR) space, allowing users to manipulate the VR space.

[0563] System Configuration

[0564] 1. Data collection terminal

[0565] The device is equipped with a laser sensor and a camera to collect detailed geometric and image data of the physical space, and also collects environmental data using a temperature sensor, allowing the shape, layout, and environmental data of the physical space to be obtained in real time.

[0566] 2. Server

[0567] The server has a high-performance processor and storage for receiving and analyzing data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space. It also has the ability to detect abnormalities and send notifications in real time.

[0568] 3. VR System

[0569] The VR system used by users consists of VR goggles, gloves, and an audio device. This allows users to perform visual and tactile manipulation within the VR space. Users can also receive real-time audio guidance and explanations through the audio device.

[0570] Program processing

[0571] Data collection and transmission

[0572] The device uses a laser sensor to scan the physical space and generate a 3D point cloud, while a camera captures color images for object identification. A temperature sensor also collects environmental data, which is then compressed, packaged, and sent to a server in real time.

[0573] Data processing and VR space generation

[0574] The server analyzes the received data, generates a 3D point cloud from the laser scan data, and fills in any missing data using generative AI and a database. This generates a complete 3D model, which is then converted into a data format optimized for VR systems.

[0575] Interface with VR space

[0576] The user wears VR goggles and uses gloves to navigate within the VR space. The VR system receives optimized data sent from the server and displays it to the user. The user receives real-time audio guidance and explanations through the audio device, and uses voice commands to display and operate information.

[0577] Specific examples

[0578] 1. Data collection example:

[0579] The terminal scans a section of the factory, captures the shape of the equipment with a laser sensor, photographs the layout of each piece of equipment with a camera, measures the temperature of each piece of equipment with a temperature sensor, and sends all the data to a server.

[0580] 2. Data processing examples:

[0581] The server analyzes the received data and generates a 3D model of the factory, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[0582] 3. Examples of anomaly detection and response:

[0583] Users put on VR goggles to access a virtual factory and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature data. They receive audio guidance through an audio device, identify the cause, and share the steps to take to address the issue with other team members.

[0584] Examples of prompt statements

[0585] "Generate a 3D point cloud from the factory's lidar and camera data and fill in any gaps. Model all the equipment in the factory in detail and convert it into a data format optimized for VR systems and output it."

[0586] In this way, users can remotely acquire and manipulate detailed data about the physical space, enabling highly efficient and accurate monitoring of factory equipment and response to anomalies.

[0587] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0588] Step 1:

[0589] Data collection

[0590] The device collects data from the physical space. Specifically, it uses a laser sensor to obtain shape data of the object, a camera to capture color images, and a temperature sensor to obtain environmental data. This data is then sent to a server in real time.

[0591] Input: Physical space geometry, color image, temperature data

[0592] Output: Package of collected shape data, image data, and temperature data

[0593] Step 2:

[0594] Data analysis

[0595] The server receives the data sent from the device and begins analysis. It generates a 3D point cloud based on the laser scan data, identifies objects from camera images, and analyzes data from the temperature sensor to calculate the temperature distribution.

[0596] Input: Collected shape data, image data, temperature data

[0597] Output: 3D point cloud, identified object information, temperature distribution data

[0598] Step 3:

[0599] Data Completion

[0600] The server uses generative artificial intelligence (AI) to fill in any missing data based on the analyzed data. It also connects with the database to fill in any missing data. It also applies anomaly detection algorithms to detect abnormalities.

[0601] Input: 3D point cloud, identified object information, temperature distribution data

[0602] Output: Interpolated 3D point cloud, anomaly detection results

[0603] Step 4:

[0604] VR data generation

[0605] The server converts the data from the interpolated 3D point cloud into a format that can be used by the VR system, and optimizes the generated 3D model into a format suitable for the VR device.

[0606] Input: Interpolated 3D point cloud

[0607] Output: Data format optimized for VR systems

[0608] Step 5:

[0609] Providing a VR environment

[0610] The user wears VR goggles and accesses the VR space using gloves and an audio device. The user manipulates the VR space visually and tactilely, and is notified via the audio device if an abnormality is detected.

[0611] Input: Data format optimized for VR systems, anomaly detection results

[0612] Output: VR space operated by the user, abnormality notification

[0613] Step 6:

[0614] On-site simulation and response

[0615] Users can check the status of equipment in the VR space and operate any abnormalities. Specifically, they can access the abnormal area with gloves, check temperature data, and receive audio guidance through an audio device. They can then identify the cause of the abnormality and take countermeasures.

[0616] Input: VR space, audio guide, temperature data

[0617] Output: User operation results, identification of the cause of the abnormality, and countermeasures

[0618] In this way, users can efficiently monitor equipment in factories and respond to abnormalities from a remote location, eliminating the need to go to the site in person.

[0619] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0620] The present invention combines a system that collects data from a physical space, analyzes it to generate a virtual reality space (hereinafter referred to as a VR space), and provides an interface for a user to operate the VR space with an emotion engine that recognizes the user's emotions. An embodiment of the present invention will be described in detail below.

[0621] System Configuration

[0622] 1. Data collection terminal

[0623] The device is equipped with a laser sensor and a camera to collect geometric and image data of the physical space, and also collects environmental data using temperature sensors, etc.

[0624] 2. Server

[0625] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space.

[0626] The server is equipped with an emotion engine that recognizes the user's emotions by analyzing the user's facial expressions and voice data.

[0627] 3. VR System

[0628] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[0629] Program processing

[0630] 1. Physical space data collection:

[0631] The device uses a laser sensor to scan the shape of the physical space and obtain the location of objects, a camera to capture color images and perform object recognition, and a temperature sensor to collect environmental data, which is then compressed, packaged, and sent to a server in real time.

[0632] 2. Data processing and VR space generation:

[0633] The server receives the collected data and begins analysis. It generates a 3D point cloud from the laser scan data and fills in any missing data using generative AI and a database. This creates a complete 3D model. The generated 3D model is then converted into a data format optimized for VR systems.

[0634] 3. Emotion Recognition with Emotion Engine:

[0635] The emotion engine installed on the server analyzes facial expression and voice data acquired from the user to recognize the user's emotions. For example, it scans the user's face with a camera and captures the user's voice with a voice input device.

[0636] 4. VR space manipulation and emotional response:

[0637] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins. The user uses the gloves to manipulate objects in the VR space.

[0638] The server's emotion engine monitors the user's emotions in real time and dynamically adjusts the VR environment and interface based on the user's emotions. For example, if the user feels anxious, the system will change the environment to use brighter, warmer colors.

[0639] Specific examples

[0640] 1. Data collection example:

[0641] The device scans an office room. It uses a laser sensor to capture the shape of the walls, floor, and ceiling, and a camera to capture the position and shape of each piece of furniture. It uses a temperature sensor to measure the temperature at each point in the room, and sends all the data to a server.

[0642] 2. Data processing examples:

[0643] The server analyzes the received data and generates a 3D model of the office, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[0644] 3. Example of emotion recognition and VR space adjustment:

[0645] Users put on VR goggles to access a virtual office and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature sensor data. They receive audio guidance through the audio glasses, identify the cause, and share steps to take with other teams.

[0646] If the server's emotion engine detects the user's anxiety, it will adjust the lighting in the VR space and play relaxing music to reduce the user's stress.

[0647] In this way, users can remotely acquire and manipulate detailed data of physical spaces, enabling them to carry out on-site investigations and troubleshoot problems with great efficiency and accuracy. The emotion engine also enables flexible responses that adapt to the user's emotions.

[0648] The processing flow will be explained below.

[0649] Step 1:

[0650] The device initializes the sensors: it activates the laser sensor, camera, and temperature sensor and checks that they are working properly.

[0651] Step 2:

[0652] The device begins scanning the physical space, using laser sensors to capture 3D positional data of the entire room and cameras to capture color images, while temperature sensors gather environmental data.

[0653] Step 3:

[0654] The terminal compresses and packages the collected data in real time, where it is formatted and ready to be sent to the server.

[0655] Step 4:

[0656] The device sends the compressed data packets over the network to the server, which monitors the communication to ensure the data transfer is successful.

[0657] Step 5:

[0658] The server receives data sent from the device, such as 3D position data, color images, and temperature data, and stores them in a waiting state for processing.

[0659] Step 6:

[0660] The server analyzes the received data and generates a 3D point cloud, building a complete 3D model of the physical space based on the laser scan data.

[0661] Step 7:

[0662] The server complements missing parts of the generated 3D model by working with generative artificial intelligence (AI) and a database. Missing object and texture information is generated by AI and retrieved from the database.

[0663] Step 8:

[0664] The server optimizes the interpolated 3D model for the VR system, reducing the data size and converting it into a format compatible with VR goggles.

[0665] Step 9:

[0666] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins.

[0667] Step 10:

[0668] The user uses the glove to manipulate objects in the VR space, and receives real-time audio guidance and explanations through the audio glasses to operate within the system.

[0669] Step 11:

[0670] The server's emotion engine analyzes the user's facial expression and voice data in real time, thereby recognizing the user's emotions.

[0671] Step 12:

[0672] The server dynamically adjusts the environment and interface of the VR space based on the emotions recognized by the emotion engine. For example, if the user feels anxious, the system will change the environment to use brighter, warmer colors.

[0673] Step 13:

[0674] Users can conduct real-time on-site investigations and troubleshoot problems within the VR space. If an abnormality is discovered, they can check the details and plan the necessary countermeasures.

[0675] Step 14:

[0676] The server reflects user operation information in real time and updates the data for the entire system. It also saves necessary data based on user instructions and generates reports to manage work history.

[0677] As described above, the system of the present invention includes a series of processes from collecting data from the physical space to generating and operating the VR space, enabling efficient and accurate on-site investigations and troubleshooting. Furthermore, the emotion engine enables flexible responses that adapt to the user's emotions.

[0678] Example 2

[0679] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] Current virtual reality (VR) systems are capable of collecting and analyzing highly accurate physical space data, but lack the ability to recognize the user's emotional state and dynamically adjust the VR environment accordingly. This can lead to stressful or unpleasant experiences for users. Furthermore, when there are gaps or missing data in the process of collecting and analyzing physical space data, it is difficult to easily fill in the gaps.

[0681] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting data of a physical space, means for analyzing the collected data and generating a virtual reality space, means for providing an interface for a user to operate the virtual reality space, means for recognizing the user's emotions in real time, and means for dynamically adjusting the environment of the virtual reality space based on the user's emotions. This makes it possible to adapt to the user's emotional state and provide a comfortable, stress-free VR experience. Furthermore, by utilizing generative artificial intelligence and a database, it is possible to effectively compensate for insufficient or missing collected data.

[0682] "Physical space" refers to a three-dimensional space that exists in the real world.

[0683] "Data collection means" refers to devices and technologies that measure and acquire information about the shape and environment of physical spaces.

[0684] A "laser sensor" refers to a device that uses a light laser to scan the shape data of physical space with high precision.

[0685] "Camera" refers to a photographing device for acquiring image data.

[0686] "Virtual reality space" refers to a three-dimensional virtual reality space generated on a computer.

[0687] "Analysis tools" refers to the technology and devices that process collected data and convert it into meaningful information.

[0688] "Generative AI" refers to artificial intelligence techniques used to fill in gaps in collected data.

[0689] A "database" refers to a collection of information that is systematically organized and stored so that it can be accessed and used as needed.

[0690] "Interface" refers to the devices and software that allow a user to interact with and manipulate a system or data.

[0691] "Means for recognizing emotions in real time" refers to technology or devices that analyze a user's facial expressions and voice data to identify their current emotional state.

[0692] "Means for dynamic adjustment" refers to technology or devices that instantly change environmental settings to suit the user's state based on information obtained in real time.

[0693] "Server" refers to a high-performance computing device responsible for data processing, analysis, storage, and interfacing with other systems.

[0694] "User" refers to a person who uses this system to operate the VR space.

[0695] This invention relates to a system that generates a virtual reality (VR) space by collecting and analyzing data from a physical space, and dynamically adjusts the VR space by recognizing the user's emotions. This system is mainly composed of three main components: a terminal, a server, and a VR system used by the user.

[0696] 1. Data collection by device

[0697] The device uses a variety of sensors to collect detailed data about the physical space. The device uses the following hardware:

[0698] Laser sensor: Scans the shape of physical space with high precision.

[0699] Camera: Photographs objects and their arrangement in physical space and acquires color image data.

[0700] Temperature sensor: Collects ambient environmental data.

[0701] The data acquired by these sensors is compressed, packaged, and sent to a server in real time.

[0702] Examples:

[0703] When scanning an office room, a laser sensor scans the shape of the walls, floor, and ceiling, and a camera captures the position and shape of the furniture. A temperature sensor measures the temperature of the room, and all this data is sent to a server. An example prompt is, "Which sensor should be used to collect office shape and temperature data?"

[0704] 2. Data processing and VR space generation by the server

[0705] The server receives the data sent from the terminal and performs a series of processes. The main software and technologies used are as follows:

[0706] Generative artificial intelligence (AI): Complementing gaps in collected data.

[0707] Database: Provides necessary information to complete the data.

[0708] 3D point cloud generation system: Generates 3D point clouds based on laser scan data.

[0709] VR data optimization system: Converts the generated 3D model into a data format optimized for the VR system.

[0710] Examples:

[0711] The server receives the office data and generates a 3D point cloud. It uses generative AI to fill in any incomplete data and create a complete 3D model. It then converts this 3D model into a data format optimized for the VR system. An example prompt might be, "Please tell me the steps to convert the data received by the server into a 3D point cloud and fill in any missing data using generative AI."

[0712] 3. Emotion Recognition by Emotion Engine

[0713] The server is equipped with an emotion engine that recognizes the user's emotions in real time. This engine uses the following data:

[0714] Facial expression data: The camera scans the user's face and analyzes their facial expressions.

[0715] Voice Data: We capture and analyze your voice using your voice input device.

[0716] Examples:

[0717] While the user is wearing the VR goggles, the server's camera scans the user's face and captures their facial expression data. The user's voice is also captured by a voice input device, and the emotion engine analyzes this data in real time. An example of a prompt sentence is, "Please tell me how to analyze the user's emotions in a VR environment in real time and adjust the environment based on those emotions."

[0718] 4. User manipulation of VR space and emotional response

[0719] Users operate the VR space by wearing the following devices.

[0720] VR goggles: Displays visual information.

[0721] Gloves: Allows for tactile control.

[0722] Audio Glasses: Receive audio guides and explanations in real time.

[0723] The server monitors the user's emotional state and dynamically adjusts the VR environment based on the user's emotions.

[0724] Examples:

[0725] When a user puts on VR goggles and accesses a virtual office, they discover an abnormal temperature rise. The server's emotion engine detects the user's anxiety and adjusts the lighting in the VR space and plays relaxing music to reduce stress.

[0726] In this way, our invention collects and analyzes detailed data on the physical space, creates a VR space that is easy for the user to operate, and recognizes and adapts to the user's emotions in real time, thereby realizing a comfortable and efficient VR experience.

[0727] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0728] Step 1:

[0729] The device collects data on the physical space. Specifically, it uses a laser sensor to scan the shape of the physical space and generate 3D point cloud data. It also uses a camera to capture color images of the physical space and collect object placement data. In addition, it uses a temperature sensor to capture environmental data. This data is compressed and packaged in real time and sent to the server.

[0730] Input: Physical space information

[0731] Output: Compressed and packaged shape data, color image data, and environmental data

[0732] Step 2:

[0733] The server receives the data sent from the terminal. To analyze the received data, it first extracts the data and performs preprocessing according to the format.

[0734] Input: Compressed data sent from the terminal

[0735] Output: Extracted shape data, color image data, environmental data

[0736] Step 3:

[0737] The server generates a 3D point cloud based on the received laser scan data. Missing data is then filled in using the generation AI and database. Incomplete scan data is input, and the generation AI predicts and fills in the missing parts to output 3D model data.

[0738] Input: extracted shape data

[0739] Output: Interpolated 3D point cloud data

[0740] Step 4:

[0741] The server converts the generated 3D model into a data format optimized for the VR system, performing optimization processing to reduce data size and rendering time. The server takes the 3D point cloud as input and outputs optimized data suitable for the VR system.

[0742] Input: Interpolated 3D point cloud data

[0743] Output: Data optimized for VR systems

[0744] Step 5:

[0745] The server acquires the user's facial expression and voice data and analyzes it using an emotion engine. By analyzing facial features captured by the camera and recording and analyzing the user's voice, the server identifies the user's emotional state in real time. The acquired facial expression and voice data are used as input to output the user's emotional state.

[0746] Input: facial expression data, voice data

[0747] Output: Data on the user's emotional state

[0748] Step 6:

[0749] The user puts on VR goggles, gloves, and audio glasses to access the VR environment. Optimized data is received from the server, and the experience in the VR space begins. The user uses the gloves to manipulate objects in the VR space, and receives audio guidance and explanations through the audio glasses.

[0750] Input: Data optimized for VR systems

[0751] Output: VR experience according to user operations

[0752] Step 7:

[0753] The server's emotion engine analyzes and monitors the user's emotional state in real time. If the user feels anxious or stressed, the server dynamically adjusts the VR environment, for example by brightening the colors or playing relaxing music.

[0754] Input: User's emotional state data

[0755] Output: Calibrated VR environment

[0756] Through these processing steps, our system can efficiently collect and analyze detailed data from the physical space and generate a VR space. Furthermore, by dynamically adapting to the user's emotional state, it can provide an optimal VR experience.

[0757] (Application example 2)

[0758] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0759] Conventional virtual reality (VR) systems have been successful in collecting and reproducing data from physical spaces, but they have not been able to adequately respond to dynamic adjustments based on the user's emotions. As a result, they are unable to respond adequately to situations that cause stress or that require increased satisfaction, limiting their ability to improve the quality of the user's experience. Furthermore, in physical stores, there is a need for technology that can optimize the shopping experience by changing the store environment based on the user's emotions.

[0760] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0761] In this invention, the server includes a means for collecting data on the physical space, a means for analyzing the collected data and generating a virtual reality space, a means for providing an interface for the user to operate the virtual reality space, and a means for recognizing the user's emotions and dynamically adjusting the environment of the virtual reality space. This allows the user to remotely obtain and manipulate detailed data on the physical space and further enables flexible responses that adapt to the user's emotions. Furthermore, in physical stores, lighting and music can be adjusted in real time according to the user's emotions, providing a comfortable shopping experience.

[0762] "Physical space" refers to the space that exists in the real world and the objects placed within it.

[0763] "Data collection means" refers to methods and devices that use equipment such as laser sensors and cameras to acquire information about the shape and environment of a physical space.

[0764] "Analysis means" refers to the methods and devices that process collected data and convert it into meaningful information.

[0765] "Virtual reality space" refers to a realistic three-dimensional virtual space generated using computer technology.

[0766] "Generation means" refers to the method or device that creates a virtual reality space based on collected data.

[0767] "Interface" refers to the input devices and software that allow users to interact with the virtual reality space.

[0768] "Emotion recognition means" refers to a method or device that analyzes a user's facial expressions and voice and identifies their emotions.

[0769] "Environment adjustment means" refers to a method or device that dynamically changes lighting, music, and other environmental elements in a virtual reality space based on the user's emotions.

[0770] "Server" refers to a computer system for collecting, analyzing, and storing data, and generating virtual reality spaces.

[0771] This invention combines a system that collects data from physical spaces, analyzes it to generate a virtual reality space (hereinafter referred to as VR space), and provides an interface for users to operate the VR space with an emotion engine that recognizes the user's emotions.

[0772] System Configuration

[0773] 1. Data collection terminal

[0774] The device is equipped with a laser sensor and a camera to collect shape and image data of the physical space. It also collects environmental data using temperature sensors, etc. This allows detailed data of the physical space to be obtained.

[0775] 2. Server

[0776] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space. The server also has an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions by analyzing the user's facial expressions and voice data.

[0777] 3. VR System

[0778] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[0779] Program processing description

[0780] Physical space data collection

[0781] The device uses a laser sensor to scan and capture geometric data of the physical space. It also uses a camera to identify the placement of objects and collects this data as image data. Environmental data is also collected using a temperature sensor. This data is compressed and sent to a server in real time.

[0782] Data processing and VR space generation

[0783] The server analyzes the data received from the device and generates a 3D point cloud. Missing data is filled in using generative AI and a database. This generates a complete 3D model, which is then converted into a format optimized for the VR system.

[0784] Emotion recognition by emotion engine

[0785] The server's emotion engine analyzes facial expression and voice data acquired from the user and recognizes the user's emotions in real time. For example, it can grasp emotions by scanning the user's face with a camera and capturing the user's voice with a voice input device.

[0786] Manipulating VR space and responding to emotions

[0787] The user puts on VR goggles to access the VR space. They start the experience using optimization data sent from the server and use the gloves to manipulate objects in the VR space. The server's emotion engine monitors the user's emotions in real time and dynamically adjusts the environment and interface of the VR space based on their emotions. For example, if the user is feeling stressed, the system will change the environment using brighter, warmer colors.

[0788] Specific examples

[0789] Data Collection Example

[0790] The device scans an office room, capturing the room's shape with a laser sensor, photographing the furniture layout with a camera, collecting temperature data with a temperature sensor, and sending all data to a server.

[0791] Data Processing Examples

[0792] The server analyzes the received data and generates a 3D model of the office. Imperfections are filled in by generative AI, and optimized VR data is created and exported to the VR system.

[0793] Example of emotion recognition and VR space adjustment

[0794] The user puts on VR goggles to access the virtual office and discovers an abnormal temperature rise. They manipulate objects with the gloves to check temperature sensor data and listen to audio guidance through the audio glasses. If the server's emotion engine detects the user's anxiety, the system adjusts the lighting and plays relaxing music.

[0795] Prompt Sentence Examples

[0796] "Please explain how we can use VR space and emotion engines to test the most effective placement of new in-store promotional placements."

[0797] In this way, users can remotely obtain and manipulate detailed data of physical spaces, enabling highly efficient and accurate on-site investigations and troubleshooting. The emotion engine also enables flexible responses that adapt to the user's emotions.

[0798] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0799] Step 1:

[0800] The device uses a laser sensor to scan and capture geometric data of the physical space, which is then processed into an initial 3D point cloud and combined with image data captured by the camera, which is then compressed and sent to a server in real time.

[0801] (Input: Shape of physical space, image data, Output: 3D point cloud data)

[0802] Step 2:

[0803] The server analyzes the 3D point cloud data and image data received from the device, uses generative artificial intelligence (AI) to fill in any incomplete data, and generates a complete 3D model. It then converts this 3D model into a data format optimized for the VR system.

[0804] (Input: 3D point cloud data, image data, Output: complete 3D model)

[0805] Step 3:

[0806] The user puts on the VR goggles and accesses the VR space. The server sends optimized 3D model data to the VR goggles, allowing the user to begin operating within the VR space. The user uses the gloves to manipulate objects in the VR space and receives audio guidance and explanations through the audio glasses.

[0807] (Input: VR goggles, gloves, audio glasses, Output: Visual and tactile manipulation of VR space)

[0808] Step 4:

[0809] The server's emotion engine analyzes the user's facial expressions and voice data. It recognizes emotions by scanning the user's face with a camera and capturing the user's voice with a voice input device. This allows the system to grasp the user's emotional state, such as stress level and excitement level, in real time.

[0810] (Input: facial expression data, voice data, output: user's emotional state)

[0811] Step 5:

[0812] The server dynamically adjusts the VR environment based on the user's emotional state. For example, if the user is feeling stressed, the server softens the lighting and plays relaxing music. If the user is excited, the server brightens the lighting and plays upbeat music. This provides the optimal VR experience according to the user's emotions.

[0813] (Input: user's emotional state, Output: adjusted VR environment)

[0814] Step 6:

[0815] After the user finishes their work in the VR space, the server saves the collected data for future use. It also analyzes the user's emotional data and operation logs to gather feedback for further optimization of the experience. Based on this, the server makes improvements to ensure the next VR experience is more comfortable and efficient.

[0816] (Input: Operation and emotion data during VR experience, Output: Saved data, feedback results)

[0817] In this way, the system obtains detailed data of the physical space to generate a VR space, and dynamically adjusts it according to the user's emotions to provide an optimal VR experience.

[0818] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0819] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0820] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0821] [Third embodiment]

[0822] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0823] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0824] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0825] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0826] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0827] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0828] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0829] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0830] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0831] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0832] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0833] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0834] The present invention is a system that collects data from a physical space, analyzes the data, generates a virtual reality space (hereinafter referred to as a VR space), and provides an interface for a user to operate the VR space. An embodiment of the present invention will be described in detail below.

[0835] System Configuration

[0836] 1. Data collection terminal

[0837] The device is equipped with a laser sensor and a camera, which allows it to collect detailed geometric and image data of the physical space, as well as environmental data using temperature sensors and other sensors.

[0838] 2. Server

[0839] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space.

[0840] 3. VR System

[0841] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[0842] Program processing

[0843] 1. Physical space data collection:

[0844] The device uses a laser sensor to scan the shape of the physical space and obtain the location of objects, a camera to capture color images and perform object recognition, and a temperature sensor to collect environmental data, which is then compressed, packaged, and sent to a server in real time.

[0845] 2. Data processing and VR space generation:

[0846] The server receives the collected data and begins analysis. It generates a 3D point cloud from the laser scan data and fills in any missing data using generative AI and a database. This creates a complete 3D model. The generated 3D model is then converted into a data format optimized for VR systems.

[0847] 3. VR space operation:

[0848] The user wears VR goggles and uses gloves to navigate within the VR space. The VR system receives optimization data sent from the server and displays it to the user. The user can receive real-time audio guidance and explanations through the audio glasses, and can display and operate information using voice commands.

[0849] Specific examples

[0850] 1. Data collection example:

[0851] The device scans an office room. It uses a laser sensor to capture the shape of the walls, floor, and ceiling, and a camera to capture the position and shape of each piece of furniture. It uses a temperature sensor to measure the temperature at each point in the room, and sends all the data to a server.

[0852] 2. Data processing examples:

[0853] The server analyzes the received data and generates a 3D model of the office, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[0854] 3. Examples of on-site investigations and troubleshooting:

[0855] Users put on VR goggles to access a virtual office and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature sensor data. They receive audio guidance through the audio glasses, identify the cause, and share steps to take with other teams.

[0856] In this way, users can remotely capture and manipulate detailed data about physical spaces, enabling highly efficient and accurate on-site inspections and troubleshooting.

[0857] The processing flow will be explained below.

[0858] Step 1:

[0859] The device initializes the sensors: it activates the laser sensor, camera, and temperature sensor and checks that they are working properly.

[0860] Step 2:

[0861] The device begins scanning the physical space, using laser sensors to capture 3D positional data of the entire room and cameras to capture color images, while temperature sensors gather environmental data.

[0862] Step 3:

[0863] The terminal compresses and packages the collected data in real time, where it is formatted and ready to be sent to the server.

[0864] Step 4:

[0865] The device sends the compressed data packets over the network to the server, which monitors the communication to ensure the data transfer is successful.

[0866] Step 5:

[0867] The server receives data sent from the device, such as 3D position data, color images, and temperature data, and stores them in a waiting state for processing.

[0868] Step 6:

[0869] The server analyzes the received data and generates a 3D point cloud, building a complete 3D model of the physical space based on the laser scan data.

[0870] Step 7:

[0871] The server complements missing parts of the generated 3D model by working with generative artificial intelligence (AI) and a database. Missing object and texture information is generated by AI and retrieved from the database.

[0872] Step 8:

[0873] The server optimizes the interpolated 3D model for the VR system, reducing the data size and converting it into a format compatible with VR goggles.

[0874] Step 9:

[0875] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins.

[0876] Step 10:

[0877] The user uses the glove to manipulate objects in the VR space, and receives real-time audio guidance and explanations through the audio glasses to operate within the system.

[0878] Step 11:

[0879] Users can conduct real-time on-site investigations and troubleshoot problems within the VR space. If an abnormality is discovered, they can check the details and plan the necessary countermeasures.

[0880] Step 12:

[0881] The server reflects user operation information in real time and updates the data for the entire system. It also saves necessary data based on user instructions and generates reports to manage work history.

[0882] As described above, the system of the present invention includes a series of processes from collecting data on the physical space to generating and operating the VR space, enabling efficient and accurate on-site investigations and troubleshooting.

[0883] Example 1

[0884] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0885] There is a demand for technology that can quickly and accurately collect data from physical spaces and efficiently reproduce and manipulate it in virtual reality spaces. In particular, advanced processing power and collaboration are required to complement missing data and optimize 3D models, but current systems are unable to meet these requirements. Therefore, the challenge is to generate highly accurate 3D models and provide an environment in which users can intuitively operate them.

[0886] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0887] In this invention, the server includes a means for receiving collected data, a means for analyzing the received data and generating a 3D point cloud, a means for generating a complete 3D model by utilizing generative artificial intelligence and a database to supplement missing data, and a means for providing the optimized 3D model to the user's virtual reality space, thereby enabling detailed data of a physical space to be collected quickly and accurately and efficiently reproduced and manipulated in a virtual space.

[0888] "Physical space" refers to the space in which a user acts in the real world, an environment with a concrete shape and layout.

[0889] "Data collection means" refers to a combination of hardware and software for collecting shape data, image data, environmental data, etc. of a physical space.

[0890] A "server" is a high-performance computer system that receives, analyzes, and processes data, and has the ability to link with generative artificial intelligence and databases.

[0891] A "laser sensor" is a sensor that uses laser light to measure the distance and shape of an object.

[0892] A "camera" is an optical device for capturing image data of physical space.

[0893] A "temperature sensor" is a sensor for measuring the ambient temperature of a physical space.

[0894] A "3D point cloud" is a three-dimensional data set composed of point cloud data acquired from laser sensors and other measurement devices.

[0895] "Generative AI" is AI that has the ability to use existing data to fill in missing data or generate new data.

[0896] A "database" is a system that can efficiently store and search large amounts of data.

[0897] "User" refers to a person or organization that uses the system to manipulate data in physical space.

[0898] "Virtual reality space" refers to a virtual three-dimensional space generated by computer technology that provides an environment in which users can engage in activities within that space.

[0899] "VR goggles" are devices that allow users to visually experience virtual reality spaces.

[0900] A "glove" is an input device that a user uses to manipulate objects in a virtual reality space.

[0901] An "audio device" is an audio-related device that allows a user to use voice guidance and commands.

[0902] This invention is a system that collects detailed data of a physical space, analyzes it to generate a virtual reality space (hereinafter referred to as VR space), and allows users to operate the VR space in real time.

[0903] Data collection terminal

[0904] The device is equipped with a laser sensor, camera, and temperature sensor to collect geometric data, image data, and environmental data of the physical space. Specifically, the laser sensor collects detailed geometric data of the space, and the camera takes high-resolution image data. The temperature sensor measures the environmental temperature at multiple points, compresses and packages all the data, and sends it to the server in real time.

[0905] Server data analysis

[0906] The server is equipped with a high-performance processor and storage to analyze the received data. The server first generates a 3D point cloud from the received laser scan data and integrates it with image and temperature data acquired from the camera. Missing data is then supplemented using generative artificial intelligence (generative AI models) and a database to generate a complete 3D model. The generated 3D model is then converted into a data format optimized for the VR system.

[0907] VR system operation

[0908] Users manipulate the VR space using VR goggles, gloves, and an audio device. The VR system receives optimization data sent from the server and provides the user with a highly accurate virtual space. Users can manipulate objects in the virtual space using the gloves and receive real-time audio guidance and explanations through the audio device. They can also use voice commands to obtain specific information or perform operations.

[0909] Specific examples

[0910] 1. Data collection example:

[0911] The device uses a laser sensor to scan an office room, collecting geometric data on the walls, floors, ceilings, and furniture. It also uses a camera to capture a panoramic view of the office and a temperature sensor to create a temperature map of the room. This data is then sent to a server in real time.

[0912] 2. Data processing examples:

[0913] The server analyzes the received data and uses generative AI models to fill in any incomplete data, resulting in a complete, optimized 3D model of the office.

[0914] 3. Examples of on-site investigations and troubleshooting:

[0915] Users access the virtual office by wearing VR goggles, manipulate objects with gloves, and receive real-time voice guidance about abnormal temperature rises through an audio device. They can obtain additional information through voice commands and identify the cause of the temperature anomaly.

[0916] Prompt Sentence Examples

[0917] "Check for temperature abnormalities in the VR space and explain the corrective steps using audio."

[0918] "Please explain in detail the process from collecting data from the physical space to generating the VR space."

[0919] In this way, the present invention allows detailed data of a physical space to be quickly and accurately collected and efficiently reproduced and manipulated in a virtual space.

[0920] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0921] Step 1:

[0922] The device collects data on the physical space. First, it uses a laser sensor to scan the shape of the physical space and collects layout data on walls, floors, ceilings, furniture, etc. Next, it uses a camera to capture high-resolution color images. It also uses a temperature sensor to measure the ambient temperature at multiple points. This data is then combined to create a detailed dataset of the physical space. This dataset is then compressed and sent to a server in real time.

[0923] Input: Physical space

[0924] Output: Compressed package of geometry, image data, and temperature data

[0925] Step 2:

[0926] The server receives data sent from the device. It checks the data for consistency and requests the device to resend it if it contains incomplete data. It then analyzes the received laser scan data, image data, and temperature data to generate a 3D point cloud. It then builds a detailed 3D model of the physical space based on the point cloud data.

[0927] Input: Compressed data sent from the terminal

[0928] Output: 3D point cloud data, initial 3D model

[0929] Step 3:

[0930] The server uses generative artificial intelligence (generative AI model) and a database to fill in missing or incomplete data. The generative AI model naturally fills in missing parts based on existing data. A complete 3D model is constructed based on the analyzed data.

[0931] Input: initial 3D model, generative AI model, database

[0932] Output: Full 3D model

[0933] Step 4:

[0934] The server processes the generated 3D model to optimize it for the VR system. It performs optimization processes such as simplifying the mesh, optimizing the texture, and adjusting the frame rate to ensure smooth display in the VR space. The optimized data is then sent to the user's VR system.

[0935] Input: A complete 3D model

[0936] Output: Optimized VR data

[0937] Step 5:

[0938] Users interact with the VR space using VR goggles, gloves, and an audio device. The VR system receives optimized data from the server and displays a highly accurate virtual space. Users use the gloves to manipulate objects in the virtual space and receive real-time audio guidance and explanations through the audio device. Voice commands can also be used to obtain specific information or perform operations.

[0939] Input: Optimized VR data

[0940] Output: High-precision virtual space displayed to the user, audio guidance, and operation results

[0941] (Application example 1)

[0942] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0943] In modern factories, monitoring and maintenance of complex equipment is important, but it is necessary to dispatch specialized engineers to the site, which is time-consuming and costly. Furthermore, when an abnormality occurs, a rapid response is required, but this response can be delayed if the engineer is located in a remote location. Furthermore, when an abnormality is detected, it can be difficult to obtain detailed environmental and physical information, making troubleshooting difficult. There is a need to solve these issues and achieve more efficient monitoring and maintenance of equipment within factories and a faster response to abnormalities.

[0944] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0945] In this invention, the server includes a means for collecting data on the physical space, a means for analyzing the collected data and generating a virtual reality space, a means for providing an interface for a user to operate the virtual reality space, and a means for detecting abnormalities and notifying them in real time. This makes it possible to monitor and operate detailed equipment status in a factory in real time even from a remote location, and to respond quickly even if an abnormality is detected.

[0946] "Physical space" refers to a space that exists in the real world and has a specific size and shape.

[0947] "Data collection means" refers to devices and technologies used to acquire geometric and environmental data from physical spaces.

[0948] "Analysis" refers to examining collected data in detail to understand and process its structure and characteristics.

[0949] A "virtual reality space" is a virtual three-dimensional space generated by a computer that users can experience through their senses of sight and touch.

[0950] "Means for providing an interface" refers to devices and software that allow a user to operate a virtual reality space.

[0951] "Means for detecting anomalies" refers to devices or technologies for detecting unusual behavior or values ​​from collected data.

[0952] "Means for real-time notification" refers to devices or technologies that immediately convey information to users when an abnormality is detected.

[0953] A "laser sensor" is a device that uses light to scan the shape of physical space and obtain accurate three-dimensional data.

[0954] A "camera" is a device that takes pictures or videos and stores them as digital data.

[0955] "Generative AI" is an AI technology that automatically generates missing information based on collected data.

[0956] A "database" is a system that organizes and stores large amounts of data and allows fast access when needed.

[0957] A specific embodiment of the present invention will be described below. This system collects and analyzes data from a physical space to generate a virtual reality (VR) space, allowing users to manipulate the VR space.

[0958] System Configuration

[0959] 1. Data collection terminal

[0960] The device is equipped with a laser sensor and a camera to collect detailed geometric and image data of the physical space, and also collects environmental data using a temperature sensor, allowing the shape, layout, and environmental data of the physical space to be obtained in real time.

[0961] 2. Server

[0962] The server has a high-performance processor and storage for receiving and analyzing data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space. It also has the ability to detect abnormalities and send notifications in real time.

[0963] 3. VR System

[0964] The VR system used by users consists of VR goggles, gloves, and an audio device. This allows users to perform visual and tactile manipulation within the VR space. Users can also receive real-time audio guidance and explanations through the audio device.

[0965] Program processing

[0966] Data collection and transmission

[0967] The device uses a laser sensor to scan the physical space and generate a 3D point cloud, while a camera captures color images for object identification. A temperature sensor also collects environmental data, which is then compressed, packaged, and sent to a server in real time.

[0968] Data processing and VR space generation

[0969] The server analyzes the received data, generates a 3D point cloud from the laser scan data, and fills in any missing data using generative AI and a database. This generates a complete 3D model, which is then converted into a data format optimized for VR systems.

[0970] Interface with VR space

[0971] The user wears VR goggles and uses gloves to navigate within the VR space. The VR system receives optimized data sent from the server and displays it to the user. The user receives real-time audio guidance and explanations through the audio device, and uses voice commands to display and operate information.

[0972] Specific examples

[0973] 1. Data collection example:

[0974] The terminal scans a section of the factory, captures the shape of the equipment with a laser sensor, photographs the layout of each piece of equipment with a camera, measures the temperature of each piece of equipment with a temperature sensor, and sends all the data to a server.

[0975] 2. Data processing examples:

[0976] The server analyzes the received data and generates a 3D model of the factory, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[0977] 3. Examples of anomaly detection and response:

[0978] Users put on VR goggles to access a virtual factory and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature data. They receive audio guidance through an audio device, identify the cause, and share the steps to take to address the issue with other team members.

[0979] Examples of prompt statements

[0980] "Generate a 3D point cloud from the factory's lidar and camera data and fill in any gaps. Model all the equipment in the factory in detail and convert it into a data format optimized for VR systems and output it."

[0981] In this way, users can remotely acquire and manipulate detailed data about the physical space, enabling highly efficient and accurate monitoring of factory equipment and response to anomalies.

[0982] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0983] Step 1:

[0984] Data collection

[0985] The device collects data from the physical space. Specifically, it uses a laser sensor to obtain shape data of the object, a camera to capture color images, and a temperature sensor to obtain environmental data. This data is then sent to a server in real time.

[0986] Input: Physical space geometry, color image, temperature data

[0987] Output: Package of collected shape data, image data, and temperature data

[0988] Step 2:

[0989] Data analysis

[0990] The server receives the data sent from the device and begins analysis. It generates a 3D point cloud based on the laser scan data, identifies objects from camera images, and analyzes data from the temperature sensor to calculate the temperature distribution.

[0991] Input: Collected shape data, image data, temperature data

[0992] Output: 3D point cloud, identified object information, temperature distribution data

[0993] Step 3:

[0994] Data Completion

[0995] The server uses generative artificial intelligence (AI) to fill in any missing data based on the analyzed data. It also connects with the database to fill in any missing data. It also applies anomaly detection algorithms to detect abnormalities.

[0996] Input: 3D point cloud, identified object information, temperature distribution data

[0997] Output: Interpolated 3D point cloud, anomaly detection results

[0998] Step 4:

[0999] VR data generation

[1000] The server converts the data from the interpolated 3D point cloud into a format that can be used by the VR system, and optimizes the generated 3D model into a format suitable for the VR device.

[1001] Input: Interpolated 3D point cloud

[1002] Output: Data format optimized for VR systems

[1003] Step 5:

[1004] Providing a VR environment

[1005] The user wears VR goggles and accesses the VR space using gloves and an audio device. The user manipulates the VR space visually and tactilely, and is notified via the audio device if an abnormality is detected.

[1006] Input: Data format optimized for VR systems, anomaly detection results

[1007] Output: VR space operated by the user, abnormality notification

[1008] Step 6:

[1009] On-site simulation and response

[1010] Users can check the status of equipment in the VR space and operate any abnormalities. Specifically, they can access the abnormal area with gloves, check temperature data, and receive audio guidance through an audio device. They can then identify the cause of the abnormality and take countermeasures.

[1011] Input: VR space, audio guide, temperature data

[1012] Output: User operation results, identification of the cause of the abnormality, and countermeasures

[1013] In this way, users can efficiently monitor equipment in factories and respond to abnormalities from a remote location, eliminating the need to go to the site in person.

[1014] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1015] The present invention combines a system that collects data from a physical space, analyzes it to generate a virtual reality space (hereinafter referred to as a VR space), and provides an interface for a user to operate the VR space with an emotion engine that recognizes the user's emotions. An embodiment of the present invention will be described in detail below.

[1016] System Configuration

[1017] 1. Data collection terminal

[1018] The device is equipped with a laser sensor and a camera to collect geometric and image data of the physical space, and also collects environmental data using temperature sensors, etc.

[1019] 2. Server

[1020] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space.

[1021] The server is equipped with an emotion engine that recognizes the user's emotions by analyzing the user's facial expressions and voice data.

[1022] 3. VR System

[1023] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[1024] Program processing

[1025] 1. Physical space data collection:

[1026] The device uses a laser sensor to scan the shape of the physical space and obtain the location of objects, a camera to capture color images and perform object recognition, and a temperature sensor to collect environmental data, which is then compressed, packaged, and sent to a server in real time.

[1027] 2. Data processing and VR space generation:

[1028] The server receives the collected data and begins analysis. It generates a 3D point cloud from the laser scan data and fills in any missing data using generative AI and a database. This creates a complete 3D model. The generated 3D model is then converted into a data format optimized for VR systems.

[1029] 3. Emotion Recognition with Emotion Engine:

[1030] The emotion engine installed on the server analyzes facial expression and voice data acquired from the user to recognize the user's emotions. For example, it scans the user's face with a camera and captures the user's voice with a voice input device.

[1031] 4. VR space manipulation and emotional response:

[1032] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins. The user uses the gloves to manipulate objects in the VR space.

[1033] The server's emotion engine monitors the user's emotions in real time and dynamically adjusts the VR environment and interface based on the user's emotions. For example, if the user feels anxious, the system will change the environment to use brighter, warmer colors.

[1034] Specific examples

[1035] 1. Data collection example:

[1036] The device scans an office room. It uses a laser sensor to capture the shape of the walls, floor, and ceiling, and a camera to capture the position and shape of each piece of furniture. It uses a temperature sensor to measure the temperature at each point in the room, and sends all the data to a server.

[1037] 2. Data processing examples:

[1038] The server analyzes the received data and generates a 3D model of the office, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[1039] 3. Example of emotion recognition and VR space adjustment:

[1040] Users put on VR goggles to access a virtual office and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature sensor data. They receive audio guidance through the audio glasses, identify the cause, and share steps to take with other teams.

[1041] If the server's emotion engine detects the user's anxiety, it will adjust the lighting in the VR space and play relaxing music to reduce the user's stress.

[1042] In this way, users can remotely acquire and manipulate detailed data of physical spaces, enabling them to carry out on-site investigations and troubleshoot problems with great efficiency and accuracy. The emotion engine also enables flexible responses that adapt to the user's emotions.

[1043] The processing flow will be explained below.

[1044] Step 1:

[1045] The device initializes the sensors: it activates the laser sensor, camera, and temperature sensor and checks that they are working properly.

[1046] Step 2:

[1047] The device begins scanning the physical space, using laser sensors to capture 3D positional data of the entire room and cameras to capture color images, while temperature sensors gather environmental data.

[1048] Step 3:

[1049] The terminal compresses and packages the collected data in real time, where it is formatted and ready to be sent to the server.

[1050] Step 4:

[1051] The device sends the compressed data packets over the network to the server, which monitors the communication to ensure the data transfer is successful.

[1052] Step 5:

[1053] The server receives data sent from the device, such as 3D position data, color images, and temperature data, and stores them in a waiting state for processing.

[1054] Step 6:

[1055] The server analyzes the received data and generates a 3D point cloud, building a complete 3D model of the physical space based on the laser scan data.

[1056] Step 7:

[1057] The server complements missing parts of the generated 3D model by working with generative artificial intelligence (AI) and a database. Missing object and texture information is generated by AI and retrieved from the database.

[1058] Step 8:

[1059] The server optimizes the interpolated 3D model for the VR system, reducing the data size and converting it into a format compatible with VR goggles.

[1060] Step 9:

[1061] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins.

[1062] Step 10:

[1063] The user uses the glove to manipulate objects in the VR space, and receives real-time audio guidance and explanations through the audio glasses to operate within the system.

[1064] Step 11:

[1065] The server's emotion engine analyzes the user's facial expression and voice data in real time, thereby recognizing the user's emotions.

[1066] Step 12:

[1067] The server dynamically adjusts the environment and interface of the VR space based on the emotions recognized by the emotion engine. For example, if the user feels anxious, the system will change the environment to use brighter, warmer colors.

[1068] Step 13:

[1069] Users can conduct real-time on-site investigations and troubleshoot problems within the VR space. If an abnormality is discovered, they can check the details and plan the necessary countermeasures.

[1070] Step 14:

[1071] The server reflects user operation information in real time and updates the data for the entire system. It also saves necessary data based on user instructions and generates reports to manage work history.

[1072] As described above, the system of the present invention includes a series of processes from collecting data from the physical space to generating and operating the VR space, enabling efficient and accurate on-site investigations and troubleshooting. Furthermore, the emotion engine enables flexible responses that adapt to the user's emotions.

[1073] Example 2

[1074] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1075] Current virtual reality (VR) systems are capable of collecting and analyzing highly accurate physical space data, but lack the ability to recognize the user's emotional state and dynamically adjust the VR environment accordingly. This can lead to stressful or unpleasant experiences for users. Furthermore, when there are gaps or missing data in the process of collecting and analyzing physical space data, it is difficult to easily fill in the gaps.

[1076] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting data of a physical space, means for analyzing the collected data and generating a virtual reality space, means for providing an interface for a user to operate the virtual reality space, means for recognizing the user's emotions in real time, and means for dynamically adjusting the environment of the virtual reality space based on the user's emotions. This makes it possible to adapt to the user's emotional state and provide a comfortable, stress-free VR experience. Furthermore, by utilizing generative artificial intelligence and a database, it is possible to effectively compensate for insufficient or missing collected data.

[1077] "Physical space" refers to a three-dimensional space that exists in the real world.

[1078] "Data collection means" refers to devices and technologies that measure and acquire information about the shape and environment of physical spaces.

[1079] A "laser sensor" refers to a device that uses a light laser to scan the shape data of physical space with high precision.

[1080] "Camera" refers to a photographing device for acquiring image data.

[1081] "Virtual reality space" refers to a three-dimensional virtual reality space generated on a computer.

[1082] "Analysis tools" refers to the technology and devices that process collected data and convert it into meaningful information.

[1083] "Generative AI" refers to artificial intelligence techniques used to fill in gaps in collected data.

[1084] A "database" refers to a collection of information that is systematically organized and stored so that it can be accessed and used as needed.

[1085] "Interface" refers to the devices and software that allow a user to interact with and manipulate a system or data.

[1086] "Means for recognizing emotions in real time" refers to technology or devices that analyze a user's facial expressions and voice data to identify their current emotional state.

[1087] "Means for dynamic adjustment" refers to technology or devices that instantly change environmental settings to suit the user's state based on information obtained in real time.

[1088] "Server" refers to a high-performance computing device responsible for data processing, analysis, storage, and interfacing with other systems.

[1089] "User" refers to a person who uses this system to operate the VR space.

[1090] This invention relates to a system that generates a virtual reality (VR) space by collecting and analyzing data from a physical space, and dynamically adjusts the VR space by recognizing the user's emotions. This system is mainly composed of three main components: a terminal, a server, and a VR system used by the user.

[1091] 1. Data collection by device

[1092] The device uses a variety of sensors to collect detailed data about the physical space. The device uses the following hardware:

[1093] Laser sensor: Scans the shape of physical space with high precision.

[1094] Camera: Photographs objects and their arrangement in physical space and acquires color image data.

[1095] Temperature sensor: Collects ambient environmental data.

[1096] The data acquired by these sensors is compressed, packaged, and sent to a server in real time.

[1097] Examples:

[1098] When scanning an office room, a laser sensor scans the shape of the walls, floor, and ceiling, and a camera captures the position and shape of the furniture. A temperature sensor measures the temperature of the room, and all this data is sent to a server. An example prompt is, "Which sensor should be used to collect office shape and temperature data?"

[1099] 2. Data processing and VR space generation by the server

[1100] The server receives the data sent from the terminal and performs a series of processes. The main software and technologies used are as follows:

[1101] Generative artificial intelligence (AI): Complementing gaps in collected data.

[1102] Database: Provides necessary information to complete the data.

[1103] 3D point cloud generation system: Generates 3D point clouds based on laser scan data.

[1104] VR data optimization system: Converts the generated 3D model into a data format optimized for the VR system.

[1105] Examples:

[1106] The server receives the office data and generates a 3D point cloud. It uses generative AI to fill in any incomplete data and create a complete 3D model. It then converts this 3D model into a data format optimized for the VR system. An example prompt might be, "Please tell me the steps to convert the data received by the server into a 3D point cloud and fill in any missing data using generative AI."

[1107] 3. Emotion Recognition by Emotion Engine

[1108] The server is equipped with an emotion engine that recognizes the user's emotions in real time. This engine uses the following data:

[1109] Facial expression data: The camera scans the user's face and analyzes their facial expressions.

[1110] Voice Data: We capture and analyze your voice using your voice input device.

[1111] Examples:

[1112] While the user is wearing the VR goggles, the server's camera scans the user's face and captures their facial expression data. The user's voice is also captured by a voice input device, and the emotion engine analyzes this data in real time. An example of a prompt sentence is, "Please tell me how to analyze the user's emotions in a VR environment in real time and adjust the environment based on those emotions."

[1113] 4. User manipulation of VR space and emotional response

[1114] Users operate the VR space by wearing the following devices.

[1115] VR goggles: Displays visual information.

[1116] Gloves: Allows for tactile control.

[1117] Audio Glasses: Receive audio guides and explanations in real time.

[1118] The server monitors the user's emotional state and dynamically adjusts the VR environment based on the user's emotions.

[1119] Examples:

[1120] When a user puts on VR goggles and accesses a virtual office, they discover an abnormal temperature rise. The server's emotion engine detects the user's anxiety and adjusts the lighting in the VR space and plays relaxing music to reduce stress.

[1121] In this way, our invention collects and analyzes detailed data on the physical space, creates a VR space that is easy for the user to operate, and recognizes and adapts to the user's emotions in real time, thereby realizing a comfortable and efficient VR experience.

[1122] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1123] Step 1:

[1124] The device collects data on the physical space. Specifically, it uses a laser sensor to scan the shape of the physical space and generate 3D point cloud data. It also uses a camera to capture color images of the physical space and collect object placement data. In addition, it uses a temperature sensor to capture environmental data. This data is compressed and packaged in real time and sent to the server.

[1125] Input: Physical space information

[1126] Output: Compressed and packaged shape data, color image data, and environmental data

[1127] Step 2:

[1128] The server receives the data sent from the terminal. To analyze the received data, it first extracts the data and performs preprocessing according to the format.

[1129] Input: Compressed data sent from the terminal

[1130] Output: Extracted shape data, color image data, environmental data

[1131] Step 3:

[1132] The server generates a 3D point cloud based on the received laser scan data. Missing data is then filled in using the generation AI and database. Incomplete scan data is input, and the generation AI predicts and fills in the missing parts to output 3D model data.

[1133] Input: extracted shape data

[1134] Output: Interpolated 3D point cloud data

[1135] Step 4:

[1136] The server converts the generated 3D model into a data format optimized for the VR system, performing optimization processing to reduce data size and rendering time. The server takes the 3D point cloud as input and outputs optimized data suitable for the VR system.

[1137] Input: Interpolated 3D point cloud data

[1138] Output: Data optimized for VR systems

[1139] Step 5:

[1140] The server acquires the user's facial expression and voice data and analyzes it using an emotion engine. By analyzing facial features captured by the camera and recording and analyzing the user's voice, the server identifies the user's emotional state in real time. The acquired facial expression and voice data are used as input to output the user's emotional state.

[1141] Input: facial expression data, voice data

[1142] Output: Data on the user's emotional state

[1143] Step 6:

[1144] The user puts on VR goggles, gloves, and audio glasses to access the VR environment. Optimized data is received from the server, and the experience in the VR space begins. The user uses the gloves to manipulate objects in the VR space, and receives audio guidance and explanations through the audio glasses.

[1145] Input: Data optimized for VR systems

[1146] Output: VR experience according to user operations

[1147] Step 7:

[1148] The server's emotion engine analyzes and monitors the user's emotional state in real time. If the user feels anxious or stressed, the server dynamically adjusts the VR environment, for example by brightening the colors or playing relaxing music.

[1149] Input: User's emotional state data

[1150] Output: Calibrated VR environment

[1151] Through these processing steps, our system can efficiently collect and analyze detailed data from the physical space and generate a VR space. Furthermore, by dynamically adapting to the user's emotional state, it can provide an optimal VR experience.

[1152] (Application example 2)

[1153] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1154] Conventional virtual reality (VR) systems have been successful in collecting and reproducing data from physical spaces, but they have not been able to adequately respond to dynamic adjustments based on the user's emotions. As a result, they are unable to respond adequately to situations that cause stress or that require increased satisfaction, limiting their ability to improve the quality of the user's experience. Furthermore, in physical stores, there is a need for technology that can optimize the shopping experience by changing the store environment based on the user's emotions.

[1155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1156] In this invention, the server includes a means for collecting data on the physical space, a means for analyzing the collected data and generating a virtual reality space, a means for providing an interface for the user to operate the virtual reality space, and a means for recognizing the user's emotions and dynamically adjusting the environment of the virtual reality space. This allows the user to remotely obtain and manipulate detailed data on the physical space and further enables flexible responses that adapt to the user's emotions. Furthermore, in physical stores, lighting and music can be adjusted in real time according to the user's emotions, providing a comfortable shopping experience.

[1157] "Physical space" refers to the space that exists in the real world and the objects placed within it.

[1158] "Data collection means" refers to methods and devices that use equipment such as laser sensors and cameras to acquire information about the shape and environment of a physical space.

[1159] "Analysis means" refers to the methods and devices that process collected data and convert it into meaningful information.

[1160] "Virtual reality space" refers to a realistic three-dimensional virtual space generated using computer technology.

[1161] "Generation means" refers to the method or device that creates a virtual reality space based on collected data.

[1162] "Interface" refers to the input devices and software that allow users to interact with the virtual reality space.

[1163] "Emotion recognition means" refers to a method or device that analyzes a user's facial expressions and voice and identifies their emotions.

[1164] "Environment adjustment means" refers to a method or device that dynamically changes lighting, music, and other environmental elements in a virtual reality space based on the user's emotions.

[1165] "Server" refers to a computer system for collecting, analyzing, and storing data, and generating virtual reality spaces.

[1166] This invention combines a system that collects data from physical spaces, analyzes it to generate a virtual reality space (hereinafter referred to as VR space), and provides an interface for users to operate the VR space with an emotion engine that recognizes the user's emotions.

[1167] System Configuration

[1168] 1. Data collection terminal

[1169] The device is equipped with a laser sensor and a camera to collect shape and image data of the physical space. It also collects environmental data using temperature sensors, etc. This allows detailed data of the physical space to be obtained.

[1170] 2. Server

[1171] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space. The server also has an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions by analyzing the user's facial expressions and voice data.

[1172] 3. VR System

[1173] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[1174] Program processing description

[1175] Physical space data collection

[1176] The device uses a laser sensor to scan and capture geometric data of the physical space. It also uses a camera to identify the placement of objects and collects this data as image data. Environmental data is also collected using a temperature sensor. This data is compressed and sent to a server in real time.

[1177] Data processing and VR space generation

[1178] The server analyzes the data received from the device and generates a 3D point cloud. Missing data is filled in using generative AI and a database. This generates a complete 3D model, which is then converted into a format optimized for the VR system.

[1179] Emotion recognition by emotion engine

[1180] The server's emotion engine analyzes facial expression and voice data acquired from the user and recognizes the user's emotions in real time. For example, it can grasp emotions by scanning the user's face with a camera and capturing the user's voice with a voice input device.

[1181] Manipulating VR space and responding to emotions

[1182] The user puts on VR goggles to access the VR space. They start the experience using optimization data sent from the server and use the gloves to manipulate objects in the VR space. The server's emotion engine monitors the user's emotions in real time and dynamically adjusts the environment and interface of the VR space based on their emotions. For example, if the user is feeling stressed, the system will change the environment using brighter, warmer colors.

[1183] Specific examples

[1184] Data Collection Example

[1185] The device scans an office room, capturing the room's shape with a laser sensor, photographing the furniture layout with a camera, collecting temperature data with a temperature sensor, and sending all data to a server.

[1186] Data Processing Examples

[1187] The server analyzes the received data and generates a 3D model of the office. Imperfections are filled in by generative AI, and optimized VR data is created and exported to the VR system.

[1188] Example of emotion recognition and VR space adjustment

[1189] The user puts on VR goggles to access the virtual office and discovers an abnormal temperature rise. They manipulate objects with the gloves to check temperature sensor data and listen to audio guidance through the audio glasses. If the server's emotion engine detects the user's anxiety, the system adjusts the lighting and plays relaxing music.

[1190] Prompt Sentence Examples

[1191] "Please explain how we can use VR space and emotion engines to test the most effective placement of new in-store promotional placements."

[1192] In this way, users can remotely obtain and manipulate detailed data of physical spaces, enabling highly efficient and accurate on-site investigations and troubleshooting. The emotion engine also enables flexible responses that adapt to the user's emotions.

[1193] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1194] Step 1:

[1195] The device uses a laser sensor to scan and capture geometric data of the physical space, which is then processed into an initial 3D point cloud and combined with image data captured by the camera, which is then compressed and sent to a server in real time.

[1196] (Input: Shape of physical space, image data, Output: 3D point cloud data)

[1197] Step 2:

[1198] The server analyzes the 3D point cloud data and image data received from the device, uses generative artificial intelligence (AI) to fill in any incomplete data, and generates a complete 3D model. It then converts this 3D model into a data format optimized for the VR system.

[1199] (Input: 3D point cloud data, image data, Output: complete 3D model)

[1200] Step 3:

[1201] The user puts on the VR goggles and accesses the VR space. The server sends optimized 3D model data to the VR goggles, allowing the user to begin operating within the VR space. The user uses the gloves to manipulate objects in the VR space and receives audio guidance and explanations through the audio glasses.

[1202] (Input: VR goggles, gloves, audio glasses, Output: Visual and tactile manipulation of VR space)

[1203] Step 4:

[1204] The server's emotion engine analyzes the user's facial expressions and voice data. It recognizes emotions by scanning the user's face with a camera and capturing the user's voice with a voice input device. This allows the system to grasp the user's emotional state, such as stress level and excitement level, in real time.

[1205] (Input: facial expression data, voice data, output: user's emotional state)

[1206] Step 5:

[1207] The server dynamically adjusts the VR environment based on the user's emotional state. For example, if the user is feeling stressed, the server softens the lighting and plays relaxing music. If the user is excited, the server brightens the lighting and plays upbeat music. This provides the optimal VR experience according to the user's emotions.

[1208] (Input: user's emotional state, Output: adjusted VR environment)

[1209] Step 6:

[1210] After the user finishes their work in the VR space, the server saves the collected data for future use. It also analyzes the user's emotional data and operation logs to gather feedback for further optimization of the experience. Based on this, the server makes improvements to ensure the next VR experience is more comfortable and efficient.

[1211] (Input: Operation and emotion data during VR experience, Output: Saved data, feedback results)

[1212] In this way, the system obtains detailed data of the physical space to generate a VR space, and dynamically adjusts it according to the user's emotions to provide an optimal VR experience.

[1213] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1214] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1215] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1216] [Fourth embodiment]

[1217] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1218] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1219] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1220] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1221] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1223] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1224] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1225] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1226] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1227] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1228] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1229] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1230] The present invention is a system that collects data from a physical space, analyzes the data, generates a virtual reality space (hereinafter referred to as a VR space), and provides an interface for a user to operate the VR space. An embodiment of the present invention will be described in detail below.

[1231] System Configuration

[1232] 1. Data collection terminal

[1233] The device is equipped with a laser sensor and a camera, which allows it to collect detailed geometric and image data of the physical space, as well as environmental data using temperature sensors and other sensors.

[1234] 2. Server

[1235] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space.

[1236] 3. VR System

[1237] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[1238] Program processing

[1239] 1. Physical space data collection:

[1240] The device uses a laser sensor to scan the shape of the physical space and obtain the location of objects, a camera to capture color images and perform object recognition, and a temperature sensor to collect environmental data, which is then compressed, packaged, and sent to a server in real time.

[1241] 2. Data processing and VR space generation:

[1242] The server receives the collected data and begins analysis. It generates a 3D point cloud from the laser scan data and fills in any missing data using generative AI and a database. This creates a complete 3D model. The generated 3D model is then converted into a data format optimized for VR systems.

[1243] 3. VR space operation:

[1244] The user wears VR goggles and uses gloves to navigate within the VR space. The VR system receives optimization data sent from the server and displays it to the user. The user can receive real-time audio guidance and explanations through the audio glasses, and can display and operate information using voice commands.

[1245] Specific examples

[1246] 1. Data collection example:

[1247] The device scans an office room. It uses a laser sensor to capture the shape of the walls, floor, and ceiling, and a camera to capture the position and shape of each piece of furniture. It uses a temperature sensor to measure the temperature at each point in the room, and sends all the data to a server.

[1248] 2. Data processing examples:

[1249] The server analyzes the received data and generates a 3D model of the office, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[1250] 3. Examples of on-site investigations and troubleshooting:

[1251] Users put on VR goggles to access a virtual office and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature sensor data. They receive audio guidance through the audio glasses, identify the cause, and share steps to take with other teams.

[1252] In this way, users can remotely capture and manipulate detailed data about physical spaces, enabling highly efficient and accurate on-site inspections and troubleshooting.

[1253] The processing flow will be explained below.

[1254] Step 1:

[1255] The device initializes the sensors: it activates the laser sensor, camera, and temperature sensor and checks that they are working properly.

[1256] Step 2:

[1257] The device begins scanning the physical space, using laser sensors to capture 3D positional data of the entire room and cameras to capture color images, while temperature sensors gather environmental data.

[1258] Step 3:

[1259] The terminal compresses and packages the collected data in real time, where it is formatted and ready to be sent to the server.

[1260] Step 4:

[1261] The device sends the compressed data packets over the network to the server, which monitors the communication to ensure the data transfer is successful.

[1262] Step 5:

[1263] The server receives data sent from the device, such as 3D position data, color images, and temperature data, and stores them in a waiting state for processing.

[1264] Step 6:

[1265] The server analyzes the received data and generates a 3D point cloud, building a complete 3D model of the physical space based on the laser scan data.

[1266] Step 7:

[1267] The server complements missing parts of the generated 3D model by working with generative artificial intelligence (AI) and a database. Missing object and texture information is generated by AI and retrieved from the database.

[1268] Step 8:

[1269] The server optimizes the interpolated 3D model for the VR system, reducing the data size and converting it into a format compatible with VR goggles.

[1270] Step 9:

[1271] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins.

[1272] Step 10:

[1273] The user uses the glove to manipulate objects in the VR space, and receives real-time audio guidance and explanations through the audio glasses to operate within the system.

[1274] Step 11:

[1275] Users can conduct real-time on-site investigations and troubleshoot problems within the VR space. If an abnormality is discovered, they can check the details and plan the necessary countermeasures.

[1276] Step 12:

[1277] The server reflects user operation information in real time and updates the data for the entire system. It also saves necessary data based on user instructions and generates reports to manage work history.

[1278] As described above, the system of the present invention includes a series of processes from collecting data on the physical space to generating and operating the VR space, enabling efficient and accurate on-site investigations and troubleshooting.

[1279] Example 1

[1280] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1281] There is a demand for technology that can quickly and accurately collect data from physical spaces and efficiently reproduce and manipulate it in virtual reality spaces. In particular, advanced processing power and collaboration are required to complement missing data and optimize 3D models, but current systems are unable to meet these requirements. Therefore, the challenge is to generate highly accurate 3D models and provide an environment in which users can intuitively operate them.

[1282] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1283] In this invention, the server includes a means for receiving collected data, a means for analyzing the received data and generating a 3D point cloud, a means for generating a complete 3D model by utilizing generative artificial intelligence and a database to supplement missing data, and a means for providing the optimized 3D model to the user's virtual reality space, thereby enabling detailed data of a physical space to be collected quickly and accurately and efficiently reproduced and manipulated in a virtual space.

[1284] "Physical space" refers to the space in which a user acts in the real world, an environment with a concrete shape and layout.

[1285] "Data collection means" refers to a combination of hardware and software for collecting shape data, image data, environmental data, etc. of a physical space.

[1286] A "server" is a high-performance computer system that receives, analyzes, and processes data, and has the ability to link with generative artificial intelligence and databases.

[1287] A "laser sensor" is a sensor that uses laser light to measure the distance and shape of an object.

[1288] A "camera" is an optical device for capturing image data of physical space.

[1289] A "temperature sensor" is a sensor for measuring the ambient temperature of a physical space.

[1290] A "3D point cloud" is a three-dimensional data set composed of point cloud data acquired from laser sensors and other measurement devices.

[1291] "Generative AI" is AI that has the ability to use existing data to fill in missing data or generate new data.

[1292] A "database" is a system that can efficiently store and search large amounts of data.

[1293] "User" refers to a person or organization that uses the system to manipulate data in physical space.

[1294] "Virtual reality space" refers to a virtual three-dimensional space generated by computer technology that provides an environment in which users can engage in activities within that space.

[1295] "VR goggles" are devices that allow users to visually experience virtual reality spaces.

[1296] A "glove" is an input device that a user uses to manipulate objects in a virtual reality space.

[1297] An "audio device" is an audio-related device that allows a user to use voice guidance and commands.

[1298] This invention is a system that collects detailed data of a physical space, analyzes it to generate a virtual reality space (hereinafter referred to as VR space), and allows users to operate the VR space in real time.

[1299] Data collection terminal

[1300] The device is equipped with a laser sensor, camera, and temperature sensor to collect geometric data, image data, and environmental data of the physical space. Specifically, the laser sensor collects detailed geometric data of the space, and the camera takes high-resolution image data. The temperature sensor measures the environmental temperature at multiple points, compresses and packages all the data, and sends it to the server in real time.

[1301] Server data analysis

[1302] The server is equipped with a high-performance processor and storage to analyze the received data. The server first generates a 3D point cloud from the received laser scan data and integrates it with image and temperature data acquired from the camera. Missing data is then supplemented using generative artificial intelligence (generative AI models) and a database to generate a complete 3D model. The generated 3D model is then converted into a data format optimized for the VR system.

[1303] VR system operation

[1304] Users manipulate the VR space using VR goggles, gloves, and an audio device. The VR system receives optimization data sent from the server and provides the user with a highly accurate virtual space. Users can manipulate objects in the virtual space using the gloves and receive real-time audio guidance and explanations through the audio device. They can also use voice commands to obtain specific information or perform operations.

[1305] Specific examples

[1306] 1. Data collection example:

[1307] The device uses a laser sensor to scan an office room, collecting geometric data on the walls, floors, ceilings, and furniture. It also uses a camera to capture a panoramic view of the office and a temperature sensor to create a temperature map of the room. This data is then sent to a server in real time.

[1308] 2. Data processing examples:

[1309] The server analyzes the received data and uses generative AI models to fill in any incomplete data, resulting in a complete, optimized 3D model of the office.

[1310] 3. Examples of on-site investigations and troubleshooting:

[1311] Users access the virtual office by wearing VR goggles, manipulate objects with gloves, and receive real-time voice guidance about abnormal temperature rises through an audio device. They can obtain additional information through voice commands and identify the cause of the temperature anomaly.

[1312] Prompt Sentence Examples

[1313] "Check for temperature abnormalities in the VR space and explain the corrective steps using audio."

[1314] "Please explain in detail the process from collecting data from the physical space to generating the VR space."

[1315] In this way, the present invention allows detailed data of a physical space to be quickly and accurately collected and efficiently reproduced and manipulated in a virtual space.

[1316] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1317] Step 1:

[1318] The device collects data on the physical space. First, it uses a laser sensor to scan the shape of the physical space and collects layout data on walls, floors, ceilings, furniture, etc. Next, it uses a camera to capture high-resolution color images. It also uses a temperature sensor to measure the ambient temperature at multiple points. This data is then combined to create a detailed dataset of the physical space. This dataset is then compressed and sent to a server in real time.

[1319] Input: Physical space

[1320] Output: Compressed package of geometry, image data, and temperature data

[1321] Step 2:

[1322] The server receives data sent from the device. It checks the data for consistency and requests the device to resend it if it contains incomplete data. It then analyzes the received laser scan data, image data, and temperature data to generate a 3D point cloud. It then builds a detailed 3D model of the physical space based on the point cloud data.

[1323] Input: Compressed data sent from the terminal

[1324] Output: 3D point cloud data, initial 3D model

[1325] Step 3:

[1326] The server uses generative artificial intelligence (generative AI model) and a database to fill in missing or incomplete data. The generative AI model naturally fills in missing parts based on existing data. A complete 3D model is constructed based on the analyzed data.

[1327] Input: initial 3D model, generative AI model, database

[1328] Output: Full 3D model

[1329] Step 4:

[1330] The server processes the generated 3D model to optimize it for the VR system. It performs optimization processes such as simplifying the mesh, optimizing the texture, and adjusting the frame rate to ensure smooth display in the VR space. The optimized data is then sent to the user's VR system.

[1331] Input: A complete 3D model

[1332] Output: Optimized VR data

[1333] Step 5:

[1334] Users interact with the VR space using VR goggles, gloves, and an audio device. The VR system receives optimized data from the server and displays a highly accurate virtual space. Users use the gloves to manipulate objects in the virtual space and receive real-time audio guidance and explanations through the audio device. Voice commands can also be used to obtain specific information or perform operations.

[1335] Input: Optimized VR data

[1336] Output: High-precision virtual space displayed to the user, audio guidance, and operation results

[1337] (Application example 1)

[1338] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1339] In modern factories, monitoring and maintenance of complex equipment is important, but it is necessary to dispatch specialized engineers to the site, which is time-consuming and costly. Furthermore, when an abnormality occurs, a rapid response is required, but this response can be delayed if the engineer is located in a remote location. Furthermore, when an abnormality is detected, it can be difficult to obtain detailed environmental and physical information, making troubleshooting difficult. There is a need to solve these issues and achieve more efficient monitoring and maintenance of equipment within factories and a faster response to abnormalities.

[1340] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1341] In this invention, the server includes a means for collecting data on the physical space, a means for analyzing the collected data and generating a virtual reality space, a means for providing an interface for a user to operate the virtual reality space, and a means for detecting abnormalities and notifying them in real time. This makes it possible to monitor and operate detailed equipment status in a factory in real time even from a remote location, and to respond quickly even if an abnormality is detected.

[1342] "Physical space" refers to a space that exists in the real world and has a specific size and shape.

[1343] "Data collection means" refers to devices and technologies used to acquire geometric and environmental data from physical spaces.

[1344] "Analysis" refers to examining collected data in detail to understand and process its structure and characteristics.

[1345] A "virtual reality space" is a virtual three-dimensional space generated by a computer that users can experience through their senses of sight and touch.

[1346] "Means for providing an interface" refers to devices and software that allow a user to operate a virtual reality space.

[1347] "Means for detecting anomalies" refers to devices or technologies for detecting unusual behavior or values ​​from collected data.

[1348] "Means for real-time notification" refers to devices or technologies that immediately convey information to users when an abnormality is detected.

[1349] A "laser sensor" is a device that uses light to scan the shape of physical space and obtain accurate three-dimensional data.

[1350] A "camera" is a device that takes pictures or videos and stores them as digital data.

[1351] "Generative AI" is an AI technology that automatically generates missing information based on collected data.

[1352] A "database" is a system that organizes and stores large amounts of data and allows fast access when needed.

[1353] A specific embodiment of the present invention will be described below. This system collects and analyzes data from a physical space to generate a virtual reality (VR) space, allowing users to manipulate the VR space.

[1354] System Configuration

[1355] 1. Data collection terminal

[1356] The device is equipped with a laser sensor and a camera to collect detailed geometric and image data of the physical space, and also collects environmental data using a temperature sensor, allowing the shape, layout, and environmental data of the physical space to be obtained in real time.

[1357] 2. Server

[1358] The server has a high-performance processor and storage for receiving and analyzing data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space. It also has the ability to detect abnormalities and send notifications in real time.

[1359] 3. VR System

[1360] The VR system used by users consists of VR goggles, gloves, and an audio device. This allows users to perform visual and tactile manipulation within the VR space. Users can also receive real-time audio guidance and explanations through the audio device.

[1361] Program processing

[1362] Data collection and transmission

[1363] The device uses a laser sensor to scan the physical space and generate a 3D point cloud, while a camera captures color images for object identification. A temperature sensor also collects environmental data, which is then compressed, packaged, and sent to a server in real time.

[1364] Data processing and VR space generation

[1365] The server analyzes the received data, generates a 3D point cloud from the laser scan data, and fills in any missing data using generative AI and a database. This generates a complete 3D model, which is then converted into a data format optimized for VR systems.

[1366] Interface with VR space

[1367] The user wears VR goggles and uses gloves to navigate within the VR space. The VR system receives optimized data sent from the server and displays it to the user. The user receives real-time audio guidance and explanations through the audio device, and uses voice commands to display and operate information.

[1368] Specific examples

[1369] 1. Data collection example:

[1370] The terminal scans a section of the factory, captures the shape of the equipment with a laser sensor, photographs the layout of each piece of equipment with a camera, measures the temperature of each piece of equipment with a temperature sensor, and sends all the data to a server.

[1371] 2. Data processing examples:

[1372] The server analyzes the received data and generates a 3D model of the factory, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[1373] 3. Examples of anomaly detection and response:

[1374] Users put on VR goggles to access a virtual factory and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature data. They receive audio guidance through an audio device, identify the cause, and share the steps to take to address the issue with other team members.

[1375] Examples of prompt statements

[1376] "Generate a 3D point cloud from the factory's lidar and camera data and fill in any gaps. Model all the equipment in the factory in detail and convert it into a data format optimized for VR systems and output it."

[1377] In this way, users can remotely acquire and manipulate detailed data about the physical space, enabling highly efficient and accurate monitoring of factory equipment and response to anomalies.

[1378] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1379] Step 1:

[1380] Data collection

[1381] The device collects data from the physical space. Specifically, it uses a laser sensor to obtain shape data of the object, a camera to capture color images, and a temperature sensor to obtain environmental data. This data is then sent to a server in real time.

[1382] Input: Physical space geometry, color image, temperature data

[1383] Output: Package of collected shape data, image data, and temperature data

[1384] Step 2:

[1385] Data analysis

[1386] The server receives the data sent from the device and begins analysis. It generates a 3D point cloud based on the laser scan data, identifies objects from camera images, and analyzes data from the temperature sensor to calculate the temperature distribution.

[1387] Input: Collected shape data, image data, temperature data

[1388] Output: 3D point cloud, identified object information, temperature distribution data

[1389] Step 3:

[1390] Data Completion

[1391] The server uses generative artificial intelligence (AI) to fill in any missing data based on the analyzed data. It also connects with the database to fill in any missing data. It also applies anomaly detection algorithms to detect abnormalities.

[1392] Input: 3D point cloud, identified object information, temperature distribution data

[1393] Output: Interpolated 3D point cloud, anomaly detection results

[1394] Step 4:

[1395] VR data generation

[1396] The server converts the data from the interpolated 3D point cloud into a format that can be used by the VR system, and optimizes the generated 3D model into a format suitable for the VR device.

[1397] Input: Interpolated 3D point cloud

[1398] Output: Data format optimized for VR systems

[1399] Step 5:

[1400] Providing a VR environment

[1401] The user wears VR goggles and accesses the VR space using gloves and an audio device. The user manipulates the VR space visually and tactilely, and is notified via the audio device if an abnormality is detected.

[1402] Input: Data format optimized for VR systems, anomaly detection results

[1403] Output: VR space operated by the user, abnormality notification

[1404] Step 6:

[1405] On-site simulation and response

[1406] Users can check the status of equipment in the VR space and operate any abnormalities. Specifically, they can access the abnormal area with gloves, check temperature data, and receive audio guidance through an audio device. They can then identify the cause of the abnormality and take countermeasures.

[1407] Input: VR space, audio guide, temperature data

[1408] Output: User operation results, identification of the cause of the abnormality, and countermeasures

[1409] In this way, users can efficiently monitor equipment in factories and respond to abnormalities from a remote location, eliminating the need to go to the site in person.

[1410] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1411] The present invention combines a system that collects data from a physical space, analyzes it to generate a virtual reality space (hereinafter referred to as a VR space), and provides an interface for a user to operate the VR space with an emotion engine that recognizes the user's emotions. An embodiment of the present invention will be described in detail below.

[1412] System Configuration

[1413] 1. Data collection terminal

[1414] The device is equipped with a laser sensor and a camera to collect geometric and image data of the physical space, and also collects environmental data using temperature sensors, etc.

[1415] 2. Server

[1416] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space.

[1417] The server is equipped with an emotion engine that recognizes the user's emotions by analyzing the user's facial expressions and voice data.

[1418] 3. VR System

[1419] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[1420] Program processing

[1421] 1. Physical space data collection:

[1422] The device uses a laser sensor to scan the shape of the physical space and obtain the location of objects, a camera to capture color images and perform object recognition, and a temperature sensor to collect environmental data, which is then compressed, packaged, and sent to a server in real time.

[1423] 2. Data processing and VR space generation:

[1424] The server receives the collected data and begins analysis. It generates a 3D point cloud from the laser scan data and fills in any missing data using generative AI and a database. This creates a complete 3D model. The generated 3D model is then converted into a data format optimized for VR systems.

[1425] 3. Emotion Recognition with Emotion Engine:

[1426] The emotion engine installed on the server analyzes facial expression and voice data acquired from the user to recognize the user's emotions. For example, it scans the user's face with a camera and captures the user's voice with a voice input device.

[1427] 4. VR space manipulation and emotional response:

[1428] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins. The user uses the gloves to manipulate objects in the VR space.

[1429] The server's emotion engine monitors the user's emotions in real time and dynamically adjusts the VR environment and interface based on the user's emotions. For example, if the user feels anxious, the system will change the environment to use brighter, warmer colors.

[1430] Specific examples

[1431] 1. Data collection example:

[1432] The device scans an office room. It uses a laser sensor to capture the shape of the walls, floor, and ceiling, and a camera to capture the position and shape of each piece of furniture. It uses a temperature sensor to measure the temperature at each point in the room, and sends all the data to a server.

[1433] 2. Data processing examples:

[1434] The server analyzes the received data and generates a 3D model of the office, complementing imperfections with generative AI, creating optimized VR data, and exporting it to the VR system.

[1435] 3. Example of emotion recognition and VR space adjustment:

[1436] Users put on VR goggles to access a virtual office and discover an abnormal temperature rise. They manipulate objects with the glove and check the temperature sensor data. They receive audio guidance through the audio glasses, identify the cause, and share steps to take with other teams.

[1437] If the server's emotion engine detects the user's anxiety, it will adjust the lighting in the VR space and play relaxing music to reduce the user's stress.

[1438] In this way, users can remotely acquire and manipulate detailed data of physical spaces, enabling them to carry out on-site investigations and troubleshoot problems with great efficiency and accuracy. The emotion engine also enables flexible responses that adapt to the user's emotions.

[1439] The processing flow will be explained below.

[1440] Step 1:

[1441] The device initializes the sensors: it activates the laser sensor, camera, and temperature sensor and checks that they are working properly.

[1442] Step 2:

[1443] The device begins scanning the physical space, using laser sensors to capture 3D positional data of the entire room and cameras to capture color images, while temperature sensors gather environmental data.

[1444] Step 3:

[1445] The terminal compresses and packages the collected data in real time, where it is formatted and ready to be sent to the server.

[1446] Step 4:

[1447] The device sends the compressed data packets over the network to the server, which monitors the communication to ensure the data transfer is successful.

[1448] Step 5:

[1449] The server receives data sent from the device, such as 3D position data, color images, and temperature data, and stores them in a waiting state for processing.

[1450] Step 6:

[1451] The server analyzes the received data and generates a 3D point cloud, building a complete 3D model of the physical space based on the laser scan data.

[1452] Step 7:

[1453] The server complements missing parts of the generated 3D model by working with generative artificial intelligence (AI) and a database. Missing object and texture information is generated by AI and retrieved from the database.

[1454] Step 8:

[1455] The server optimizes the interpolated 3D model for the VR system, reducing the data size and converting it into a format compatible with VR goggles.

[1456] Step 9:

[1457] The user puts on the VR goggles and accesses the VR space. The optimization data sent from the server is received and the experience begins.

[1458] Step 10:

[1459] The user uses the glove to manipulate objects in the VR space, and receives real-time audio guidance and explanations through the audio glasses to operate within the system.

[1460] Step 11:

[1461] The server's emotion engine analyzes the user's facial expression and voice data in real time, thereby recognizing the user's emotions.

[1462] Step 12:

[1463] The server dynamically adjusts the environment and interface of the VR space based on the emotions recognized by the emotion engine. For example, if the user feels anxious, the system will change the environment to use brighter, warmer colors.

[1464] Step 13:

[1465] Users can conduct real-time on-site investigations and troubleshoot problems within the VR space. If an abnormality is discovered, they can check the details and plan the necessary countermeasures.

[1466] Step 14:

[1467] The server reflects user operation information in real time and updates the data for the entire system. It also saves necessary data based on user instructions and generates reports to manage work history.

[1468] As described above, the system of the present invention includes a series of processes from collecting data from the physical space to generating and operating the VR space, enabling efficient and accurate on-site investigations and troubleshooting. Furthermore, the emotion engine enables flexible responses that adapt to the user's emotions.

[1469] Example 2

[1470] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1471] Current virtual reality (VR) systems are capable of collecting and analyzing highly accurate physical space data, but lack the ability to recognize the user's emotional state and dynamically adjust the VR environment accordingly. This can lead to stressful or unpleasant experiences for users. Furthermore, when there are gaps or missing data in the process of collecting and analyzing physical space data, it is difficult to easily fill in the gaps.

[1472] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting data of a physical space, means for analyzing the collected data and generating a virtual reality space, means for providing an interface for a user to operate the virtual reality space, means for recognizing the user's emotions in real time, and means for dynamically adjusting the environment of the virtual reality space based on the user's emotions. This makes it possible to adapt to the user's emotional state and provide a comfortable, stress-free VR experience. Furthermore, by utilizing generative artificial intelligence and a database, it is possible to effectively compensate for insufficient or missing collected data.

[1473] "Physical space" refers to a three-dimensional space that exists in the real world.

[1474] "Data collection means" refers to devices and technologies that measure and acquire information about the shape and environment of physical spaces.

[1475] A "laser sensor" refers to a device that uses a light laser to scan the shape data of physical space with high precision.

[1476] "Camera" refers to a photographing device for acquiring image data.

[1477] "Virtual reality space" refers to a three-dimensional virtual reality space generated on a computer.

[1478] "Analysis tools" refers to the technology and devices that process collected data and convert it into meaningful information.

[1479] "Generative AI" refers to artificial intelligence techniques used to fill in gaps in collected data.

[1480] A "database" refers to a collection of information that is systematically organized and stored so that it can be accessed and used as needed.

[1481] "Interface" refers to the devices and software that allow a user to interact with and manipulate a system or data.

[1482] "Means for recognizing emotions in real time" refers to technology or devices that analyze a user's facial expressions and voice data to identify their current emotional state.

[1483] "Means for dynamic adjustment" refers to technology or devices that instantly change environmental settings to suit the user's state based on information obtained in real time.

[1484] "Server" refers to a high-performance computing device responsible for data processing, analysis, storage, and interfacing with other systems.

[1485] "User" refers to a person who uses this system to operate the VR space.

[1486] This invention relates to a system that generates a virtual reality (VR) space by collecting and analyzing data from a physical space, and dynamically adjusts the VR space by recognizing the user's emotions. This system is mainly composed of three main components: a terminal, a server, and a VR system used by the user.

[1487] 1. Data collection by device

[1488] The device uses a variety of sensors to collect detailed data about the physical space. The device uses the following hardware:

[1489] Laser sensor: Scans the shape of physical space with high precision.

[1490] Camera: Photographs objects and their arrangement in physical space and acquires color image data.

[1491] Temperature sensor: Collects ambient environmental data.

[1492] The data acquired by these sensors is compressed, packaged, and sent to a server in real time.

[1493] Examples:

[1494] When scanning an office room, a laser sensor scans the shape of the walls, floor, and ceiling, and a camera captures the position and shape of the furniture. A temperature sensor measures the temperature of the room, and all this data is sent to a server. An example prompt is, "Which sensor should be used to collect office shape and temperature data?"

[1495] 2. Data processing and VR space generation by the server

[1496] The server receives the data sent from the terminal and performs a series of processes. The main software and technologies used are as follows:

[1497] Generative artificial intelligence (AI): Complementing gaps in collected data.

[1498] Database: Provides necessary information to complete the data.

[1499] 3D point cloud generation system: Generates 3D point clouds based on laser scan data.

[1500] VR data optimization system: Converts the generated 3D model into a data format optimized for the VR system.

[1501] Examples:

[1502] The server receives the office data and generates a 3D point cloud. It uses generative AI to fill in any incomplete data and create a complete 3D model. It then converts this 3D model into a data format optimized for the VR system. An example prompt might be, "Please tell me the steps to convert the data received by the server into a 3D point cloud and fill in any missing data using generative AI."

[1503] 3. Emotion Recognition by Emotion Engine

[1504] The server is equipped with an emotion engine that recognizes the user's emotions in real time. This engine uses the following data:

[1505] Facial expression data: The camera scans the user's face and analyzes their facial expressions.

[1506] Voice Data: We capture and analyze your voice using your voice input device.

[1507] Examples:

[1508] While the user is wearing the VR goggles, the server's camera scans the user's face and captures their facial expression data. The user's voice is also captured by a voice input device, and the emotion engine analyzes this data in real time. An example of a prompt sentence is, "Please tell me how to analyze the user's emotions in a VR environment in real time and adjust the environment based on those emotions."

[1509] 4. User manipulation of VR space and emotional response

[1510] Users operate the VR space by wearing the following devices.

[1511] VR goggles: Displays visual information.

[1512] Gloves: Allows for tactile control.

[1513] Audio Glasses: Receive audio guides and explanations in real time.

[1514] The server monitors the user's emotional state and dynamically adjusts the VR environment based on the user's emotions.

[1515] Examples:

[1516] When a user puts on VR goggles and accesses a virtual office, they discover an abnormal temperature rise. The server's emotion engine detects the user's anxiety and adjusts the lighting in the VR space and plays relaxing music to reduce stress.

[1517] In this way, our invention collects and analyzes detailed data on the physical space, creates a VR space that is easy for the user to operate, and recognizes and adapts to the user's emotions in real time, thereby realizing a comfortable and efficient VR experience.

[1518] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1519] Step 1:

[1520] The device collects data on the physical space. Specifically, it uses a laser sensor to scan the shape of the physical space and generate 3D point cloud data. It also uses a camera to capture color images of the physical space and collect object placement data. In addition, it uses a temperature sensor to capture environmental data. This data is compressed and packaged in real time and sent to the server.

[1521] Input: Physical space information

[1522] Output: Compressed and packaged shape data, color image data, and environmental data

[1523] Step 2:

[1524] The server receives the data sent from the terminal. To analyze the received data, it first extracts the data and performs preprocessing according to the format.

[1525] Input: Compressed data sent from the terminal

[1526] Output: Extracted shape data, color image data, environmental data

[1527] Step 3:

[1528] The server generates a 3D point cloud based on the received laser scan data. Missing data is then filled in using the generation AI and database. Incomplete scan data is input, and the generation AI predicts and fills in the missing parts to output 3D model data.

[1529] Input: extracted shape data

[1530] Output: Interpolated 3D point cloud data

[1531] Step 4:

[1532] The server converts the generated 3D model into a data format optimized for the VR system, performing optimization processing to reduce data size and rendering time. The server takes the 3D point cloud as input and outputs optimized data suitable for the VR system.

[1533] Input: Interpolated 3D point cloud data

[1534] Output: Data optimized for VR systems

[1535] Step 5:

[1536] The server acquires the user's facial expression and voice data and analyzes it using an emotion engine. By analyzing facial features captured by the camera and recording and analyzing the user's voice, the server identifies the user's emotional state in real time. The acquired facial expression and voice data are used as input to output the user's emotional state.

[1537] Input: facial expression data, voice data

[1538] Output: Data on the user's emotional state

[1539] Step 6:

[1540] The user puts on VR goggles, gloves, and audio glasses to access the VR environment. Optimized data is received from the server, and the experience in the VR space begins. The user uses the gloves to manipulate objects in the VR space, and receives audio guidance and explanations through the audio glasses.

[1541] Input: Data optimized for VR systems

[1542] Output: VR experience according to user operations

[1543] Step 7:

[1544] The server's emotion engine analyzes and monitors the user's emotional state in real time. If the user feels anxious or stressed, the server dynamically adjusts the VR environment, for example by brightening the colors or playing relaxing music.

[1545] Input: User's emotional state data

[1546] Output: Calibrated VR environment

[1547] Through these processing steps, our system can efficiently collect and analyze detailed data from the physical space and generate a VR space. Furthermore, by dynamically adapting to the user's emotional state, it can provide an optimal VR experience.

[1548] (Application example 2)

[1549] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1550] Conventional virtual reality (VR) systems have been successful in collecting and reproducing data from physical spaces, but they have not been able to adequately respond to dynamic adjustments based on the user's emotions. As a result, they are unable to respond adequately to situations that cause stress or that require increased satisfaction, limiting their ability to improve the quality of the user's experience. Furthermore, in physical stores, there is a need for technology that can optimize the shopping experience by changing the store environment based on the user's emotions.

[1551] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1552] In this invention, the server includes a means for collecting data on the physical space, a means for analyzing the collected data and generating a virtual reality space, a means for providing an interface for the user to operate the virtual reality space, and a means for recognizing the user's emotions and dynamically adjusting the environment of the virtual reality space. This allows the user to remotely obtain and manipulate detailed data on the physical space and further enables flexible responses that adapt to the user's emotions. Furthermore, in physical stores, lighting and music can be adjusted in real time according to the user's emotions, providing a comfortable shopping experience.

[1553] "Physical space" refers to the space that exists in the real world and the objects placed within it.

[1554] "Data collection means" refers to methods and devices that use equipment such as laser sensors and cameras to acquire information about the shape and environment of a physical space.

[1555] "Analysis means" refers to the methods and devices that process collected data and convert it into meaningful information.

[1556] "Virtual reality space" refers to a realistic three-dimensional virtual space generated using computer technology.

[1557] "Generation means" refers to the method or device that creates a virtual reality space based on collected data.

[1558] "Interface" refers to the input devices and software that allow users to interact with the virtual reality space.

[1559] "Emotion recognition means" refers to a method or device that analyzes a user's facial expressions and voice and identifies their emotions.

[1560] "Environment adjustment means" refers to a method or device that dynamically changes lighting, music, and other environmental elements in a virtual reality space based on the user's emotions.

[1561] "Server" refers to a computer system for collecting, analyzing, and storing data, and generating virtual reality spaces.

[1562] This invention combines a system that collects data from physical spaces, analyzes it to generate a virtual reality space (hereinafter referred to as VR space), and provides an interface for users to operate the VR space with an emotion engine that recognizes the user's emotions.

[1563] System Configuration

[1564] 1. Data collection terminal

[1565] The device is equipped with a laser sensor and a camera to collect shape and image data of the physical space. It also collects environmental data using temperature sensors, etc. This allows detailed data of the physical space to be obtained.

[1566] 2. Server

[1567] The server is equipped with a high-performance processor and storage to receive and analyze data sent from the device. The server works in conjunction with generative artificial intelligence (AI) and a database to complement the data and generate the VR space. The server also has an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions by analyzing the user's facial expressions and voice data.

[1568] 3. VR System

[1569] The VR system used by users consists of VR goggles, gloves, and audio glasses. This allows users to perform visual and tactile manipulation within the VR space. The audio glasses also provide real-time audio guidance and explanations.

[1570] Program processing description

[1571] Physical space data collection

[1572] The device uses a laser sensor to scan and capture geometric data of the physical space. It also uses a camera to identify the placement of objects and collects this data as image data. Environmental data is also collected using a temperature sensor. This data is compressed and sent to a server in real time.

[1573] Data processing and VR space generation

[1574] The server analyzes the data received from the device and generates a 3D point cloud. Missing data is filled in using generative AI and a database. This generates a complete 3D model, which is then converted into a format optimized for the VR system.

[1575] Emotion recognition by emotion engine

[1576] The server's emotion engine analyzes facial expression and voice data acquired from the user and recognizes the user's emotions in real time. For example, it can grasp emotions by scanning the user's face with a camera and capturing the user's voice with a voice input device.

[1577] Manipulating VR space and responding to emotions

[1578] The user puts on VR goggles to access the VR space. They start the experience using optimization data sent from the server and use the gloves to manipulate objects in the VR space. The server's emotion engine monitors the user's emotions in real time and dynamically adjusts the environment and interface of the VR space based on their emotions. For example, if the user is feeling stressed, the system will change the environment using brighter, warmer colors.

[1579] Specific examples

[1580] Data Collection Example

[1581] The device scans an office room, capturing the room's shape with a laser sensor, photographing the furniture layout with a camera, collecting temperature data with a temperature sensor, and sending all data to a server.

[1582] Data Processing Examples

[1583] The server analyzes the received data and generates a 3D model of the office. Imperfections are filled in by generative AI, and optimized VR data is created and exported to the VR system.

[1584] Example of emotion recognition and VR space adjustment

[1585] The user puts on VR goggles to access the virtual office and discovers an abnormal temperature rise. They manipulate objects with the gloves to check temperature sensor data and listen to audio guidance through the audio glasses. If the server's emotion engine detects the user's anxiety, the system adjusts the lighting and plays relaxing music.

[1586] Prompt Sentence Examples

[1587] "Please explain how we can use VR space and emotion engines to test the most effective placement of new in-store promotional placements."

[1588] In this way, users can remotely obtain and manipulate detailed data of physical spaces, enabling highly efficient and accurate on-site investigations and troubleshooting. The emotion engine also enables flexible responses that adapt to the user's emotions.

[1589] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1590] Step 1:

[1591] The device uses a laser sensor to scan and capture geometric data of the physical space, which is then processed into an initial 3D point cloud and combined with image data captured by the camera, which is then compressed and sent to a server in real time.

[1592] (Input: Shape of physical space, image data, Output: 3D point cloud data)

[1593] Step 2:

[1594] The server analyzes the 3D point cloud data and image data received from the device, uses generative artificial intelligence (AI) to fill in any incomplete data, and generates a complete 3D model. It then converts this 3D model into a data format optimized for the VR system.

[1595] (Input: 3D point cloud data, image data, Output: complete 3D model)

[1596] Step 3:

[1597] The user puts on the VR goggles and accesses the VR space. The server sends optimized 3D model data to the VR goggles, allowing the user to begin operating within the VR space. The user uses the gloves to manipulate objects in the VR space and receives audio guidance and explanations through the audio glasses.

[1598] (Input: VR goggles, gloves, audio glasses, Output: Visual and tactile manipulation of VR space)

[1599] Step 4:

[1600] The server's emotion engine analyzes the user's facial expressions and voice data. It recognizes emotions by scanning the user's face with a camera and capturing the user's voice with a voice input device. This allows the system to grasp the user's emotional state, such as stress level and excitement level, in real time.

[1601] (Input: facial expression data, voice data, output: user's emotional state)

[1602] Step 5:

[1603] The server dynamically adjusts the VR environment based on the user's emotional state. For example, if the user is feeling stressed, the server softens the lighting and plays relaxing music. If the user is excited, the server brightens the lighting and plays upbeat music. This provides the optimal VR experience according to the user's emotions.

[1604] (Input: user's emotional state, Output: adjusted VR environment)

[1605] Step 6:

[1606] After the user finishes their work in the VR space, the server saves the collected data for future use. It also analyzes the user's emotional data and operation logs to gather feedback for further optimization of the experience. Based on this, the server makes improvements to ensure the next VR experience is more comfortable and efficient.

[1607] (Input: Operation and emotion data during VR experience, Output: Saved data, feedback results)

[1608] In this way, the system obtains detailed data of the physical space to generate a VR space, and dynamically adjusts it according to the user's emotions to provide an optimal VR experience.

[1609] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1610] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1611] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1612] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1613] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1614] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1615] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1616] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1617] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1618] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1619] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1620] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1621] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1622] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1623] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1624] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1625] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1626] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1627] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1628] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1629] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1630] The following is further disclosed regarding the above embodiment.

[1631] (Claim 1)

[1632] a means for collecting data of a physical space;

[1633] A means for analyzing the collected data and generating a virtual reality space;

[1634] means for providing an interface for a user to manipulate the virtual reality space;

[1635] A system including:

[1636] (Claim 2)

[1637] 2. The system of claim 1, wherein the means for collecting data on the physical space collects the shape of the physical space and the arrangement of objects using a laser sensor and a camera.

[1638] (Claim 3)

[1639] 2. The system according to claim 1, wherein the means for generating the virtual reality space cooperates with a generative artificial intelligence and a database to supplement missing data.

[1640] "Example 1"

[1641] (Claim 1)

[1642] a means for collecting data of a physical space;

[1643] means for transmitting the collected data to a server;

[1644] The server analyzes the received data, generates a 3D point cloud, and uses generative artificial intelligence and a database to fill in missing data and generate a complete 3D model;

[1645] means for optimizing the generated 3D model and providing an interface for a user to manipulate the virtual reality space;

[1646] A system including:

[1647] (Claim 2)

[1648] 2. The system of claim 1, wherein the means for collecting data on the physical space collects the shape of the physical space and the arrangement of objects using a laser sensor and a camera.

[1649] (Claim 3)

[1650] 2. The system of claim 1, wherein the means for optimizing the generated 3D model uses a high-performance processor and storage to convert the 3D model into a data format optimized for a VR system, and provides data for a user to manipulate a virtual reality space using VR goggles, gloves, and an audio device.

[1651] "Application Example 1"

[1652] (Claim 1)

[1653] a means for collecting data of a physical space;

[1654] A means for analyzing the collected data and generating a virtual reality space;

[1655] means for providing an interface for a user to manipulate the virtual reality space;

[1656] A means of detecting anomalies and notifying them in real time,

[1657] A system including:

[1658] (Claim 2)

[1659] 2. The system of claim 1, wherein the means for collecting data of the physical space uses a laser sensor and a camera to collect the shape of the physical space and the arrangement of objects, and also collects environmental data using a temperature sensor.

[1660] (Claim 3)

[1661] The system of claim 1, wherein the means for generating the virtual reality space cooperates with generative artificial intelligence and a database to supplement missing data and anomaly detection data.

[1662] "Example 2: Combining Emotion Engines"

[1663] (Claim 1)

[1664] a means for collecting data of a physical space;

[1665] A means for analyzing the collected data and generating a virtual reality space;

[1666] means for providing an interface for a user to manipulate the virtual reality space;

[1667] means for recognizing user emotions in real time;

[1668] means for dynamically adjusting the environment of the virtual reality space based on the user's emotions;

[1669] A system including:

[1670] (Claim 2)

[1671] 2. The system of claim 1, wherein the means for collecting data on the physical space collects the shape of the physical space and the arrangement of objects using a laser sensor and a camera.

[1672] (Claim 3)

[1673] 2. The system according to claim 1, wherein the means for generating the virtual reality space cooperates with a generative artificial intelligence and a database to supplement missing data.

[1674] "Application example 2 when combining emotion engines"

[1675] (Claim 1)

[1676] a means for collecting data of a physical space;

[1677] A means for analyzing the collected data and generating a virtual reality space;

[1678] means for providing an interface for a user to manipulate the virtual reality space;

[1679] A means for recognizing a user's emotion and dynamically adjusting the environment of the virtual reality space;

[1680] A system including:

[1681] (Claim 2)

[1682] 2. The system of claim 1, wherein the means for collecting data on the physical space collects the shape of the physical space and the arrangement of objects using a laser sensor and a camera.

[1683] (Claim 3)

[1684] 2. The system according to claim 1, wherein the means for generating the virtual reality space cooperates with a generative artificial intelligence and a database to supplement missing data. [Explanation of symbols]

[1685] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting data of a physical space; A means for analyzing the collected data and generating a virtual reality space; means for providing an interface for a user to manipulate the virtual reality space; A system including:

2. 2. The system according to claim 1, wherein the means for collecting data of the physical space collects the shape of the physical space and the arrangement of objects using a laser sensor and a camera.

3. 2. The system according to claim 1, wherein the means for generating the virtual reality space cooperates with a generative artificial intelligence and a database to supplement missing data.

Images