system

Abstract

We provide the system. [Solution] A 3D scanning method for acquiring point cloud data, A means for analyzing acquired point cloud data and generating a building information model, A means for automatically adding equipment attribute information based on the generated building information model, A generative artificial intelligence means that proposes the optimal equipment layout and piping route using equipment attribute information, A means for the user to select from several proposed equipment layout options and perform a digital simulation, A means of outputting the final design, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional facility management places a great burden on managers as facilities age and the scope of management expands. Also, due to its relation to manufacturing secrets, it is difficult to externalize management operations, and the establishment of an efficient management system is required. Thus, it is an issue to perform necessary facility repairs quickly and accurately and to achieve management efficiency and labor saving.

Means for Solving the Problems

[0005] This invention provides a means for acquiring and analyzing point cloud data to generate a building information model, and a means for automatically assigning equipment attribute information based on that model. Furthermore, it provides a system for efficiently formulating equipment renovation plans by using artificial intelligence to propose the optimal equipment layout and piping routes, and simulating the plan selected by the user. This reduces the burden of management tasks and enables rapid and accurate equipment management.

[0006] "Point cloud data" refers to a dataset containing three-dimensional coordinate information of numerous points that make up an object or space.

[0007] "3D scanning methods" refer to devices and technologies used to measure objects and spaces in three dimensions and acquire point cloud data.

[0008] A "building information model" is a digital model that represents the structure, dimensions, and location information of a building or its facilities in three dimensions.

[0009] "Attribute information" refers to information about the characteristics and specifications of equipment and structures, including model, material, and year of manufacture.

[0010] "Generative artificial intelligence means" refers to technologies that use artificial intelligence to analyze data and generate condition-based options or optimal solutions.

[0011] "Digital simulation" is a method of virtually reproducing equipment layout and operation on a computer to perform predictions and evaluations.

[0012] "Final design" refers to the completed design drawings and configuration based on the selected plan. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention provides a system for rapid and efficient facility management. First, a terminal equipped with 3D scanning capabilities moves around the facility, collecting detailed point cloud data. This data is reviewed in real time, and rescanning is performed as needed. Next, the terminal transmits the collected point cloud data to a server.

[0035] The server analyzes the received point cloud data and generates a building information model that represents the detailed 3D structure of the facility. This model integrates the shape, dimensions, and location information of all the equipment within the facility. The server then further analyzes the model and automatically adds attribute information to each piece of equipment. This attribute information includes the equipment's model, material, and installation date, encompassing all the information necessary for equipment management.

[0036] Based on the generated building information model, the server uses a generation AI to propose the optimal placement of equipment and piping routes. This process considers legal regulations, technical standards, and equipment compatibility, resulting in the generation of multiple realistic and implementable proposals. These proposals are then presented to the user via a terminal.

[0037] The user operates a terminal and reviews multiple equipment layout proposals generated by the AI. They select the plan that best matches their vision and then perform a digital simulation based on that selection. The simulation allows them to consider the appearance and functionality of the equipment within the facility in advance.

[0038] Finally, the server generates detailed design drawings based on the selected design proposal. These design drawings contain all the necessary information and can be used directly for actual construction or renovation.

[0039] Thus, by using the system of the present invention, facility managers can formulate quick and accurate renovation plans and implement planned and efficient facility management. A specific example of its use is scanning the boiler room of an aging factory and proposing an optimal piping configuration to improve the maintainability of the piping.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The terminal moves around the facility, performing 3D scanning and acquiring point cloud data. The terminal monitors the data in real time and rescans any areas where the imaging is insufficient.

[0043] Step 2:

[0044] The terminal collects point cloud data and transfers it to the server using data compression technology. The compressed data is transmitted at high speed with minimal latency.

[0045] Step 3:

[0046] The server analyzes the received point cloud data and generates a building information model that shows the three-dimensional structure of the entire facility. The server uses detailed algorithms to create a highly accurate model down to the smallest detail.

[0047] Step 4:

[0048] The server analyzes the building information model, identifies and automatically adds the dimensions and attribute information of each piece of equipment. It then compares this information with the existing database to fill in any missing details.

[0049] Step 5:

[0050] The server operates the generating artificial intelligence to propose the optimal placement of equipment and piping routes. The generating AI generates multiple design options, each of which is checked for compliance with regulations, technical standards, and compatibility between equipment.

[0051] Step 6:

[0052] The user operates the device and reviews multiple proposals presented by the server. The user compares each proposal and selects the one that best meets their requirements.

[0053] Step 7:

[0054] The terminal performs a digital simulation based on the user's selection. The simulation results are visualized in three dimensions, allowing the user to check the equipment layout on the screen.

[0055] Step 8:

[0056] The server finalizes the user's selected design and generates detailed design drawings and supporting documents. The drawings contain all the information necessary for the installation or modification of the equipment.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] In facility management, accurately understanding the facility's structure and efficiently and appropriately arranging equipment is crucial. However, conventional methods involve time-consuming data collection and analysis, making it difficult to quickly provide highly accurate recommendations. Furthermore, creating layout plans that take legal regulations and technical standards into account is not easy, hindering operational efficiency.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes a three-dimensional scanning element for acquiring point cloud data, an element for analyzing the acquired point cloud data and generating a structural information model, an element for automatically adding equipment attribute information based on the generated structural information model, and a generation machine learning element that uses the equipment attribute information to propose the optimal equipment layout and piping route. This enables rapid and highly accurate equipment management and the proposal of efficient layout plans.

[0062] "Point cloud data" is data consisting of countless points in space, used to represent the shape of objects or facilities.

[0063] A "three-dimensional scanning element" is a device or technology for acquiring the three-dimensional shape of an object with high precision.

[0064] A "structural information model" is a model that represents the detailed three-dimensional structure of a facility as digital data, including the position, dimensions, and shape of each element.

[0065] "Attribute information" refers to information that includes specific characteristics of devices and equipment, such as model number, material, and introduction date.

[0066] "Generative machine learning elements" are artificial intelligence technologies used to propose optimal equipment placement and piping routes based on data input.

[0067] "Digital simulation" is a simulation technology used to verify proposed equipment placement and piping route designs in a virtual environment.

[0068] "Final design" refers to design drawings and data that show the final details of the structure and equipment layout.

[0069] This invention is a system that integrates different technical elements for managing and arranging equipment within a facility. Its embodiments are described below.

[0070] The terminal first moves around the facility using a 3D scanner to collect detailed point cloud data. The collected point cloud data is acquired through laser measurement and camera photography and displayed visually on the terminal's screen in real time. The terminal is also equipped with a function to detect missing parts of the scanned data and rescan.

[0071] Subsequently, the terminal transmits this point cloud data to the server via a communication network. The server analyzes the point cloud data using dedicated analysis software (e.g., architectural CAD software) and generates a structural information model with a detailed three-dimensional structure of the facility. This structural information model integrates detailed shape, dimensions, and positional information for each device.

[0072] The server then adds attribute information for each device to the generated structural information model. A database system (e.g., a relational database) is used for this, and information such as the model number, material, and introduction date for each device is automatically added.

[0073] Next, the server uses a generative AI model to propose the optimal placement of equipment and piping routes. The generative AI model is pre-trained, taking into account legal regulations, technical standards, and equipment compatibility, and generates multiple realistic placement options based on this. An example of a prompt is, "Provide a piping design to optimize the boiler room of this facility."

[0074] These proposals are presented to the user on the device, allowing them to compare each proposal and choose the most suitable one. The selected configuration is simulated through a digital simulation experiment, allowing the user to check the visual arrangement and functionality of the equipment in advance.

[0075] Ultimately, based on the user's selection, the server generates detailed design drawings. These drawings completely include all the information necessary for the actual construction and renovation work.

[0076] One concrete example is the use of this system to scan the boiler room of an aging factory, propose an optimal piping configuration using a generated AI model, and improve maintainability. This system enables rapid and efficient equipment management and optimization.

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The terminal uses a 3D scanner to collect point cloud data within the facility. The point cloud data is acquired using the scanner's laser ranging and camera vision elements. Physical shape information within the facility is used as input, and a series of point cloud data is generated as output, displayed in real time on the terminal's screen. The user can rescan as needed to fill in any missing data.

[0080] Step 2:

[0081] The terminal sends the collected point cloud data to the server. Data transmission takes place over a communication network, and the terminal encodes the point cloud data obtained as output and provides it to the server as input. At this time, error checking is performed to maintain the integrity and completeness of the data.

[0082] Step 3:

[0083] The server analyzes the received point cloud data and generates a structural information model. Architectural CAD software is used for the analysis, processing the point cloud information as input data and generating a three-dimensional structural information model as output. This model integrates the shape, position, and dimensional information of each device.

[0084] Step 4:

[0085] The system automatically adds device attribute information to the structural information model generated by the server. This process uses information from a relational database as input and adds attribute information to the model as output. As a result, the model type, material, and installation date of each device are integrated with the model.

[0086] Step 5:

[0087] The server utilizes a generated AI model to propose optimal equipment placement and piping routes. The generated structural information model and related legal regulations and technical standards are used as input, and the AI ​​proposes multiple equipment placement options as output. A specific prompt might be, "Please provide a piping design to optimize the boiler room in this facility."

[0088] Step 6:

[0089] The terminal presents the user with proposals from the server. The user visually receives the proposed options as input and operates an interface to determine which plan to select as output. This allows the user to compare multiple options and choose the most suitable configuration.

[0090] Step 7:

[0091] A digital simulation experiment is conducted based on the user's most frequently selected layout. The selected layout is used as input, and the functionality and visual evaluation of that layout are verified through simulation as output.

[0092] Step 8:

[0093] The server generates detailed design drawings based on the ultimately selected layout. Using the selected layout as input, it completes design drawings containing all the information necessary for construction and renovation as output. These design drawings are generated by specific software and provided in an actionable format.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In large-scale production facilities such as factories, the placement of equipment and piping, and the installation of production lines, often require considerable effort and time. Furthermore, if these arrangements are not optimal, they can negatively impact efficiency and safety. In addition, specialized knowledge and experience are necessary to effectively utilize the limited space within the facility while meeting legal regulations and technical standards for equipment placement. There is a need for solutions that can address these challenges quickly and efficiently.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes a three-dimensional scanning means for acquiring point cloud information, a means for analyzing the acquired point cloud information and generating a structural information model, and a means for automatically adding attribute information of fixtures based on the generated structural information model. This makes it possible to propose optimal fixture placement and piping routes, and to generate optimal layout plans for manufacturing lines within production facilities.

[0099] "Point cloud information" is a collection of numerous points arranged in three dimensions to represent the shape and position of an object in space.

[0100] A "three-dimensional scanning means" is a device or method for acquiring three-dimensional information of a target space or object.

[0101] A "structural information model" is a three-dimensional model that includes detailed information such as the shape, dimensions, and location within a facility, generated by analyzing acquired point cloud information.

[0102] "Equipment attribute information" refers to data that includes detailed information necessary for management, such as the equipment model, material, and year of introduction.

[0103] A "generative intelligence model" is a technology that uses artificial intelligence to propose the optimal layout and routes within a facility based on data obtained.

[0104] "Virtual simulation" is a method of using digital models to consider actual designs and layouts in advance.

[0105] "Final design" refers to a detailed, actionable design drawing or plan based on the proposed plan.

[0106] To implement this invention, various specialized hardware and software are used. First, a device is needed to perform three-dimensional scanning in cooperation with a server in order to optimize the arrangement of equipment and piping within the facility. Specifically, a three-dimensional scanning means is used to collect basic point cloud information. The point cloud information obtained through this scanning is detailed three-dimensional information of equipment within the facility and their positional relationships.

[0107] Next, the server receives and analyzes this point cloud information to generate a structural information model. This information model integrates detailed information such as the shape, position, and dimensions of all the fixtures, and a generative intelligence model on the server automatically adds attribute information for the equipment based on this. This generative intelligence model utilizes artificial intelligence technology to propose the optimal fixture placement and routes from the point cloud data.

[0108] Users can perform virtual simulations based on these proposals and select the most suitable plan from the proposed options. This selected plan is then refined into a detailed design by the server and can be used to support the actual facility design.

[0109] A concrete example is optimizing the layout of production lines in a manufacturing facility. For instance, the server, based on acquired point cloud information, presents optimal layout proposals for existing piping and newly planned production lines, which the user then evaluates through virtual simulation. Finally, the design is finalized.

[0110] An example of a prompt for the generating AI model is: "Based on the following 3D point cloud data, generate an optimal equipment placement plan for the facility. We would like a feasible plan that takes safety standards into consideration." This prompt allows the generating AI model to provide a more precise and realistic placement plan.

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The terminal moves around the facility and acquires point cloud information using a three-dimensional scanning device. This point cloud information includes three-dimensional data regarding the position and shape of equipment and structures within the facility. The terminal collects this information in real time and rescans as needed.

[0114] Step 2:

[0115] The terminal transmits the acquired point cloud information to the server. The server receives this data as input and performs point cloud analysis as data processing. As a result of the analysis, a detailed structural information model of the facility is generated. This model includes details such as the shape, location, and dimensions of the fixtures.

[0116] Step 3:

[0117] The server automatically adds attribute information to each fixture using the generated structural information model. This attribute information includes the fixture's model, material, and year of introduction. This increases the comprehensiveness of the data.

[0118] Step 4:

[0119] The server generates prompt statements using a generative intelligence model based on structural information models and attribute information. This generative intelligence model optimizes equipment placement and piping routes, generating proposals that are realistic and take into account legal regulations and technical standards.

[0120] Step 5:

[0121] The user reviews multiple proposed layouts via a terminal and runs a virtual simulation. The user then selects the optimal plan, considering aesthetics and functionality. This selection process deepens the user's interaction with the system and facilitates their decision-making.

[0122] Step 6:

[0123] The server outputs the final design based on the user's selection. This design drawing contains all the necessary information and can be used directly for actual facility design and renovation.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention is a system that combines an emotion engine with an equipment management system to dynamically adjust the user interface and improve the user experience. In this embodiment, the emotion engine analyzes the user's facial expressions and voice, and grasps their emotional state in real time. Next, based on the obtained emotional data, the system adjusts the layout of the user interface and the priority of the information presented to increase user satisfaction.

[0126] Specifically, the device acquires the user's camera images and voice input and sends them to the emotion engine. The emotion engine uses modern algorithms to analyze facial expressions and tone of voice to understand emotions such as joy, surprise, and stress. Once this information is gathered, the server uses the emotion data to rearrange the order in which the generation AI proposes equipment layouts, displaying the option that causes the least stress to the user at the top.

[0127] Furthermore, if a user is experiencing stress, the server can provide supplementary information and interactive help to alleviate their anxieties and questions. For example, if a user has doubts about a new equipment layout proposal, the emotion engine can detect this, and the server can immediately display a detailed explanation in a pop-up window. This allows the user to proceed with their decision with confidence.

[0128] Furthermore, when the terminal performs a digital simulation of the selected equipment layout, the emotion engine further personalizes the experience. It automatically highlights detailed views that the user has shown interest in during the simulation process, making the interaction more intuitive. In this way, the present invention utilizes the emotion engine to provide a flexible interface that responds to the user's emotional state, thereby achieving more comfortable equipment management.

[0129] The following describes the processing flow.

[0130] Step 1:

[0131] The device acquires the user's camera images and audio data. This data is collected in real time and used to accurately understand the user's emotional state.

[0132] Step 2:

[0133] The device sends collected image and audio data to the emotion engine. The emotion engine uses advanced algorithms to analyze the user's facial expressions and tone of voice, and evaluates emotions such as joy, surprise, and stress in real time.

[0134] Step 3:

[0135] The emotion engine sends the analysis results to the server. The server receives the emotion data and determines the user's current psychological state.

[0136] Step 4:

[0137] Based on the emotional data received by the server, the generating AI adjusts the order in which it presents multiple proposed facility layouts. For example, if the user is feeling stressed, the server will either narrow down the options or prioritize displaying the simplest and easiest-to-understand proposal.

[0138] Step 5:

[0139] The user reviews the suggested options via their device. The emotion engine continuously monitors the user's reactions, and if it detects signs of dissatisfaction, the server displays additional information as a pop-up to try and alleviate their concerns.

[0140] Step 6:

[0141] After the user selects a layout option that interests them, the device performs a digital simulation of that option. During the simulation, the emotion engine monitors the user's emotional state and makes dynamic adjustments, such as highlighting aspects of interest.

[0142] Step 7:

[0143] Based on the final design approved by the user, the server outputs a detailed blueprint. This blueprint contains all the information necessary for on-site implementation, supporting planned and efficient work.

[0144] (Example 2)

[0145] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0146] Conventional facility management systems have the problem of limiting the user experience because they provide a fixed user interface without considering the user's emotional state. Furthermore, when presenting various facility layout options, they fail to consider the impact of the selection order on the user's emotions, thus failing to support optimal decision-making.

[0147] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0148] In this invention, the server includes means for acquiring the user's visual and auditory data and performing emotion analysis; means for dynamically adjusting the content of the user interface based on the user's emotional state; and means for generating artificial intelligence to generate equipment layout plans considering regulations, technical standards, and compatibility, and adjusting the presentation order of the layout plans based on the results of the emotion analysis. This enables the provision of a flexible interface that responds to the user's emotional state and the presentation of optimal equipment layout plans.

[0149] "Point cloud data" is a collection of numerous points that indicate positions in three-dimensional space, and is digital data that represents the shape and surface features of an object.

[0150] A "three-dimensional scanning method" is a device or method for acquiring three-dimensional data of an object or environment, often using lasers or optical technology.

[0151] A "structural information model" is a digital model that describes the structure of a building or equipment in three dimensions, including shape, dimensions, and location information.

[0152] "Equipment attribute information" refers to information that represents the characteristics of equipment, such as its function, performance, and materials, and is data used for equipment management and analysis.

[0153] "Generative artificial intelligence means for proposing routes" refers to a method that utilizes artificial intelligence technology to automatically plan the optimal placement of multiple pieces of equipment, as well as the piping and routes connecting them.

[0154] "Digital simulation" is a technology that uses a computer to recreate virtual environments and scenarios for conducting experiments and tests.

[0155] "Emotion analysis" is a technology that analyzes a user's facial expressions and voice to identify their emotional state, often utilizing computer vision or speech recognition.

[0156] "Means for dynamically adjusting the content presented in the user interface" refers to methods that change the arrangement of information on the screen and the priority of displayed content in real time based on the user's emotional state.

[0157] "Supporting optimal decision-making" means presenting users with the most effective and least stressful options, and assisting them in making appropriate decisions.

[0158] This invention provides a mechanism for a facility management system that offers a dynamic user interface that takes into account the user's emotional state. The aim is to improve the user experience and support effective decision-making.

[0159] The entire system consists mainly of terminals, servers, an emotion analysis engine, and a generative AI model. The terminals are devices with high-performance cameras and voice recognition capabilities (e.g., smartphones or dedicated tablets). These capture the user's facial expressions and voice data in real time and send it to the server.

[0160] The server uses an emotion analysis engine to analyze the user's emotional state. Deep learning techniques are used to identify emotions such as "joy," "surprise," and "stress" from facial expressions. The resulting emotional data is sent to a generative AI model and used to adjust the order in which equipment layout proposals are presented.

[0161] A concrete example is improving the layout of office equipment. When a user views multiple equipment layout options on screen, if they experience stress, the emotion analysis engine detects this reaction. The server then considers this emotional data and displays the layout options that are more relaxing for the user at the top of the list.

[0162] In this process, the following example prompt will be used: "Use the sentiment engine to analyze how the user is reacting to the current equipment layout proposal, and based on the results, suggest the next layout proposal to display."

[0163] In this way, the combination of emotion analysis and a dynamic user interface provides users with a comfortable and effective facility management experience.

[0164] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0165] Step 1:

[0166] The device uses a high-performance camera and microphone to capture the user's facial expressions and voice in real time. Input consists of the user's image and voice data, which are captured by sensors. Output is the acquired raw data, which is then sent to a server.

[0167] Step 2:

[0168] The server passes the user's image and audio data received from the terminal to the emotion analysis engine. Here, the input is the image and audio data from the terminal. The emotion analysis engine uses a deep learning algorithm to analyze the data and identify the user's emotional state (e.g., joy, surprise, stress). The output is the analyzed emotion data.

[0169] Step 3:

[0170] The server sends a prompt message to the generative AI model based on the emotion data obtained through emotion analysis. This prompt message is used to present an optimized facility layout for the user, and is in the form of "Based on the user's current emotion, please suggest the next layout to display." The input is emotion data and the prompt message, and the output is the layout proposal returned by the generative AI model.

[0171] Step 4:

[0172] The server receives the proposed facility layout from the generated AI model and reflects it in the user interface. During this process, it adjusts the presentation order based on emotional data, prioritizing the display of the layout that causes the user the least stress. The input is the generated layout, and the output is the adjusted user interface display.

[0173] Step 5:

[0174] The user selects from the provided equipment layout options and runs a digital simulation. The terminal reproduces the user's selected layout in a virtual environment and visually displays the simulation results. Here, the input is the user's selected layout option, and the output is the simulated visual data.

[0175] Step 6:

[0176] The server performs further sentiment analysis based on the simulation results and provides additional information and interactive help as needed. This is intended to address any questions or frustrations the user may have. The input is simulation feedback and sentiment data, and the output is tailored help information.

[0177] (Application Example 2)

[0178] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0179] Traditional facility management systems often provided a fixed user interface without considering the emotional state of the user. As a result, they were unable to flexibly provide information according to the user's situation, making it difficult to deliver a highly satisfying user experience. Furthermore, there was a lack of mechanisms to immediately provide appropriate support for the anxieties and questions users faced when selecting facility layout options, leading to a decrease in the efficiency of decision-making.

[0180] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0181] In this invention, the server includes means for acquiring the user's facial expressions and voice to analyze their emotions and dynamically adjusting the user interface based on the analysis results; means for providing appropriate additional information based on the analysis results to support better decision-making; and means for performing a digital simulation of the equipment layout plan selected by the user. This makes it possible to optimize the interface according to the user's emotional state and provide personalized information and support in real time.

[0182] A "three-dimensional scanning device" is a device used to acquire point cloud data, and it is a technology that scans the three-dimensional shape of an object and records it as digital data.

[0183] A "building information model" is a digital model generated by analyzing acquired point cloud data, and it includes information such as the shape, structure, and physical characteristics of a building.

[0184] "Equipment attribute information" refers to information that indicates the characteristics of equipment installed inside or outside a building, and includes data on shape, dimensions, and location.

[0185] "Data generation means" refers to methods and algorithms that use equipment attribute information to propose the optimal equipment layout and piping routes.

[0186] A "user interface" refers to an interactive screen or means of operation used for communication between the user and the system.

[0187] "Emotion analysis" is a process that determines a user's emotional state from their facial expressions and voice, and is a method for acquiring user emotional data in real time.

[0188] "Digital simulation" is a technology that aims to visualize and verify plans by reproducing selected equipment layouts in a virtual environment.

[0189] "Means of providing additional information" refers to a mechanism that dynamically presents relevant information and explanations to support user decisions and alleviate anxieties and questions.

[0190] The present invention relates to a system for equipment management that dynamically adjusts the interface based on user emotions to provide a more comfortable operating environment. This system includes a three-dimensional scanning device, a user terminal, a server, and a generative AI model.

[0191] The user terminal functions as a device such as smart glasses, capturing the user's facial expressions and voice. The acquired data is sent to an emotion analysis algorithm via the camera and microphone on the terminal. Here, libraries such as OpenCV and TENSORFLOW® in Python are used to analyze facial expressions and voice characteristics in real time and determine emotions such as joy, surprise, and stress.

[0192] The server dynamically adjusts the user interface based on data obtained through sentiment analysis. This enables optimal information display and interaction tailored to the user's emotional state. Furthermore, the server provides additional information as needed to support user decision-making. This process utilizes cloud services such as Google® Cloud API.

[0193] For example, if sentiment analysis determines that a user is confused or has questions about a new equipment layout plan, the server can immediately display a detailed explanation as a pop-up on the user's device. This allows the user to proceed to the next step with confidence.

[0194] For example, one possible prompt command to be input into the AI ​​model is, "Suggest a conversational response when the customer smiles." In this way, a variety of countermeasures can be provided to improve the user experience.

[0195] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0196] Step 1:

[0197] The device captures the user's facial expressions and voice through its camera and microphone. It collects the user's facial image data and voice data as input. This data is converted into a format that can be processed in real time and then transmitted to the next step.

[0198] Step 2:

[0199] On the device, an emotion analysis algorithm is executed using the acquired facial image data and audio data. Using libraries such as Python's OpenCV and TensorFlow, facial expressions and voice tone are analyzed. This process outputs emotion data such as joy, surprise, and stress.

[0200] Step 3:

[0201] The terminal sends the analyzed emotion data to the server. The server receives the emotion data as input and uses it to identify the user's emotional state. Based on this, the server performs the following interface adjustments.

[0202] Step 4:

[0203] The server dynamically adjusts the user interface layout and the information presented based on the received sentiment data. Using a generative AI model, it prioritizes displaying information of the user's greatest interest, reducing stress (e.g., highlighting specific instructions). The output of this process is the adjusted interface data.

[0204] Step 5:

[0205] The server provides additional information to users if they have any questions or concerns. Based on the user's emotional state and related system data as input, it selects and generates appropriate supplementary information and displays it as a pop-up on the screen. This output serves as supplementary information and guidance.

[0206] Step 6:

[0207] The user makes the final decision using the interface and information provided by the server. Furthermore, they receive specific equipment layout options and perform digital simulations as needed. The system receives the selected equipment layout as input, performs a simulation, and outputs a visualization of the final layout.

[0208] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0209] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0210] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0211] [Second Embodiment]

[0212] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0213] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0214] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0215] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0216] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0217] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0218] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0219] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0220] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0221] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0222] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0223] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0224] This invention provides a system for rapid and efficient facility management. First, a terminal equipped with 3D scanning capabilities moves around the facility, collecting detailed point cloud data. This data is reviewed in real time, and rescanning is performed as needed. Next, the terminal transmits the collected point cloud data to a server.

[0225] The server analyzes the received point cloud data and generates a building information model that represents the detailed 3D structure of the facility. This model integrates the shape, dimensions, and location information of all the equipment within the facility. The server then further analyzes the model and automatically adds attribute information to each piece of equipment. This attribute information includes the equipment's model, material, and installation date, encompassing all the information necessary for equipment management.

[0226] Based on the generated building information model, the server uses a generation AI to propose the optimal placement of equipment and piping routes. This process considers legal regulations, technical standards, and equipment compatibility, resulting in the generation of multiple realistic and implementable proposals. These proposals are then presented to the user via a terminal.

[0227] The user operates a terminal and reviews multiple equipment layout proposals generated by the AI. They select the plan that best matches their vision and then perform a digital simulation based on that selection. The simulation allows them to consider the appearance and functionality of the equipment within the facility in advance.

[0228] Finally, the server generates detailed design drawings based on the selected design proposal. These design drawings contain all the necessary information and can be used directly for actual construction or renovation.

[0229] Thus, by using the system of the present invention, facility managers can formulate quick and accurate renovation plans and implement planned and efficient facility management. A specific example of its use is scanning the boiler room of an aging factory and proposing an optimal piping configuration to improve the maintainability of the piping.

[0230] The following describes the processing flow.

[0231] Step 1:

[0232] The terminal moves around the facility, performing 3D scanning and acquiring point cloud data. The terminal monitors the data in real time and rescans any areas where the imaging is insufficient.

[0233] Step 2:

[0234] The terminal collects point cloud data and transfers it to the server using data compression technology. The compressed data is transmitted at high speed with minimal latency.

[0235] Step 3:

[0236] The server analyzes the received point cloud data and generates a building information model that shows the three-dimensional structure of the entire facility. The server uses detailed algorithms to create a highly accurate model down to the smallest detail.

[0237] Step 4:

[0238] The server analyzes the building information model, identifies and automatically adds the dimensions and attribute information of each piece of equipment. It then compares this information with the existing database to fill in any missing details.

[0239] Step 5:

[0240] The server operates the generating artificial intelligence to propose the optimal placement of equipment and piping routes. The generating AI generates multiple design options, each of which is checked for compliance with regulations, technical standards, and compatibility between equipment.

[0241] Step 6:

[0242] The user operates the device and reviews multiple proposals presented by the server. The user compares each proposal and selects the one that best meets their requirements.

[0243] Step 7:

[0244] The terminal performs a digital simulation based on the user's selection. The simulation results are visualized in three dimensions, allowing the user to check the equipment layout on the screen.

[0245] Step 8:

[0246] The server finalizes the user's selected design and generates detailed design drawings and supporting documents. The drawings contain all the information necessary for the installation or modification of the equipment.

[0247] (Example 1)

[0248] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0249] In facility management, accurately understanding the facility's structure and efficiently and appropriately arranging equipment is crucial. However, conventional methods involve time-consuming data collection and analysis, making it difficult to quickly provide highly accurate recommendations. Furthermore, creating layout plans that take legal regulations and technical standards into account is not easy, hindering operational efficiency.

[0250] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0251] In this invention, the server includes a three-dimensional scanning element for acquiring point cloud data, an element for analyzing the acquired point cloud data and generating a structural information model, an element for automatically adding equipment attribute information based on the generated structural information model, and a generation machine learning element that uses the equipment attribute information to propose the optimal equipment layout and piping route. This enables rapid and highly accurate equipment management and the proposal of efficient layout plans.

[0252] "Point cloud data" is data consisting of countless points in space, used to represent the shape of objects or facilities.

[0253] A "three-dimensional scanning element" is a device or technology for acquiring the three-dimensional shape of an object with high precision.

[0254] A "structural information model" is a model that represents the detailed three-dimensional structure of a facility as digital data, including the position, dimensions, and shape of each element.

[0255] "Attribute information" refers to information that includes specific characteristics of devices and equipment, such as model number, material, and introduction date.

[0256] "Generative machine learning elements" are artificial intelligence technologies used to propose optimal equipment placement and piping routes based on data input.

[0257] "Digital simulation" is a simulation technology used to verify proposed equipment placement and piping route designs in a virtual environment.

[0258] "Final design" refers to design drawings and data that show the final determined details of the structure and equipment layout.

[0259] This invention is a system that integrates different technical elements for managing and arranging equipment within a facility. Its embodiments are described below.

[0260] The terminal first moves around the facility using a 3D scanner to collect detailed point cloud data. The collected point cloud data is acquired through laser measurement and camera photography and displayed visually on the terminal's screen in real time. The terminal is also equipped with a function to detect missing parts of the scanned data and rescan.

[0261] Subsequently, the terminal transmits this point cloud data to the server via a communication network. The server analyzes the point cloud data using dedicated analysis software (e.g., architectural CAD software) and generates a structural information model with a detailed three-dimensional structure of the facility. This structural information model integrates detailed shape, dimensions, and positional information for each device.

[0262] The server then adds attribute information for each device to the generated structural information model. A database system (e.g., a relational database) is used for this, and information such as the model number, material, and introduction date for each device is automatically added.

[0263] Next, the server uses a generative AI model to propose the optimal placement of equipment and piping routes. The generative AI model is pre-trained, taking into account legal regulations, technical standards, and equipment compatibility, and generates multiple realistic placement options based on this. An example of a prompt is, "Provide a piping design to optimize the boiler room of this facility."

[0264] These proposals are presented to the user on the device, allowing them to compare each proposal and choose the most suitable one. The selected configuration is simulated through a digital simulation experiment, allowing the user to check the visual arrangement and functionality of the equipment in advance.

[0265] Ultimately, based on the user's selection, the server generates detailed design drawings. These drawings completely include all the information necessary for the actual construction and renovation work.

[0266] One concrete example is the use of this system to scan the boiler room of an aging factory, propose an optimal piping configuration using a generated AI model, and improve maintainability. This system enables rapid and efficient equipment management and optimization.

[0267] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0268] Step 1:

[0269] The terminal uses a 3D scanner to collect point cloud data within the facility. The point cloud data is acquired using the scanner's laser ranging and camera vision elements. Physical shape information within the facility is used as input, and a series of point cloud data is generated as output, displayed in real time on the terminal's screen. The user can rescan as needed to fill in any missing data.

[0270] Step 2:

[0271] The terminal sends the collected point cloud data to the server. Data transmission takes place over a communication network, and the terminal encodes the point cloud data obtained as output and provides it to the server as input. At this time, error checking is performed to maintain the integrity and completeness of the data.

[0272] Step 3:

[0273] The server analyzes the received point cloud data and generates a structural information model. Architectural CAD software is used for the analysis, processing the point cloud information as input data and generating a three-dimensional structural information model as output. This model integrates the shape, position, and dimensions of each device.

[0274] Step 4:

[0275] The system automatically adds device attribute information to the structural information model generated by the server. This process uses information from a relational database as input and adds attribute information to the model as output. As a result, the model type, material, and installation date of each device are integrated with the model.

[0276] Step 5:

[0277] The server utilizes a generative AI model to propose optimal equipment layouts and piping routes. The generated structural information model and related regulations and technical standards are used as inputs, and multiple equipment layout plans are proposed by the generative AI as outputs. A specific prompt sentence is, "Please provide a piping design to optimize the boiler room of this facility."

[0278] Step 6:

[0279] The terminal presents the server's proposal to the user. The user visually receives the proposed plans provided as inputs and operates an interface to determine the plan to be selected as the output. This enables the user to compare multiple plans and select the most suitable layout.

[0280] Step 7:

[0281] Based on the layout plan most selected by the user, a digital simulation experiment is conducted. Using the selected layout plan as an input, the functionality and visual evaluation of the layout are confirmed by simulation as outputs.

[0282] Step 8:

[0283] The server generates detailed design drawings based on the finally selected layout plan. Using the selected layout plan as an input, design drawings containing all the information necessary for construction and renovation are completed as outputs. These design drawings are generated by specific software and provided in an executable form.

[0284] (Application Example 1)

[0285] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0286] In large-scale production facilities such as factories, it often takes a great deal of labor and time to arrange instruments and pipes and introduce production lines. Also, if these arrangements are not optimal, they may have an adverse effect on efficiency and safety. Furthermore, specialized knowledge and experience are required to effectively utilize the limited space within the facility while meeting the regulatory and technical standards for equipment layout. There is a desire to provide a means for quickly and efficiently solving these problems.

[0287] The specific processing by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is realized by the following respective means.

[0288] In this invention, the server includes three-dimensional scanning means for acquiring point cloud information, means for analyzing the acquired point cloud information and generating a structure information model, and means for automatically adding attribute information of instruments based on the generated structure information model. As a result, it becomes possible to propose an optimal instrument layout and piping route and generate an optimal layout plan for production lines within the production facility.

[0289] The "point cloud information" is a set of a large number of points arranged three-dimensionally to represent the shape and position of an object in space.

[0290] The "three-dimensional scanning means" is a device or method for acquiring three-dimensional information of a target space or object.

[0291] The "structure information model" is a three-dimensional model generated by analyzing the acquired point cloud information and including detailed information such as the shape, dimensions, and position within the facility.

[0292] The "attribute information of instruments" is data including detailed information necessary for management, such as the type, material, and introduction year of the instruments.

[0293] The "generated intelligent model" is a technology for proposing an optimal layout and route within the facility based on data obtained using artificial intelligence.

[0294] "Virtual simulation" is a method of using digital models to consider actual designs and layouts in advance.

[0295] "Final design" refers to a detailed, actionable design drawing or plan based on the proposed plan.

[0296] To implement this invention, various specialized hardware and software are used. First, a device is needed to perform three-dimensional scanning in cooperation with a server in order to optimize the arrangement of equipment and piping within the facility. Specifically, a three-dimensional scanning means is used to collect basic point cloud information. The point cloud information obtained through this scanning is detailed three-dimensional information of equipment within the facility and their positional relationships.

[0297] Next, the server receives and analyzes this point cloud information to generate a structural information model. This information model integrates detailed information such as the shape, position, and dimensions of all the fixtures, and a generative intelligence model on the server automatically adds attribute information for the equipment based on this. This generative intelligence model utilizes artificial intelligence technology to propose the optimal fixture placement and routes from the point cloud data.

[0298] Users can perform virtual simulations based on these proposals and select the most suitable plan from the proposed options. This selected plan is then refined into a detailed design by the server and can be used to support the actual facility design.

[0299] A concrete example is optimizing the layout of production lines in a manufacturing facility. For instance, the server, based on acquired point cloud information, presents optimal layout proposals for existing piping and newly planned production lines, which the user then evaluates through virtual simulation. Finally, the design is finalized.

[0300] Examples of prompt texts for the generation AI model include "Please generate an optimal appliance layout plan within the facility based on the following 3D point cloud data. A plan that is implementable and takes safety standards into consideration is desired." With this prompt, the generation AI model provides a more refined and realistic layout plan.

[0301] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0302] Step 1:

[0303] The terminal acquires point cloud information using three-dimensional scanning means while moving within the facility. This point cloud information includes three-dimensional data on the positions and shapes of appliances and structures within the facility. The terminal collects this information in real time and performs rescan as necessary.

[0304] Step 2:

[0305] The terminal transmits the acquired point cloud information to the server. The server receives this data as input and performs analysis of the point cloud as data processing. As a result of the analysis, a detailed structure information model within the facility is generated. This model includes details such as the shape, position, and dimensions of the appliances.

[0306] Step 3:

[0307] The server automatically adds attribute information to each appliance using the generated structure information model. The attribute information here is the type, material, introduction year, etc. of the appliance. This enhances the comprehensiveness of the data.

[0308] Step 4:

[0309] The server generates a prompt text using the generation intelligence model based on the structure information model and the attribute information. This generation intelligence model aims to optimize the appliance layout and piping route and generates proposals. The proposals are realistic considering legal regulations and technical standards.

[0310] Step 5:

[0311] The user reviews multiple proposed layouts via a terminal and runs a virtual simulation. The user then selects the optimal plan, considering aesthetics and functionality. This selection process deepens the user's interaction with the system and facilitates their decision-making.

[0312] Step 6:

[0313] The server outputs the final design based on the user's selection. This design drawing contains all the necessary information and can be used directly for actual facility design and renovation.

[0314] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0315] This invention is a system that combines an emotion engine with an equipment management system to dynamically adjust the user interface and improve the user experience. In this embodiment, the emotion engine analyzes the user's facial expressions and voice, and grasps their emotional state in real time. Next, based on the obtained emotional data, the system adjusts the layout of the user interface and the priority of the information presented to increase user satisfaction.

[0316] Specifically, the device acquires the user's camera images and voice input and sends them to the emotion engine. The emotion engine uses modern algorithms to analyze facial expressions and tone of voice to understand emotions such as joy, surprise, and stress. Once this information is gathered, the server uses the emotion data to rearrange the order in which the generation AI proposes equipment layouts, displaying the option that causes the least stress to the user at the top.

[0317] Furthermore, if a user is experiencing stress, the server can provide supplementary information and interactive help to alleviate their anxieties and questions. For example, if a user has doubts about a new equipment layout proposal, the emotion engine can detect this, and the server can immediately display a detailed explanation in a pop-up window. This allows the user to proceed with their decision with confidence.

[0318] Furthermore, when the terminal performs a digital simulation of the selected equipment layout, the emotion engine further personalizes the experience. It automatically highlights detailed views that the user has shown interest in during the simulation process, making the interaction more intuitive. In this way, the present invention utilizes the emotion engine to provide a flexible interface that responds to the user's emotional state, thereby achieving more comfortable equipment management.

[0319] The following describes the processing flow.

[0320] Step 1:

[0321] The device acquires the user's camera images and audio data. This data is collected in real time and used to accurately understand the user's emotional state.

[0322] Step 2:

[0323] The device sends collected image and audio data to the emotion engine. The emotion engine uses advanced algorithms to analyze the user's facial expressions and tone of voice, and evaluates emotions such as joy, surprise, and stress in real time.

[0324] Step 3:

[0325] The emotion engine sends the analysis results to the server. The server receives the emotion data and determines the user's current psychological state.

[0326] Step 4:

[0327] Based on the emotional data received by the server, the generating AI adjusts the order in which it presents multiple proposed facility layouts. For example, if the user is feeling stressed, the server will either narrow down the options or prioritize displaying the simplest and easiest-to-understand proposal.

[0328] Step 5:

[0329] The user reviews the suggested options via their device. The emotion engine continuously monitors the user's reactions, and if it detects signs of dissatisfaction, the server displays additional information as a pop-up to try and alleviate their concerns.

[0330] Step 6:

[0331] After the user selects a layout option that interests them, the device performs a digital simulation of that option. During the simulation, the emotion engine monitors the user's emotional state and makes dynamic adjustments, such as highlighting aspects of interest.

[0332] Step 7:

[0333] Based on the final design approved by the user, the server outputs a detailed blueprint. This blueprint contains all the information necessary for on-site implementation, supporting planned and efficient work.

[0334] (Example 2)

[0335] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0336] Conventional facility management systems have the problem of limiting the user experience because they provide a fixed user interface without considering the user's emotional state. Furthermore, when presenting various facility layout options, they fail to consider the impact of the selection order on the user's emotions, thus failing to support optimal decision-making.

[0337] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0338] In this invention, the server includes means for acquiring the user's visual and auditory data and performing emotion analysis; means for dynamically adjusting the content of the user interface based on the user's emotional state; and means for generating artificial intelligence to generate equipment layout plans considering regulations, technical standards, and compatibility, and adjusting the presentation order of the layout plans based on the results of the emotion analysis. This enables the provision of a flexible interface that responds to the user's emotional state and the presentation of optimal equipment layout plans.

[0339] "Point cloud data" is a collection of numerous points that indicate positions in three-dimensional space, and is digital data that represents the shape and surface features of an object.

[0340] A "three-dimensional scanning method" is a device or method for acquiring three-dimensional data of an object or environment, often using lasers or optical technology.

[0341] A "structural information model" is a digital model that describes the structure of a building or equipment in three dimensions, including shape, dimensions, and location information.

[0342] "Equipment attribute information" refers to information that represents the characteristics of equipment, such as its function, performance, and materials, and is data used for equipment management and analysis.

[0343] "Generative artificial intelligence means for proposing routes" refers to a method that utilizes artificial intelligence technology to automatically plan the optimal placement of multiple pieces of equipment, as well as the piping and routes connecting them.

[0344] "Digital simulation" is a technology that uses a computer to recreate virtual environments and scenarios for conducting experiments and tests.

[0345] "Emotion analysis" is a technology that analyzes a user's facial expressions and voice to identify their emotional state, often utilizing computer vision or speech recognition.

[0346] "Means for dynamically adjusting the content presented in the user interface" refers to methods that change the arrangement of information on the screen and the priority of displayed content in real time based on the user's emotional state.

[0347] "Supporting optimal decision-making" means presenting users with the most effective and least stressful options, and assisting them in making appropriate decisions.

[0348] This invention provides a mechanism for a facility management system that offers a dynamic user interface that takes into account the user's emotional state. The aim is to improve the user experience and support effective decision-making.

[0349] The entire system consists mainly of terminals, servers, an emotion analysis engine, and a generative AI model. The terminals are devices with high-performance cameras and voice recognition capabilities (e.g., smartphones or dedicated tablets). These capture the user's facial expressions and voice data in real time and send it to the server.

[0350] The server uses an emotion analysis engine to analyze the user's emotional state. Deep learning techniques are used to identify emotions such as "joy," "surprise," and "stress" from facial expressions. The resulting emotional data is sent to a generative AI model and used to adjust the order in which equipment layout proposals are presented.

[0351] A concrete example is improving the layout of office equipment. When a user views multiple equipment layout options on screen, if they experience stress, the emotion analysis engine detects this reaction. The server then considers this emotional data and displays the layout options that are more relaxing for the user at the top of the list.

[0352] In this process, the following example prompt will be used: "Use the sentiment engine to analyze how the user is reacting to the current equipment layout proposal, and based on the results, suggest the next layout proposal to display."

[0353] In this way, the combination of emotion analysis and a dynamic user interface provides users with a comfortable and effective facility management experience.

[0354] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0355] Step 1:

[0356] The device uses a high-performance camera and microphone to capture the user's facial expressions and voice in real time. Input consists of the user's image and voice data, which are captured by sensors. Output is the acquired raw data, which is then sent to a server.

[0357] Step 2:

[0358] The server passes the user's image and audio data received from the terminal to the emotion analysis engine. Here, the input is the image and audio data from the terminal. The emotion analysis engine uses a deep learning algorithm to analyze the data and identify the user's emotional state (e.g., joy, surprise, stress). The output is the analyzed emotion data.

[0359] Step 3:

[0360] The server sends a prompt message to the generative AI model based on the emotion data obtained through emotion analysis. This prompt message is used to present an optimized facility layout for the user, and is in the form of "Based on the user's current emotion, please suggest the next layout to display." The input is emotion data and the prompt message, and the output is the layout proposal returned by the generative AI model.

[0361] Step 4:

[0362] The server receives the proposed facility layout from the generated AI model and reflects it in the user interface. During this process, it adjusts the presentation order based on emotional data, prioritizing the display of the layout that causes the user the least stress. The input is the generated layout, and the output is the adjusted user interface display.

[0363] Step 5:

[0364] The user selects from the provided equipment layout options and runs a digital simulation. The terminal reproduces the user's selected layout in a virtual environment and visually displays the simulation results. Here, the input is the user's selected layout option, and the output is the simulated visual data.

[0365] Step 6:

[0366] The server performs further sentiment analysis based on the simulation results and provides additional information and interactive help as needed. This is intended to address any questions or frustrations the user may have. The input is simulation feedback and sentiment data, and the output is tailored help information.

[0367] (Application Example 2)

[0368] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0369] Traditional facility management systems often provided a fixed user interface without considering the emotional state of the user. As a result, they were unable to flexibly provide information according to the user's situation, making it difficult to deliver a highly satisfying user experience. Furthermore, there was a lack of mechanisms to immediately provide appropriate support for the anxieties and questions users faced when selecting facility layout options, leading to a decrease in the efficiency of decision-making.

[0370] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0371] In this invention, the server includes means for acquiring the user's facial expressions and voice to analyze their emotions and dynamically adjusting the user interface based on the analysis results; means for providing appropriate additional information based on the analysis results to support better decision-making; and means for performing a digital simulation of the equipment layout plan selected by the user. This makes it possible to optimize the interface according to the user's emotional state and provide personalized information and support in real time.

[0372] A "three-dimensional scanning device" is a device used to acquire point cloud data, and it is a technology that scans the three-dimensional shape of an object and records it as digital data.

[0373] A "building information model" is a digital model generated by analyzing acquired point cloud data, and it includes information such as the shape, structure, and physical characteristics of a building.

[0374] "Equipment attribute information" refers to information that indicates the characteristics of equipment installed inside or outside a building, and includes data on shape, dimensions, and location.

[0375] "Data generation means" refers to methods and algorithms that use equipment attribute information to propose the optimal equipment layout and piping routes.

[0376] A "user interface" refers to an interactive screen or means of operation used for communication between the user and the system.

[0377] "Emotion analysis" is a process that determines a user's emotional state from their facial expressions and voice, and is a method for acquiring user emotional data in real time.

[0378] "Digital simulation" is a technology that aims to visualize and verify plans by reproducing selected equipment layouts in a virtual environment.

[0379] "Means of providing additional information" refers to a mechanism that dynamically presents relevant information and explanations to support user decisions and alleviate anxieties and questions.

[0380] The present invention relates to a system for equipment management that dynamically adjusts the interface based on user emotions to provide a more comfortable operating environment. This system includes a three-dimensional scanning device, a user terminal, a server, and a generative AI model.

[0381] The user terminal functions as a device such as smart glasses, capturing the user's facial expressions and voice. The acquired data is sent to an emotion analysis algorithm via the camera and microphone on the terminal. Here, libraries such as OpenCV and TensorFlow in Python are used to analyze facial expressions and voice characteristics in real time and determine emotions such as joy, surprise, and stress.

[0382] The server dynamically adjusts the user interface based on data obtained through sentiment analysis. This enables optimal information display and interaction tailored to the user's emotional state. Furthermore, the server provides additional information as needed to support user decision-making. This process utilizes cloud services such as Google Cloud APIs.

[0383] For example, if sentiment analysis determines that a user is confused or has questions about a new equipment layout plan, the server can immediately display a detailed explanation as a pop-up on the user's device. This allows the user to proceed to the next step with confidence.

[0384] For example, one possible prompt command to be input into the AI ​​model is, "Suggest a conversational response when the customer smiles." In this way, a variety of countermeasures can be provided to improve the user experience.

[0385] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0386] Step 1:

[0387] The device captures the user's facial expressions and voice through its camera and microphone. It collects the user's facial image data and voice data as input. This data is converted into a format that can be processed in real time and then transmitted to the next step.

[0388] Step 2:

[0389] On the device, an emotion analysis algorithm is executed using the acquired facial image data and audio data. Using libraries such as Python's OpenCV and TensorFlow, facial expressions and voice tone are analyzed. This process outputs emotion data such as joy, surprise, and stress.

[0390] Step 3:

[0391] The terminal sends the analyzed emotion data to the server. The server receives the emotion data as input and uses it to identify the user's emotional state. Based on this, the server performs the following interface adjustments.

[0392] Step 4:

[0393] The server dynamically adjusts the user interface layout and the information presented based on the received sentiment data. Using a generative AI model, it prioritizes displaying information of the user's greatest interest, reducing stress (e.g., highlighting specific instructions). The output of this process is the adjusted interface data.

[0394] Step 5:

[0395] The server provides additional information to users if they have any questions or concerns. Based on the user's emotional state and related system data as input, it selects and generates appropriate supplementary information and displays it as a pop-up on the screen. This output serves as supplementary information and guidance.

[0396] Step 6:

[0397] The user makes the final decision using the interface and information provided by the server. Furthermore, they receive specific equipment layout options and perform digital simulations as needed. The system receives the selected equipment layout as input, performs a simulation, and outputs a visualization of the final layout.

[0398] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0399] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0400] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0401] [Third Embodiment]

[0402] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0403] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0404] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0405] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0406] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0407] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0408] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0409] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0410] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0411] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0412] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0413] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0414] This invention provides a system for rapid and efficient facility management. First, a terminal equipped with 3D scanning capabilities moves around the facility, collecting detailed point cloud data. This data is reviewed in real time, and rescanning is performed as needed. Next, the terminal transmits the collected point cloud data to a server.

[0415] The server analyzes the received point cloud data and generates a building information model that represents the detailed 3D structure of the facility. This model integrates the shape, dimensions, and location information of all the equipment within the facility. The server then further analyzes the model and automatically adds attribute information to each piece of equipment. This attribute information includes the equipment's model, material, and installation date, encompassing all the information necessary for equipment management.

[0416] Based on the generated building information model, the server uses a generation AI to propose the optimal placement of equipment and piping routes. This process considers legal regulations, technical standards, and equipment compatibility, resulting in the generation of multiple realistic and implementable proposals. These proposals are then presented to the user via a terminal.

[0417] The user operates a terminal and reviews multiple equipment layout proposals generated by the AI. They select the plan that best matches their vision and then perform a digital simulation based on that selection. The simulation allows them to consider the appearance and functionality of the equipment within the facility in advance.

[0418] Finally, the server generates detailed design drawings based on the selected design proposal. These design drawings contain all the necessary information and can be used directly for actual construction or renovation.

[0419] Thus, by using the system of the present invention, facility managers can formulate quick and accurate renovation plans and implement planned and efficient facility management. A specific example of its use is scanning the boiler room of an aging factory and proposing an optimal piping configuration to improve the maintainability of the piping.

[0420] The following describes the processing flow.

[0421] Step 1:

[0422] The terminal moves around the facility, performing 3D scanning and acquiring point cloud data. The terminal monitors the data in real time and rescans any areas where the imaging is insufficient.

[0423] Step 2:

[0424] The terminal collects point cloud data and transfers it to the server using data compression technology. The compressed data is transmitted at high speed with minimal latency.

[0425] Step 3:

[0426] The server analyzes the received point cloud data and generates a building information model that shows the three-dimensional structure of the entire facility. The server uses detailed algorithms to create a highly accurate model down to the smallest detail.

[0427] Step 4:

[0428] The server analyzes the building information model, identifies and automatically adds the dimensions and attribute information of each piece of equipment. It then compares this information with the existing database to fill in any missing details.

[0429] Step 5:

[0430] The server operates the generating artificial intelligence to propose the optimal placement of equipment and piping routes. The generating AI generates multiple design options, each of which is checked for compliance with regulations, technical standards, and compatibility between equipment.

[0431] Step 6:

[0432] The user operates the device and reviews multiple proposals presented by the server. The user compares each proposal and selects the one that best meets their requirements.

[0433] Step 7:

[0434] The terminal performs a digital simulation based on the user's selection. The simulation results are visualized in three dimensions, allowing the user to check the equipment layout on the screen.

[0435] Step 8:

[0436] The server finalizes the user's selected design and generates detailed design drawings and supporting documents. The drawings contain all the information necessary for the installation or modification of the equipment.

[0437] (Example 1)

[0438] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0439] In facility management, accurately understanding the facility's structure and efficiently and appropriately arranging equipment is crucial. However, conventional methods involve time-consuming data collection and analysis, making it difficult to quickly provide highly accurate recommendations. Furthermore, creating layout plans that take legal regulations and technical standards into account is not easy, hindering operational efficiency.

[0440] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0441] In this invention, the server includes a three-dimensional scanning element for acquiring point cloud data, an element for analyzing the acquired point cloud data and generating a structural information model, an element for automatically adding equipment attribute information based on the generated structural information model, and a generation machine learning element that uses the equipment attribute information to propose the optimal equipment layout and piping route. This enables rapid and highly accurate equipment management and the proposal of efficient layout plans.

[0442] "Point cloud data" is data consisting of countless points in space, used to represent the shape of objects or facilities.

[0443] A "three-dimensional scanning element" is a device or technology for acquiring the three-dimensional shape of an object with high precision.

[0444] A "structural information model" is a model that represents the detailed three-dimensional structure of a facility as digital data, including the position, dimensions, and shape of each element.

[0445] "Attribute information" refers to information that includes specific characteristics of devices and equipment, such as model number, material, and introduction date.

[0446] "Generative machine learning elements" are artificial intelligence technologies used to propose optimal equipment placement and piping routes based on data input.

[0447] "Digital simulation" is a simulation technology used to verify proposed equipment placement and piping route designs in a virtual environment.

[0448] "Final design" refers to design drawings and data that show the final determined details of the structure and equipment layout.

[0449] This invention is a system that integrates different technical elements for managing and arranging equipment within a facility. Its embodiments are described below.

[0450] The terminal first moves around the facility using a 3D scanner to collect detailed point cloud data. The collected point cloud data is acquired through laser measurement and camera photography and displayed visually on the terminal's screen in real time. The terminal is also equipped with a function to detect missing parts of the scanned data and rescan.

[0451] Subsequently, the terminal transmits this point cloud data to the server via a communication network. The server analyzes the point cloud data using dedicated analysis software (e.g., architectural CAD software) and generates a structural information model with a detailed three-dimensional structure of the facility. This structural information model integrates detailed shape, dimensions, and positional information for each device.

[0452] The server then adds attribute information for each device to the generated structural information model. A database system (e.g., a relational database) is used for this, and information such as the model number, material, and introduction date for each device is automatically added.

[0453] Next, the server uses a generative AI model to propose the optimal placement of equipment and piping routes. The generative AI model is pre-trained, taking into account legal regulations, technical standards, and equipment compatibility, and generates multiple realistic placement options based on this. An example of a prompt is, "Provide a piping design to optimize the boiler room of this facility."

[0454] These proposals are presented to the user on the device, allowing them to compare each proposal and choose the most suitable one. The selected configuration is simulated through a digital simulation experiment, allowing the user to check the visual arrangement and functionality of the equipment in advance.

[0455] Ultimately, based on the user's selection, the server generates detailed design drawings. These drawings completely include all the information necessary for the actual construction and renovation work.

[0456] One concrete example is the use of this system to scan the boiler room of an aging factory, propose an optimal piping configuration using a generated AI model, and improve maintainability. This system enables rapid and efficient equipment management and optimization.

[0457] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0458] Step 1:

[0459] The terminal uses a 3D scanner to collect point cloud data within the facility. The point cloud data is acquired using the scanner's laser ranging and camera vision elements. Physical shape information within the facility is used as input, and a series of point cloud data is generated as output, displayed in real time on the terminal's screen. The user can rescan as needed to fill in any missing data.

[0460] Step 2:

[0461] The terminal sends the collected point cloud data to the server. Data transmission takes place over a communication network, and the terminal encodes the point cloud data obtained as output and provides it to the server as input. At this time, error checking is performed to maintain the integrity and completeness of the data.

[0462] Step 3:

[0463] The server analyzes the received point cloud data and generates a structural information model. Architectural CAD software is used for the analysis, processing the point cloud information as input data and generating a three-dimensional structural information model as output. This model integrates the shape, position, and dimensions of each device.

[0464] Step 4:

[0465] The system automatically adds device attribute information to the structural information model generated by the server. This process uses information from a relational database as input and adds attribute information to the model as output. As a result, the model type, material, and installation date of each device are integrated with the model.

[0466] Step 5:

[0467] The server utilizes a generated AI model to propose optimal equipment placement and piping routes. The generated structural information model and related legal regulations and technical standards are used as input, and the AI ​​proposes multiple equipment placement options as output. A specific prompt might be, "Please provide a piping design to optimize the boiler room in this facility."

[0468] Step 6:

[0469] The terminal presents the user with proposals from the server. The user visually receives the proposed options as input and operates an interface to determine which plan to select as output. This allows the user to compare multiple options and choose the most suitable configuration.

[0470] Step 7:

[0471] A digital simulation experiment is conducted based on the user's most frequently selected layout. The selected layout is used as input, and the functionality and visual evaluation of that layout are verified through simulation as output.

[0472] Step 8:

[0473] The server generates detailed design drawings based on the ultimately selected layout. Using the selected layout as input, it completes design drawings containing all the information necessary for construction and renovation as output. These design drawings are generated by specific software and provided in an actionable format.

[0474] (Application Example 1)

[0475] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0476] In large-scale production facilities such as factories, the placement of equipment and piping, and the installation of production lines, often require considerable effort and time. Furthermore, if these arrangements are not optimal, they can negatively impact efficiency and safety. In addition, specialized knowledge and experience are necessary to effectively utilize the limited space within the facility while meeting legal regulations and technical standards for equipment placement. There is a need for solutions that can address these challenges quickly and efficiently.

[0477] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0478] In this invention, the server includes a three-dimensional scanning means for acquiring point cloud information, a means for analyzing the acquired point cloud information and generating a structural information model, and a means for automatically adding attribute information of fixtures based on the generated structural information model. This makes it possible to propose optimal fixture placement and piping routes, and to generate optimal layout plans for manufacturing lines within production facilities.

[0479] "Point cloud information" is a collection of numerous points arranged in three dimensions to represent the shape and position of an object in space.

[0480] A "three-dimensional scanning means" is a device or method for acquiring three-dimensional information of a target space or object.

[0481] A "structural information model" is a three-dimensional model that includes detailed information such as the shape, dimensions, and location within a facility, generated by analyzing acquired point cloud information.

[0482] "Equipment attribute information" refers to data that includes detailed information necessary for management, such as the equipment model, material, and year of introduction.

[0483] A "generative intelligence model" is a technology that uses artificial intelligence to propose the optimal layout and routes within a facility based on data obtained.

[0484] "Virtual simulation" is a method of using digital models to consider actual designs and layouts in advance.

[0485] "Final design" refers to a detailed, actionable design drawing or plan based on the proposed plan.

[0486] To implement this invention, various specialized hardware and software are used. First, a device is needed to perform three-dimensional scanning in cooperation with a server in order to optimize the arrangement of equipment and piping within the facility. Specifically, a three-dimensional scanning means is used to collect basic point cloud information. The point cloud information obtained through this scanning is detailed three-dimensional information of equipment within the facility and their positional relationships.

[0487] Next, the server receives and analyzes this point cloud information to generate a structural information model. This information model integrates detailed information such as the shape, position, and dimensions of all the fixtures, and a generative intelligence model on the server automatically adds attribute information for the equipment based on this. This generative intelligence model utilizes artificial intelligence technology to propose the optimal fixture placement and routes from the point cloud data.

[0488] Users can perform virtual simulations based on these proposals and select the most suitable plan from among the suggested options. This selected plan is then refined into a detailed design by the server and can be used to support the actual facility design.

[0489] A concrete example is optimizing the layout of production lines in a manufacturing facility. For instance, the server, based on acquired point cloud information, presents optimal layout proposals for existing piping and newly planned production lines, which the user then evaluates through virtual simulation. Finally, the design is finalized.

[0490] An example of a prompt for the generating AI model is: "Based on the following 3D point cloud data, generate an optimal equipment placement plan for the facility. We would like a feasible plan that takes safety standards into consideration." This prompt allows the generating AI model to provide a more precise and realistic placement plan.

[0491] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0492] Step 1:

[0493] The terminal moves around the facility and acquires point cloud information using a three-dimensional scanning device. This point cloud information includes three-dimensional data regarding the position and shape of equipment and structures within the facility. The terminal collects this information in real time and rescans as needed.

[0494] Step 2:

[0495] The terminal transmits the acquired point cloud information to the server. The server receives this data as input and performs point cloud analysis as data processing. As a result of the analysis, a detailed structural information model of the facility is generated. This model includes details such as the shape, location, and dimensions of the fixtures.

[0496] Step 3:

[0497] The server automatically adds attribute information to each fixture using the generated structural information model. This attribute information includes the fixture's model, material, and year of introduction. This increases the comprehensiveness of the data.

[0498] Step 4:

[0499] The server generates prompt messages using a generative intelligence model based on structural information models and attribute information. This generative intelligence model optimizes equipment placement and piping routes, generating proposals that are realistic and take into account legal regulations and technical standards.

[0500] Step 5:

[0501] The user reviews multiple proposed layouts via a terminal and runs a virtual simulation. The user then selects the optimal plan, considering aesthetics and functionality. This selection process deepens the user's interaction with the system and facilitates their decision-making.

[0502] Step 6:

[0503] The server outputs the final design based on the user's selection. This design drawing contains all the necessary information and can be used directly for actual facility design and renovation.

[0504] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0505] This invention is a system that combines an emotion engine with an equipment management system to dynamically adjust the user interface and improve the user experience. In this embodiment, the emotion engine analyzes the user's facial expressions and voice, and grasps their emotional state in real time. Next, based on the obtained emotional data, the system adjusts the layout of the user interface and the priority of the information presented to increase user satisfaction.

[0506] Specifically, the device acquires the user's camera images and voice input and sends them to the emotion engine. The emotion engine uses modern algorithms to analyze facial expressions and tone of voice to understand emotions such as joy, surprise, and stress. Once this information is gathered, the server uses the emotion data to rearrange the order in which the generation AI proposes equipment layouts, displaying the option that causes the least stress to the user at the top.

[0507] Furthermore, if a user is experiencing stress, the server can provide supplementary information and interactive help to alleviate their anxieties and questions. For example, if a user has doubts about a new equipment layout proposal, the emotion engine can detect this, and the server can immediately display a detailed explanation in a pop-up window. This allows the user to proceed with their decision with confidence.

[0508] Furthermore, when the terminal performs a digital simulation of the selected equipment layout, the emotion engine further personalizes the experience. It automatically highlights detailed views that the user has shown interest in during the simulation process, making the interaction more intuitive. In this way, the present invention utilizes the emotion engine to provide a flexible interface that responds to the user's emotional state, thereby achieving more comfortable equipment management.

[0509] The following describes the processing flow.

[0510] Step 1:

[0511] The device acquires the user's camera images and audio data. This data is collected in real time and used to accurately understand the user's emotional state.

[0512] Step 2:

[0513] The device sends collected image and audio data to the emotion engine. The emotion engine uses advanced algorithms to analyze the user's facial expressions and tone of voice, and evaluates emotions such as joy, surprise, and stress in real time.

[0514] Step 3:

[0515] The emotion engine sends the analysis results to the server. The server receives the emotion data and determines the user's current psychological state.

[0516] Step 4:

[0517] Based on the emotional data received by the server, the generating AI adjusts the order in which it presents multiple proposed facility layouts. For example, if the user is feeling stressed, the server will either narrow down the options or prioritize displaying the simplest and easiest-to-understand proposal.

[0518] Step 5:

[0519] The user reviews the suggested options via their device. The emotion engine continuously monitors the user's reactions, and if it detects signs of dissatisfaction, the server displays additional information as a pop-up to try and alleviate their concerns.

[0520] Step 6:

[0521] After the user selects a layout option that interests them, the device performs a digital simulation of that option. During the simulation, the emotion engine monitors the user's emotional state and makes dynamic adjustments, such as highlighting aspects of interest.

[0522] Step 7:

[0523] Based on the final design approved by the user, the server outputs a detailed blueprint. This blueprint contains all the information necessary for on-site implementation, supporting planned and efficient work.

[0524] (Example 2)

[0525] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0526] Conventional facility management systems have the problem of limiting the user experience because they provide a fixed user interface without considering the user's emotional state. Furthermore, when presenting various facility layout options, they fail to consider the impact of the selection order on the user's emotions, thus failing to support optimal decision-making.

[0527] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0528] In this invention, the server includes means for acquiring the user's visual and auditory data and performing emotion analysis; means for dynamically adjusting the content of the user interface based on the user's emotional state; and means for generating artificial intelligence to generate equipment layout plans considering regulations, technical standards, and compatibility, and adjusting the presentation order of the layout plans based on the results of the emotion analysis. This enables the provision of a flexible interface that responds to the user's emotional state and the presentation of optimal equipment layout plans.

[0529] "Point cloud data" is a collection of numerous points that indicate positions in three-dimensional space, and is digital data that represents the shape and surface features of an object.

[0530] A "three-dimensional scanning method" is a device or method for acquiring three-dimensional data of an object or environment, often using lasers or optical technology.

[0531] A "structural information model" is a digital model that describes the structure of a building or equipment in three dimensions, including shape, dimensions, and location information.

[0532] "Equipment attribute information" refers to information that represents the characteristics of equipment, such as its function, performance, and materials, and is data used for equipment management and analysis.

[0533] "Generative artificial intelligence means for proposing routes" refers to a method that utilizes artificial intelligence technology to automatically plan the optimal placement of multiple pieces of equipment, as well as the piping and routes connecting them.

[0534] "Digital simulation" is a technology that uses a computer to recreate virtual environments and scenarios for conducting experiments and tests.

[0535] "Emotion analysis" is a technology that analyzes a user's facial expressions and voice to identify their emotional state, often utilizing computer vision or speech recognition.

[0536] "Means for dynamically adjusting the content presented in the user interface" refers to methods that change the arrangement of information on the screen and the priority of displayed content in real time based on the user's emotional state.

[0537] "Supporting optimal decision-making" means presenting users with the most effective and least stressful options, and assisting them in making appropriate decisions.

[0538] This invention provides a mechanism for a facility management system that offers a dynamic user interface that takes into account the user's emotional state. The aim is to improve the user experience and support effective decision-making.

[0539] The entire system consists mainly of terminals, servers, an emotion analysis engine, and a generative AI model. The terminals are devices with high-performance cameras and voice recognition capabilities (e.g., smartphones or dedicated tablets). These capture the user's facial expressions and voice data in real time and send it to the server.

[0540] The server uses an emotion analysis engine to analyze the user's emotional state. Deep learning techniques are used to identify emotions such as "joy," "surprise," and "stress" from facial expressions. The resulting emotional data is sent to a generative AI model and used to adjust the order in which equipment layout proposals are presented.

[0541] A concrete example is improving the layout of office equipment. When a user views multiple equipment layout options on screen, if they experience stress, the emotion analysis engine detects this reaction. The server then considers this emotional data and displays the layout options that are more relaxing for the user at the top of the list.

[0542] In this process, the following example prompt will be used: "Use the sentiment engine to analyze how the user is reacting to the current equipment layout proposal, and based on the results, suggest the next layout proposal to display."

[0543] In this way, the combination of emotion analysis and a dynamic user interface provides users with a comfortable and effective facility management experience.

[0544] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0545] Step 1:

[0546] The device uses a high-performance camera and microphone to capture the user's facial expressions and voice in real time. Input consists of the user's image and voice data, which are captured by sensors. Output is the acquired raw data, which is then sent to a server.

[0547] Step 2:

[0548] The server passes the user's image and audio data received from the terminal to the emotion analysis engine. Here, the input is the image and audio data from the terminal. The emotion analysis engine uses a deep learning algorithm to analyze the data and identify the user's emotional state (e.g., joy, surprise, stress). The output is the analyzed emotion data.

[0549] Step 3:

[0550] The server sends a prompt message to the generative AI model based on the emotion data obtained through emotion analysis. This prompt message is used to present an optimized facility layout for the user, and is in the form of "Based on the user's current emotion, please suggest the next layout to display." The input is emotion data and the prompt message, and the output is the layout proposal returned by the generative AI model.

[0551] Step 4:

[0552] The server receives the proposed facility layout from the generated AI model and reflects it in the user interface. During this process, it adjusts the presentation order based on emotional data, prioritizing the display of the layout that causes the user the least stress. The input is the generated layout, and the output is the adjusted user interface display.

[0553] Step 5:

[0554] The user selects from the provided equipment layout options and runs a digital simulation. The terminal reproduces the user's selected layout in a virtual environment and visually displays the simulation results. Here, the input is the user's selected layout option, and the output is the simulated visual data.

[0555] Step 6:

[0556] The server performs further sentiment analysis based on the simulation results and provides additional information and interactive help as needed. This is intended to address any questions or frustrations the user may have. The input is simulation feedback and sentiment data, and the output is tailored help information.

[0557] (Application Example 2)

[0558] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0559] Traditional facility management systems often provided a fixed user interface without considering the emotional state of the user. As a result, they were unable to flexibly provide information according to the user's situation, making it difficult to deliver a highly satisfying user experience. Furthermore, there was a lack of mechanisms to immediately provide appropriate support for the anxieties and questions users faced when selecting facility layout options, leading to a decrease in the efficiency of decision-making.

[0560] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0561] In this invention, the server includes means for acquiring the user's facial expressions and voice to analyze their emotions and dynamically adjusting the user interface based on the analysis results; means for providing appropriate additional information based on the analysis results to support better decision-making; and means for performing a digital simulation of the equipment layout plan selected by the user. This makes it possible to optimize the interface according to the user's emotional state and provide personalized information and support in real time.

[0562] A "three-dimensional scanning device" is a device used to acquire point cloud data, and it is a technology that scans the three-dimensional shape of an object and records it as digital data.

[0563] A "building information model" is a digital model generated by analyzing acquired point cloud data, and it includes information such as the shape, structure, and physical characteristics of a building.

[0564] "Equipment attribute information" refers to information that indicates the characteristics of equipment installed inside or outside a building, and includes data on shape, dimensions, and location.

[0565] "Data generation means" refers to methods and algorithms that use equipment attribute information to propose the optimal equipment layout and piping routes.

[0566] A "user interface" refers to an interactive screen or means of operation used for communication between the user and the system.

[0567] "Emotion analysis" is a process that determines a user's emotional state from their facial expressions and voice, and is a method for acquiring user emotional data in real time.

[0568] "Digital simulation" is a technology that aims to visualize and verify plans by reproducing selected equipment layouts in a virtual environment.

[0569] "Means of providing additional information" refers to a mechanism that dynamically presents relevant information and explanations to support user decisions and alleviate anxieties and questions.

[0570] The present invention relates to a system for equipment management that dynamically adjusts the interface based on user emotions to provide a more comfortable operating environment. This system includes a three-dimensional scanning device, a user terminal, a server, and a generative AI model.

[0571] The user terminal functions as a device such as smart glasses, capturing the user's facial expressions and voice. The acquired data is sent to an emotion analysis algorithm via the camera and microphone on the terminal. Here, libraries such as OpenCV and TensorFlow in Python are used to analyze facial expressions and voice characteristics in real time and determine emotions such as joy, surprise, and stress.

[0572] The server dynamically adjusts the user interface based on data obtained through sentiment analysis. This enables optimal information display and interaction tailored to the user's emotional state. Furthermore, the server provides additional information as needed to support user decision-making. This process utilizes cloud services such as Google Cloud APIs.

[0573] For example, if sentiment analysis determines that a user is confused or has questions about a new equipment layout plan, the server can immediately display a detailed explanation as a pop-up on the user's device. This allows the user to proceed to the next step with confidence.

[0574] For example, one possible prompt command to be input into the AI ​​model is, "Suggest a conversational response when the customer smiles." In this way, a variety of countermeasures can be provided to improve the user experience.

[0575] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0576] Step 1:

[0577] The device captures the user's facial expressions and voice through its camera and microphone. It collects the user's facial image data and voice data as input. This data is converted into a format that can be processed in real time and then transmitted to the next step.

[0578] Step 2:

[0579] On the device, an emotion analysis algorithm is executed using the acquired facial image data and audio data. Using libraries such as Python's OpenCV and TensorFlow, facial expressions and voice tone are analyzed. This process outputs emotion data such as joy, surprise, and stress.

[0580] Step 3:

[0581] The terminal sends the analyzed emotion data to the server. The server receives the emotion data as input and uses it to identify the user's emotional state. Based on this, the server performs the following interface adjustments.

[0582] Step 4:

[0583] The server dynamically adjusts the user interface layout and the information presented based on the received sentiment data. Using a generative AI model, it prioritizes displaying information of the user's greatest interest, reducing stress (e.g., highlighting specific instructions). The output of this process is the adjusted interface data.

[0584] Step 5:

[0585] The server provides additional information to users if they have any questions or concerns. Based on the user's emotional state and related system data as input, it selects and generates appropriate supplementary information and displays it as a pop-up on the screen. This output serves as supplementary information and guidance.

[0586] Step 6:

[0587] The user makes the final decision using the interface and information provided by the server. Furthermore, they receive specific equipment layout options and perform digital simulations as needed. The system receives the selected equipment layout as input, performs a simulation, and outputs a visualization of the final layout.

[0588] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0589] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0590] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0591] [Fourth Embodiment]

[0592] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0593] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0594] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0595] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0596] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0597] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0598] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0599] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0600] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0601] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0602] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0603] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0604] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0605] This invention provides a system for rapid and efficient facility management. First, a terminal equipped with 3D scanning capabilities moves around the facility, collecting detailed point cloud data. This data is reviewed in real time, and rescanning is performed as needed. Next, the terminal transmits the collected point cloud data to a server.

[0606] The server analyzes the received point cloud data and generates a building information model that represents the detailed 3D structure of the facility. This model integrates the shape, dimensions, and location information of all the equipment within the facility. The server then further analyzes the model and automatically adds attribute information to each piece of equipment. This attribute information includes the equipment's model, material, and installation date, encompassing all the information necessary for equipment management.

[0607] Based on the generated building information model, the server uses a generation AI to propose the optimal placement of equipment and piping routes. This process considers legal regulations, technical standards, and equipment compatibility, resulting in the generation of multiple realistic and implementable proposals. These proposals are then presented to the user via a terminal.

[0608] The user operates a terminal and reviews multiple equipment layout proposals generated by the AI. They select the plan that best matches their vision and then perform a digital simulation based on that selection. The simulation allows them to consider the appearance and functionality of the equipment within the facility in advance.

[0609] Finally, the server generates detailed design drawings based on the selected design proposal. These design drawings contain all the necessary information and can be used directly for actual construction or renovation.

[0610] Thus, by using the system of the present invention, facility managers can formulate quick and accurate renovation plans and implement planned and efficient facility management. A specific example of its use is scanning the boiler room of an aging factory and proposing an optimal piping configuration to improve the maintainability of the piping.

[0611] The following describes the processing flow.

[0612] Step 1:

[0613] The terminal moves around the facility, performing 3D scanning and acquiring point cloud data. The terminal monitors the data in real time and rescans any areas where the imaging is insufficient.

[0614] Step 2:

[0615] The terminal collects point cloud data and transfers it to the server using data compression technology. The compressed data is transmitted at high speed with minimal latency.

[0616] Step 3:

[0617] The server analyzes the received point cloud data and generates a building information model that shows the three-dimensional structure of the entire facility. The server uses detailed algorithms to create a highly accurate model down to the smallest detail.

[0618] Step 4:

[0619] The server analyzes the building information model, identifies and automatically adds the dimensions and attribute information of each piece of equipment. It then compares this information with the existing database to fill in any missing details.

[0620] Step 5:

[0621] The server operates the generating artificial intelligence to propose the optimal placement of equipment and piping routes. The generating AI generates multiple design options, each of which is checked for compliance with regulations, technical standards, and compatibility between equipment.

[0622] Step 6:

[0623] The user operates the device and reviews multiple proposals presented by the server. The user compares each proposal and selects the one that best meets their requirements.

[0624] Step 7:

[0625] The terminal performs a digital simulation based on the user's selection. The simulation results are visualized in three dimensions, allowing the user to check the equipment layout on the screen.

[0626] Step 8:

[0627] The server finalizes the user's selected design and generates detailed design drawings and supporting documents. The drawings contain all the information necessary for the installation or modification of the equipment.

[0628] (Example 1)

[0629] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0630] In facility management, accurately understanding the facility's structure and efficiently and appropriately arranging equipment is crucial. However, conventional methods involve time-consuming data collection and analysis, making it difficult to quickly provide highly accurate recommendations. Furthermore, creating layout plans that take legal regulations and technical standards into account is not easy, hindering operational efficiency.

[0631] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0632] In this invention, the server includes a three-dimensional scanning element for acquiring point cloud data, an element for analyzing the acquired point cloud data and generating a structural information model, an element for automatically adding equipment attribute information based on the generated structural information model, and a generation machine learning element that uses the equipment attribute information to propose the optimal equipment layout and piping route. This enables rapid and highly accurate equipment management and the proposal of efficient layout plans.

[0633] "Point cloud data" is data consisting of countless points in space, used to represent the shape of objects or facilities.

[0634] A "three-dimensional scanning element" is a device or technology for acquiring the three-dimensional shape of an object with high precision.

[0635] A "structural information model" is a model that represents the detailed three-dimensional structure of a facility as digital data, including the position, dimensions, and shape of each element.

[0636] "Attribute information" refers to information that includes specific characteristics of devices and equipment, such as model number, material, and introduction date.

[0637] "Generative machine learning elements" are artificial intelligence technologies used to propose optimal equipment placement and piping routes based on data input.

[0638] "Digital simulation" is a simulation technology used to verify proposed equipment placement and piping route designs in a virtual environment.

[0639] "Final design" refers to design drawings and data that show the final determined details of the structure and equipment layout.

[0640] This invention is a system that integrates different technical elements for managing and arranging equipment within a facility. Its embodiments are described below.

[0641] The terminal first moves around the facility using a 3D scanner to collect detailed point cloud data. The collected point cloud data is acquired through laser measurement and camera photography and displayed visually on the terminal's screen in real time. The terminal is also equipped with a function to detect missing parts of the scanned data and rescan.

[0642] Subsequently, the terminal transmits this point cloud data to the server via a communication network. The server analyzes the point cloud data using dedicated analysis software (e.g., architectural CAD software) and generates a structural information model with a detailed three-dimensional structure of the facility. This structural information model integrates detailed shape, dimensions, and positional information for each device.

[0643] The server then adds attribute information for each device to the generated structural information model. A database system (e.g., a relational database) is used for this, and information such as the model number, material, and introduction date for each device is automatically added.

[0644] Next, the server uses a generative AI model to propose the optimal placement of equipment and piping routes. The generative AI model is pre-trained, taking into account legal regulations, technical standards, and equipment compatibility, and generates multiple realistic placement options based on this. An example of a prompt is, "Provide a piping design to optimize the boiler room of this facility."

[0645] These proposals are presented to the user on the device, allowing them to compare each proposal and choose the most suitable one. The selected configuration is simulated through a digital simulation experiment, allowing the user to check the visual arrangement and functionality of the equipment in advance.

[0646] Ultimately, based on the user's selection, the server generates detailed design drawings. These drawings completely include all the information necessary for the actual construction and renovation work.

[0647] One concrete example is the use of this system to scan the boiler room of an aging factory, propose an optimal piping configuration using a generated AI model, and improve maintainability. This system enables rapid and efficient equipment management and optimization.

[0648] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0649] Step 1:

[0650] The terminal uses a 3D scanner to collect point cloud data within the facility. The point cloud data is acquired using the scanner's laser ranging and camera vision elements. Physical shape information within the facility is used as input, and a series of point cloud data is generated as output, displayed in real time on the terminal's screen. The user can rescan as needed to fill in any missing data.

[0651] Step 2:

[0652] The terminal sends the collected point cloud data to the server. Data transmission takes place over a communication network, and the terminal encodes the point cloud data obtained as output and provides it to the server as input. At this time, error checking is performed to maintain the integrity and completeness of the data.

[0653] Step 3:

[0654] The server analyzes the received point cloud data and generates a structural information model. Architectural CAD software is used for the analysis, processing the point cloud information as input data and generating a three-dimensional structural information model as output. This model integrates the shape, position, and dimensions of each device.

[0655] Step 4:

[0656] The system automatically adds device attribute information to the structural information model generated by the server. This process uses information from a relational database as input and adds attribute information to the model as output. As a result, the model type, material, and installation date of each device are integrated with the model.

[0657] Step 5:

[0658] The server utilizes a generated AI model to propose optimal equipment placement and piping routes. The generated structural information model and related legal regulations and technical standards are used as input, and the AI ​​proposes multiple equipment placement options as output. A specific prompt might be, "Please provide a piping design to optimize the boiler room in this facility."

[0659] Step 6:

[0660] The terminal presents the user with proposals from the server. The user visually receives the proposed options as input and operates an interface to determine which plan to select as output. This allows the user to compare multiple options and choose the most suitable configuration.

[0661] Step 7:

[0662] A digital simulation experiment is conducted based on the user's most frequently selected layout. The selected layout is used as input, and the functionality and visual evaluation of that layout are verified through simulation as output.

[0663] Step 8:

[0664] The server generates detailed design drawings based on the ultimately selected layout. Using the selected layout as input, it completes design drawings containing all the information necessary for construction and renovation as output. These design drawings are generated by specific software and provided in an actionable format.

[0665] (Application Example 1)

[0666] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0667] In large-scale production facilities such as factories, the placement of equipment and piping, and the installation of production lines, often require considerable effort and time. Furthermore, if these arrangements are not optimal, they can negatively impact efficiency and safety. In addition, specialized knowledge and experience are necessary to effectively utilize the limited space within the facility while meeting legal regulations and technical standards for equipment placement. There is a need for solutions that can address these challenges quickly and efficiently.

[0668] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0669] In this invention, the server includes a three-dimensional scanning means for acquiring point cloud information, a means for analyzing the acquired point cloud information and generating a structural information model, and a means for automatically adding attribute information of fixtures based on the generated structural information model. This makes it possible to propose optimal fixture placement and piping routes, and to generate optimal layout plans for manufacturing lines within production facilities.

[0670] "Point cloud information" is a collection of numerous points arranged in three dimensions to represent the shape and position of an object in space.

[0671] A "three-dimensional scanning means" is a device or method for acquiring three-dimensional information of a target space or object.

[0672] A "structural information model" is a three-dimensional model that includes detailed information such as the shape, dimensions, and location within a facility, generated by analyzing acquired point cloud information.

[0673] "Equipment attribute information" refers to data that includes detailed information necessary for management, such as the equipment model, material, and year of introduction.

[0674] A "generative intelligence model" is a technology that uses artificial intelligence to propose the optimal layout and routes within a facility based on data obtained.

[0675] "Virtual simulation" is a method of using digital models to consider actual designs and layouts in advance.

[0676] "Final design" refers to a detailed, actionable design drawing or plan based on the proposed plan.

[0677] To implement this invention, various specialized hardware and software are used. First, a device is needed to perform three-dimensional scanning in cooperation with a server in order to optimize the arrangement of equipment and piping within the facility. Specifically, a three-dimensional scanning means is used to collect basic point cloud information. The point cloud information obtained through this scanning is detailed three-dimensional information of equipment within the facility and their positional relationships.

[0678] Next, the server receives and analyzes this point cloud information to generate a structural information model. This information model integrates detailed information such as the shape, position, and dimensions of all the fixtures, and a generative intelligence model on the server automatically adds attribute information for the equipment based on this. This generative intelligence model utilizes artificial intelligence technology to propose the optimal fixture placement and routes from the point cloud data.

[0679] Users can perform virtual simulations based on these proposals and select the most suitable plan from among the suggested options. This selected plan is then refined into a detailed design by the server and can be used to support the actual facility design.

[0680] A concrete example is optimizing the layout of production lines in a manufacturing facility. For instance, the server, based on acquired point cloud information, presents optimal layout proposals for existing piping and newly planned production lines, which the user then evaluates through virtual simulation. Finally, the design is finalized.

[0681] An example of a prompt for the generating AI model is: "Based on the following 3D point cloud data, generate an optimal equipment placement plan for the facility. We would like a feasible plan that takes safety standards into consideration." This prompt allows the generating AI model to provide a more precise and realistic placement plan.

[0682] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0683] Step 1:

[0684] The terminal moves around the facility and acquires point cloud information using a three-dimensional scanning device. This point cloud information includes three-dimensional data regarding the position and shape of equipment and structures within the facility. The terminal collects this information in real time and rescans as needed.

[0685] Step 2:

[0686] The terminal transmits the acquired point cloud information to the server. The server receives this data as input and performs point cloud analysis as data processing. As a result of the analysis, a detailed structural information model of the facility is generated. This model includes details such as the shape, location, and dimensions of the fixtures.

[0687] Step 3:

[0688] The server automatically adds attribute information to each fixture using the generated structural information model. This attribute information includes the fixture's model, material, and year of introduction. This increases the comprehensiveness of the data.

[0689] Step 4:

[0690] The server generates prompt messages using a generative intelligence model based on structural information models and attribute information. This generative intelligence model optimizes equipment placement and piping routes, generating proposals that are realistic and take into account legal regulations and technical standards.

[0691] Step 5:

[0692] The user reviews multiple proposed layouts via a terminal and runs a virtual simulation. The user then selects the optimal plan, considering aesthetics and functionality. This selection process deepens the user's interaction with the system and facilitates their decision-making.

[0693] Step 6:

[0694] The server outputs the final design based on the user's selection. This design drawing contains all the necessary information and can be used directly for actual facility design and renovation.

[0695] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0696] This invention is a system that combines an emotion engine with an equipment management system to dynamically adjust the user interface and improve the user experience. In this embodiment, the emotion engine analyzes the user's facial expressions and voice, and grasps their emotional state in real time. Next, based on the obtained emotional data, the system adjusts the layout of the user interface and the priority of the information presented to increase user satisfaction.

[0697] Specifically, the device acquires the user's camera images and voice input and sends them to the emotion engine. The emotion engine uses modern algorithms to analyze facial expressions and tone of voice to understand emotions such as joy, surprise, and stress. Once this information is gathered, the server uses the emotion data to rearrange the order in which the generation AI proposes equipment layouts, displaying the option that causes the least stress to the user at the top.

[0698] Furthermore, if a user is experiencing stress, the server can provide supplementary information and interactive help to alleviate their anxieties and questions. For example, if a user has doubts about a new equipment layout proposal, the emotion engine can detect this, and the server can immediately display a detailed explanation in a pop-up window. This allows the user to proceed with their decision with confidence.

[0699] Furthermore, when the terminal performs a digital simulation of the selected equipment layout, the emotion engine further personalizes the experience. It automatically highlights detailed views that the user has shown interest in during the simulation process, making the interaction more intuitive. In this way, the present invention utilizes the emotion engine to provide a flexible interface that responds to the user's emotional state, thereby achieving more comfortable equipment management.

[0700] The following describes the processing flow.

[0701] Step 1:

[0702] The device acquires the user's camera images and audio data. This data is collected in real time and used to accurately understand the user's emotional state.

[0703] Step 2:

[0704] The device sends collected image and audio data to the emotion engine. The emotion engine uses advanced algorithms to analyze the user's facial expressions and tone of voice, and evaluates emotions such as joy, surprise, and stress in real time.

[0705] Step 3:

[0706] The emotion engine sends the analysis results to the server. The server receives the emotion data and determines the user's current psychological state.

[0707] Step 4:

[0708] Based on the emotional data received by the server, the generating AI adjusts the order in which it presents multiple proposed facility layouts. For example, if the user is feeling stressed, the server will either narrow down the options or prioritize displaying the simplest and easiest-to-understand proposal.

[0709] Step 5:

[0710] The user reviews the suggested options via their device. The emotion engine continuously monitors the user's reactions, and if it detects signs of dissatisfaction, the server displays additional information as a pop-up to try and alleviate their concerns.

[0711] Step 6:

[0712] After the user selects a layout option that interests them, the device performs a digital simulation of that option. During the simulation, the emotion engine monitors the user's emotional state and makes dynamic adjustments, such as highlighting aspects of interest.

[0713] Step 7:

[0714] Based on the final design approved by the user, the server outputs a detailed blueprint. This blueprint contains all the information necessary for on-site implementation, supporting planned and efficient work.

[0715] (Example 2)

[0716] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0717] Conventional facility management systems have the problem of limiting the user experience because they provide a fixed user interface without considering the user's emotional state. Furthermore, when presenting various facility layout options, they fail to consider the impact of the selection order on the user's emotions, thus failing to support optimal decision-making.

[0718] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0719] In this invention, the server includes means for acquiring the user's visual and auditory data and performing emotion analysis; means for dynamically adjusting the content of the user interface based on the user's emotional state; and means for generating artificial intelligence to generate equipment layout plans considering regulations, technical standards, and compatibility, and adjusting the presentation order of the layout plans based on the results of the emotion analysis. This enables the provision of a flexible interface that responds to the user's emotional state and the presentation of optimal equipment layout plans.

[0720] "Point cloud data" is a collection of numerous points that indicate positions in three-dimensional space, and is digital data that represents the shape and surface features of an object.

[0721] A "three-dimensional scanning method" is a device or method for acquiring three-dimensional data of an object or environment, often using lasers or optical technology.

[0722] A "structural information model" is a digital model that describes the structure of a building or equipment in three dimensions, including shape, dimensions, and location information.

[0723] "Equipment attribute information" refers to information that represents the characteristics of equipment, such as its function, performance, and materials, and is data used for equipment management and analysis.

[0724] "Generative artificial intelligence means for proposing routes" refers to a method that utilizes artificial intelligence technology to automatically plan the optimal placement of multiple pieces of equipment, as well as the piping and routes connecting them.

[0725] "Digital simulation" is a technology that uses a computer to recreate virtual environments and scenarios for conducting experiments and tests.

[0726] "Emotion analysis" is a technology that analyzes a user's facial expressions and voice to identify their emotional state, often utilizing computer vision or speech recognition.

[0727] "Means for dynamically adjusting the content presented in the user interface" refers to methods that change the arrangement of information on the screen and the priority of displayed content in real time based on the user's emotional state.

[0728] "Supporting optimal decision-making" means presenting users with the most effective and least stressful options, and assisting them in making appropriate decisions.

[0729] This invention provides a mechanism for a facility management system that offers a dynamic user interface that takes into account the user's emotional state. The aim is to improve the user experience and support effective decision-making.

[0730] The entire system consists mainly of terminals, servers, an emotion analysis engine, and a generative AI model. The terminals are devices with high-performance cameras and voice recognition capabilities (e.g., smartphones or dedicated tablets). These capture the user's facial expressions and voice data in real time and send it to the server.

[0731] The server uses an emotion analysis engine to analyze the user's emotional state. Deep learning techniques are used to identify emotions such as "joy," "surprise," and "stress" from facial expressions. The resulting emotional data is sent to a generative AI model and used to adjust the order in which equipment layout proposals are presented.

[0732] A concrete example is improving the layout of office equipment. When a user views multiple equipment layout options on screen, if they experience stress, the emotion analysis engine detects this reaction. The server then considers this emotional data and displays the layout options that are more relaxing for the user at the top of the list.

[0733] In this process, the following example prompt will be used: "Use the sentiment engine to analyze how the user is reacting to the current equipment layout proposal, and based on the results, suggest the next layout proposal to display."

[0734] In this way, the combination of emotion analysis and a dynamic user interface provides users with a comfortable and effective facility management experience.

[0735] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0736] Step 1:

[0737] The device uses a high-performance camera and microphone to capture the user's facial expressions and voice in real time. Input consists of the user's image and voice data, which are captured by sensors. Output is the acquired raw data, which is then sent to a server.

[0738] Step 2:

[0739] The server passes the user's image and audio data received from the terminal to the emotion analysis engine. Here, the input is the image and audio data from the terminal. The emotion analysis engine uses a deep learning algorithm to analyze the data and identify the user's emotional state (e.g., joy, surprise, stress). The output is the analyzed emotion data.

[0740] Step 3:

[0741] The server sends a prompt message to the generative AI model based on the emotion data obtained through emotion analysis. This prompt message is used to present an optimized facility layout for the user, and is in the form of "Based on the user's current emotion, please suggest the next layout to display." The input is emotion data and the prompt message, and the output is the layout proposal returned by the generative AI model.

[0742] Step 4:

[0743] The server receives the proposed facility layout from the generated AI model and reflects it in the user interface. During this process, it adjusts the presentation order based on emotional data, prioritizing the display of the layout that causes the user the least stress. The input is the generated layout, and the output is the adjusted user interface display.

[0744] Step 5:

[0745] The user selects from the provided equipment layout options and runs a digital simulation. The terminal reproduces the user's selected layout in a virtual environment and visually displays the simulation results. Here, the input is the user's selected layout option, and the output is the simulated visual data.

[0746] Step 6:

[0747] The server performs further sentiment analysis based on the simulation results and provides additional information and interactive help as needed. This is intended to address any questions or frustrations the user may have. The input is simulation feedback and sentiment data, and the output is tailored help information.

[0748] (Application Example 2)

[0749] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0750] Traditional facility management systems often provided a fixed user interface without considering the emotional state of the user. As a result, they were unable to flexibly provide information according to the user's situation, making it difficult to deliver a highly satisfying user experience. Furthermore, there was a lack of mechanisms to immediately provide appropriate support for the anxieties and questions users faced when selecting facility layout options, leading to a decrease in the efficiency of decision-making.

[0751] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0752] In this invention, the server includes means for acquiring the user's facial expressions and voice to analyze their emotions and dynamically adjusting the user interface based on the analysis results; means for providing appropriate additional information based on the analysis results to support better decision-making; and means for performing a digital simulation of the equipment layout plan selected by the user. This makes it possible to optimize the interface according to the user's emotional state and provide personalized information and support in real time.

[0753] A "three-dimensional scanning device" is a device used to acquire point cloud data, and it is a technology that scans the three-dimensional shape of an object and records it as digital data.

[0754] A "building information model" is a digital model generated by analyzing acquired point cloud data, and it includes information such as the shape, structure, and physical characteristics of a building.

[0755] "Equipment attribute information" refers to information that indicates the characteristics of equipment installed inside or outside a building, and includes data on shape, dimensions, and location.

[0756] "Data generation means" refers to methods and algorithms that use equipment attribute information to propose the optimal equipment layout and piping routes.

[0757] A "user interface" refers to an interactive screen or means of operation used for communication between the user and the system.

[0758] "Emotion analysis" is a process that determines a user's emotional state from their facial expressions and voice, and is a method for acquiring user emotional data in real time.

[0759] "Digital simulation" is a technology that aims to visualize and verify plans by reproducing selected equipment layouts in a virtual environment.

[0760] "Means of providing additional information" refers to a mechanism that dynamically presents relevant information and explanations to support user decisions and alleviate anxieties and questions.

[0761] The present invention relates to a system for equipment management that dynamically adjusts the interface based on user emotions to provide a more comfortable operating environment. This system includes a three-dimensional scanning device, a user terminal, a server, and a generative AI model.

[0762] The user terminal functions as a device such as smart glasses, capturing the user's facial expressions and voice. The acquired data is sent to an emotion analysis algorithm via the camera and microphone on the terminal. Here, libraries such as OpenCV and TensorFlow in Python are used to analyze facial expressions and voice characteristics in real time and determine emotions such as joy, surprise, and stress.

[0763] The server dynamically adjusts the user interface based on data obtained through sentiment analysis. This enables optimal information display and interaction tailored to the user's emotional state. Furthermore, the server provides additional information as needed to support user decision-making. This process utilizes cloud services such as Google Cloud APIs.

[0764] For example, if sentiment analysis determines that a user is confused or has questions about a new equipment layout plan, the server can immediately display a detailed explanation as a pop-up on the user's device. This allows the user to proceed to the next step with confidence.

[0765] For example, one possible prompt command to be input into the AI ​​model is, "Suggest a conversational response when the customer smiles." In this way, a variety of countermeasures can be provided to improve the user experience.

[0766] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0767] Step 1:

[0768] The device captures the user's facial expressions and voice through its camera and microphone. It collects the user's facial image data and voice data as input. This data is converted into a format that can be processed in real time and then transmitted to the next step.

[0769] Step 2:

[0770] On the device, an emotion analysis algorithm is executed using the acquired facial image data and audio data. Using libraries such as Python's OpenCV and TensorFlow, facial expressions and voice tone are analyzed. This process outputs emotion data such as joy, surprise, and stress.

[0771] Step 3:

[0772] The terminal sends the analyzed emotion data to the server. The server receives the emotion data as input and uses it to identify the user's emotional state. Based on this, the server performs the following interface adjustments.

[0773] Step 4:

[0774] The server dynamically adjusts the user interface layout and the information presented based on the received sentiment data. Using a generative AI model, it prioritizes displaying information of the user's greatest interest, reducing stress (e.g., highlighting specific instructions). The output of this process is the adjusted interface data.

[0775] Step 5:

[0776] The server provides additional information to users if they have any questions or concerns. Based on the user's emotional state and related system data as input, it selects and generates appropriate supplementary information and displays it as a pop-up on the screen. This output serves as supplementary information and guidance.

[0777] Step 6:

[0778] The user makes the final decision using the interface and information provided by the server. Furthermore, they receive specific equipment layout options and perform digital simulations as needed. The system receives the selected equipment layout as input, performs a simulation, and outputs a visualization of the final layout.

[0779] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0780] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0781] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0782] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0783] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0784] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0785] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0786] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0787] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0788] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0789] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0790] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0791] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0792] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0793] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0794] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0795] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0796] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0797] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0798] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0799] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0800] The following is further disclosed regarding the embodiments described above.

[0801] (Claim 1)

[0802] A 3D scanning method for acquiring point cloud data,

[0803] A means for analyzing acquired point cloud data and generating a building information model,

[0804] A means for automatically adding equipment attribute information based on the generated building information model,

[0805] A generative artificial intelligence means that proposes the optimal equipment layout and piping route using equipment attribute information,

[0806] A means for the user to select from several proposed equipment layout options and perform a digital simulation,

[0807] A means of outputting the final design,

[0808] A system that includes this.

[0809] (Claim 2)

[0810] The system according to claim 1, comprising means for integrating the shape, dimensions, and location information of equipment in the generation of a building information model.

[0811] (Claim 3)

[0812] The system according to claim 1, wherein the generating artificial intelligence means generates a proposed equipment layout taking into account legal regulations, technical standards, and compatibility.

[0813] "Example 1"

[0814] (Claim 1)

[0815] Three-dimensional scanning elements for acquiring point cloud data,

[0816] The elements for analyzing acquired point cloud data and generating a structural information model,

[0817] Based on the generated structural information model, an element automatically adds attribute information for the device,

[0818] A generative machine learning element that uses the attribute information of the equipment to propose the optimal equipment placement and piping route,

[0819] The user selects from several proposed device layouts and conducts a digital simulation experiment, and

[0820] Elements that output the final design,

[0821] A system that includes this.

[0822] (Claim 2)

[0823] The system according to claim 1, comprising an element for integrating the shape, dimensions, and positional information of the device in the generation of a structural information model.

[0824] (Claim 3)

[0825] The system according to claim 1, wherein a generative machine learning element generates a proposed device layout taking into account rules, technical standards, and compatibility.

[0826] "Application Example 1"

[0827] (Claim 1)

[0828] A three-dimensional scanning means for acquiring point cloud information,

[0829] A means for analyzing acquired point cloud information and generating a structural information model,

[0830] A means for automatically adding attribute information of the device based on the generated structural information model,

[0831] A generative intelligent model means that proposes the optimal equipment placement and piping route using equipment attribute information,

[0832] A means by which the user selects from several proposed equipment layout options and performs a virtual simulation,

[0833] A means of outputting the final design,

[0834] A means of proposing an optimal manufacturing line layout within a production facility using a generated structural information model and a generative intelligence model,

[0835] A system that includes this.

[0836] (Claim 2)

[0837] The system according to claim 1, comprising means for integrating shape, dimensions, and positional information of an apparatus in the generation of a structural information model.

[0838] (Claim 3)

[0839] The system according to claim 1, wherein the generative intelligent model means generates a proposed fixture arrangement taking into account rules, technical standards, and compatibility.

[0840] "Example 2 of combining an emotion engine"

[0841] (Claim 1)

[0842] A three-dimensional scanning method for acquiring point cloud data,

[0843] A means for analyzing acquired point cloud data and generating a structural information model,

[0844] A means for automatically adding equipment attribute information based on the generated structural information model,

[0845] A generative artificial intelligence means that proposes the optimal equipment layout and piping route using equipment attribute information,

[0846] A means by which users can select from multiple proposed equipment layouts and perform digital simulations,

[0847] A means for acquiring user visual and audio data and performing emotion analysis,

[0848] A means for dynamically adjusting the content presented in the user interface based on the user's emotional state,

[0849] A means of outputting the final design,

[0850] A device that includes this.

[0851] (Claim 2)

[0852] The apparatus according to claim 1, comprising means for integrating the shape, dimensions, and location information of equipment in the generation of a structural information model.

[0853] (Claim 3)

[0854] The apparatus according to claim 1, characterized in that the generating artificial intelligence means generates equipment layout proposals taking into account regulations, technical standards and compatibility, and adjusts the order in which the layout proposals are presented based on the results of sentiment analysis.

[0855] "Application example 2 when combining with an emotional engine"

[0856] (Claim 1)

[0857] A three-dimensional scanning method for acquiring point cloud data,

[0858] A means for analyzing acquired point cloud data and generating a building information model,

[0859] A means for automatically adding equipment attribute information based on the generated building information model,

[0860] A data generation method that uses equipment attribute information to propose the optimal equipment layout and piping route,

[0861] A means by which users can select from multiple proposed equipment layouts and perform digital simulations,

[0862] A means for acquiring the user's facial expressions and voice, analyzing their emotions, and dynamically adjusting the user interface based on the analysis results,

[0863] A means to provide appropriate additional information based on the analysis results and support better decision-making,

[0864] A means of outputting the final design,

[0865] A system that includes this.

[0866] (Claim 2)

[0867] The system according to claim 1, comprising means for integrating the shape, dimensions, and location information of equipment in the generation of a building information model.

[0868] (Claim 3)

[0869] The system according to claim 1, wherein the data generation means generates a proposed equipment layout taking into account legal regulations, technical standards, and compatibility. [Explanation of Symbols]

[0870] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A 3D scanning method for acquiring point cloud data, A means for analyzing acquired point cloud data and generating a building information model, A means for automatically adding equipment attribute information based on the generated building information model, A generative artificial intelligence means that proposes the optimal equipment layout and piping route using equipment attribute information, A means for the user to select from several proposed equipment layout options and perform a digital simulation, A means of outputting the final design, A system that includes this.

2. The system according to claim 1, comprising means for integrating the shape, dimensions, and location information of equipment in the generation of a building information model.

3. The system according to claim 1, wherein the generating artificial intelligence means generates a proposed equipment layout taking into account legal regulations, technical standards, and compatibility.

Images