Integrated control system and integrated control method

Abstract

It enables integrated control of smart homes through natural conversations that are easy on the user. [Solution] The integrated control system 100 includes a controller 100 that receives user instruction information and an LLM 200 that executes processing according to the instruction information. If the content of the instruction information cannot be handled by the LLM 200 alone, the LLM 200 transmits service information corresponding to the instruction information from the pre-registered service information for using the service equipment 400 to the controller 100. When the controller 100 receives the service information, it communicates with the service equipment 400 based on it and executes processing to use the service.

Description

Technical Field

[0001] The present invention relates to an integrated control system and an integrated control method, and relates to a technology that enables integrated control of a smart home through natural conversation that is difficult to impose a burden on the user.

Background Art

[0002] With the evolution of so-called IoT (Internet of Things) technology, various things, not only various industrial devices but also home appliances and security devices in ordinary households, are becoming controllable via a network. Many home appliances and other IoT-compatible devices are often controlled via a smart speaker or a smart phone. A smart speaker or the like that receives an input such as a user's utterance communicates with a WEB service of a home appliance manufacturer or the like via, for example, AI (Artificial Intelligence) that performs speech recognition, and responds to the user with functions such as operation of a target home appliance, search for weather forecast, and the like. As a conventional technology related to so-called smart home and home automation as described above, a technology (see Patent Document 1) that provides shortcut information including a control command for controlling a home appliance has been proposed.

[0003] This technology relates to a communication system including: a home appliance; a communication unit that receives a request; an analysis unit that analyzes the request received by the communication unit based on a Web API; a control unit that controls the home appliance according to the control command when the analysis unit analyzes that the identifier of the home appliance and the control command defined in the Web API are described in the request received by the communication unit; and a shortcut providing unit that transmits, to a terminal capable of transmitting a request, shortcut information for executing a request in which the identifier of the home appliance and the control command of the home appliance are described and displayed on the display unit of the terminal, to the terminal.

[0004] Furthermore, a technology has been proposed (see Patent Document 2) that realizes full-fledged intelligent home automation, allowing users to freely operate various home appliances and receive various services from those products, making the interaction between humans and home appliances as natural, smooth, and harmonious as human-to-human interaction, and making these two types of home automation products indispensable as members of the family.

[0005] This technology relates to a home automation product (central controller (housekeeper)) comprising a processing unit, a memory unit, a communication unit and a power supply unit, wherein the communication unit includes WiFi, each unit is connected to the processing unit and operates in cooperation with the product described in claim 8, corresponds to an artificial intelligence server and / or a central nervous system, and operates in cooperation with other products using the format room + category name + product (or category name + room + product).

[0006] Furthermore, a technology has been proposed (see Patent Document 3) that provides an optimal control method for smart homes, which optimizes the operational plan derived from multiple functions according to the user's objectives.

[0007] This technology relates to an optimal control method for a smart house in which a plurality of power sources consisting of commercial power sources and distributed power sources are connected to a load via a power conditioner, characterized in that a computing device is made to perform the following steps: calculate item values ​​for a plurality of objective functions corresponding to individual items to be optimized, based on the actual energy values ​​of the distributed power sources stored in advance in the memory of the computing device; calculate a state evaluation function for each of the calculated item values ​​and calculate a state evaluation function value; determine the optimal value for the combination of item values ​​of the plurality of objective functions that takes the value closest to the optimal solution of the state evaluation function value stored in advance in the memory; determine the output pattern of the power output to the load for each power source based on the combination of item values ​​determined to be the optimal value; and control the power conditioner based on the output pattern. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2016-143100 [Patent Document 2] Special Publication No. 2019-509645 [Patent Document 3] Japanese Patent Publication No. 2015-65740 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] Controlling home appliances via smartphones or smart speakers, as described above, is certainly convenient and offers high levels of convenience, as it allows operation from anywhere, provided there is a network environment (or short-range wireless communication such as infrared). However, when there are multiple devices to control, the user must select and launch the appropriate application for each device (e.g., an app for controlling air conditioners, an app for controlling lights, a weather forecast app, etc.) and perform the necessary input (e.g., voice commands) according to the specifications of each application. On the other hand, businesses that provide such operating environments (smartphone and smart speaker vendors, service providers, etc.) have had to prepare dedicated applications or programs for each potentially controllable device and implement or provide them on smart speakers and smartphones.

[0010] From a different perspective, the above situation means that devices such as smartphones and smart speakers that do not have the dedicated applications mentioned above implemented cannot be controlled at all. On the other hand, a large-scale language model that has comprehensively learned various information on the network is expected to be able to handle such situations, but its learning target is past publicly available information and it cannot respond based on non-public information or the latest information. In other words, even a large-scale language model cannot respond to the latest information such as weather forecasts, or non-public information such as protocols for controlling home appliances and authentication information.

[0011] Therefore, the present invention has been made in view of the above problems, and its objective is to provide a technology that enables integrated control of a smart house through natural conversation that does not burden the user. [Means for solving the problem]

[0012] The above problems are solved by an integrated control system comprising: a controller that receives user instruction information; and a large-scale language model that analyzes the instruction information and executes processing according to the instruction information, wherein if the content of the instruction information cannot be handled by the large-scale language model alone, the large-scale language model transmits service information corresponding to the instruction information from pre-registered service information for using service equipment to the controller; and when the controller receives the service information from the large-scale language model, it communicates with the service equipment based on the service information and executes processing for using the service equipment.

[0013] According to the integrated control system of the present invention configured as described above, integrated control of a smart house becomes possible through natural conversation that does not burden the user.

[0014] Furthermore, in the above-described integrated control system, if the content of the instruction information can be handled solely by the large-scale language model, it is preferable that the large-scale language model transmits the response data it generates in response to the instruction information to the controller.

[0015] The above configuration enables natural and smoother responses using a large-scale language model. Ultimately, this allows for more integrated control of smart homes through natural conversations that are less burdensome for the user.

[0016] Furthermore, in the above-described integrated control system, it is preferable that the service information is information for controlling in-home devices.

[0017] The above configuration enables operation (control) of so-called IoT-enabled home appliances. Ultimately, it allows for more integrated control of smart homes through natural conversation that is less burdensome for the user.

[0018] Furthermore, in the above-described integrated control system, it is preferable that the service information is information for controlling the web service.

[0019] The above configuration enables the handling of search operations related to real-time web-based information such as weather forecasts, stock prices, and exchange rates. Ultimately, it allows for more integrated control of smart homes through natural conversations that are less burdensome for the user.

[0020] Furthermore, in the above-described integrated control system, it is preferable that the large-scale language model acquires service information for each of the service devices, including at least command information among the authentication information and commands necessary for using the service device, from a predetermined device in advance and registers it in a storage device, and if the content of the instruction information cannot be handled by the large-scale language model alone, it transmits the service information relating to the service device that is the target of the instruction in the instruction information, specifically the authentication information and at least command information among the commands, from the registered service information to the controller.

[0021] According to the above configuration, by adopting a form of service equipment control by a controller, and by securing and providing service information to the controller using a large-scale language model, it becomes possible to establish a natural and smooth service equipment control system within the controller as a whole. Ultimately, this enables more integrated control of smart homes through natural conversation that is less burdensome for the user.

[0022] Further, according to the integrated control method of the present invention, in a system including a controller that receives user instruction information and a large language model that analyzes the instruction information and executes processing according to the instruction information, when the content of the instruction information cannot be handled only by the large language model, among the service information for using service devices, which has been registered in advance, the service information corresponding to the instruction information is transmitted to the controller. When the controller receives the service information from the large language model, the controller communicates with the service device based on the service information and executes processing for using the service provided by the service device, thereby solving the problem.

[0023] According to the above integrated control method, through natural conversation that is less burdensome to the user, integrated control of a smart home becomes possible.

Effects of the Invention

[0024] According to the integrated control system and integrated control method of the present invention, through natural conversation that is less burdensome to the user, integrated control of a smart home becomes possible.

Brief Description of the Drawings

[0025] [Figure 1] It is a diagram showing an example of a network configuration including the integrated control system of the present embodiment. [Figure 2] It is a diagram showing an example of the configuration of a service information group in the present embodiment. [Figure 3] It is a diagram showing an example of the hardware configuration of the controller in the present embodiment. [Figure 4] It is a diagram showing an example of the hardware configuration of the service device in the present embodiment. [Figure 5] It is a diagram showing an example of a flow of the integrated control method in the present embodiment. [Figure 6] It is a diagram showing an example of an output in the present embodiment. [Figure 7] It is a diagram showing an example of an output in the present embodiment. [Figure 8A] This figure shows an example of a command in this embodiment. [Figure 8B] This figure shows an example of a command in this embodiment. [Figure 9] This figure shows an example of the output in this embodiment. [Figure 10] This figure shows an example of the output in this embodiment. [Figure 11] This figure shows an example of the flow of the integrated control method in this embodiment. [Figure 12] This figure shows an example of a command in this embodiment. [Figure 13] This figure shows an example of the output in this embodiment. [Figure 14A] This figure shows an example of a command in this embodiment. [Figure 14B] This figure shows an example of a command in this embodiment. [Figure 15] This figure shows an example of the output in this embodiment. [Modes for carrying out the invention]

[0026] <<System configuration including the integrated control system of this embodiment>> The configuration of the integrated control system 10, the methods executed by the integrated control system 10, and the programs therefor will be described below, using one embodiment of the present invention (hereinafter referred to as "this embodiment") as an example, with reference to the attached drawings. However, the embodiment described below is merely an example given to facilitate understanding of the present invention and does not limit the present invention. That is, the present invention can be modified or improved from the embodiment described below without departing from its spirit. Naturally, the present invention includes equivalents thereof.

[0027] Furthermore, the screen examples shown in the diagrams referenced in the following explanation are merely examples, and the screen configuration, the content of the displayed information, and the GUI (Graphical User Interface) can be freely designed and modified according to the system design specifications and user preferences.

[0028] Furthermore, in this specification, "device," "equipment," "system," or "terminal" includes not only a single device that performs a predetermined function on its own, but also multiple devices that are separate from each other but cooperate to perform a predetermined function.

[0029] First, the integrated control system 10 of this embodiment will be described. Figure 1 is a diagram showing an example of a network configuration including the integrated control system 10 in this embodiment. This integrated control system 10 consists of a controller 100 and an LLM (Large Language Model) 200, and is configured to be able to communicate with user terminals 300 and service equipment 400 via a network N. This integrated control system 10 is an information processing system that performs each process corresponding to the integrated control method of the present invention.

[0030] In the integrated control system 10 described above, the controller 100 is the main information processing device that realizes a so-called smart home in the building where the user lives. This controller 100 receives requests from the user and cooperates with the LLM 200 as appropriate in response to those requests to control the service equipment 400. Specific implementation forms of the controller 100 include smart speakers, smartphones, or various home appliances (in-home devices) equipped with the functions of such smart speakers.

[0031] Furthermore, LLM200 is a language model that performs analysis such as speech recognition on voice data spoken by a user at the controller 100 and estimates its content. LLM200 is also a language model that converts the results of control processing of the service device 400 performed by the controller 100 into natural language expressions for the user and responds to the controller 100. Note that the data subject to analysis by LLM200 is not limited to the voice data mentioned above, but may also include various forms of data, such as images of the user's facial expressions and gestures. Such data subject to analysis becomes input related to the user's operation requests to the service device 400. In this embodiment, LLM200 holds a service information group 210, or makes it available to other devices via the network N.

[0032] As illustrated in Figure 2, the service information group 210 described above consists of, for example, for each service device 400, information such as the attributes of the device and service, as well as the access destination (endpoint URL), device ID, device name, installation location, authentication information related to the access destination (authentication information), and commands for controlling the service device, based on a protocol defined for remote control. The endpoint URL is the access destination in the Web API, i.e., the address to which the web service is provided. The information constituting the service information group 210 is information registered by the user, for example, by operating the user terminal 300, or information obtained by the LLM200 from a specific other device on the network N specified by the user. This specific other device could be, for example, a closed information management system that can be accessed after a predetermined authentication process, or a system that is publicly accessible but can only be accessed by those who know the access destination URL (Uniform Resource Locator) (a system subject to URL-limited access). Furthermore, the service information group 210 is maintained and managed by each controller 100 in the LLM200.

[0033] On the other hand, the service equipment 400 controlled by the controller 100 may include, for example, a weather information service server, an IoT home appliance that can be remotely controlled via network N, or a server that provides membership information services to customers in a company, but is not limited to any equipment that can be controlled by the controller 100.

[0034] It should be noted that this network configuration is merely an example, and various other configurations are possible and are not limited to the present invention. These include a configuration in which the integrated control system 10 is configured as a single device with the user terminal 300 and service equipment 400, or a configuration in which at least one of the user terminal 300 or service equipment 400 is equipped with all or part of the configuration and functions of the integrated control system 10 to execute the integrated control method of the present invention.

[0035] <Controller Hardware Configuration> The controller 100, which constitutes the integrated control system 10 of this embodiment together with the LLM200, is a computer device that executes the processing of each step in the integrated control method of the present invention. Therefore, it has the hardware configuration shown in Figure 3. The controller 100 has the configuration of a general information processing device and includes an auxiliary storage device 101, a main memory device 102, a processor 103, an input device 104, an output device 105, and a communication device 106.

[0036] In this embodiment, the controller 100 may consist of a single information processing device as shown in the figure, or it may consist of multiple parallel and distributed information processing devices. Alternatively, the controller 100 may consist of information processing devices for ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service).

[0037] Here, assuming that the controller 100 in this embodiment is composed of a single information processing device, the diagram illustrates a configuration in which the auxiliary storage device 101, main storage device 102, processor 103, input device 104, output device 105, and communication device 106 are connected by a bus.

[0038] Of the above configurations, the auxiliary storage device 101 is implemented using a non-volatile storage device or storage medium such as an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, FD (Flexible Disc), MO disk (Magneto-Optical disc), CD (Compact Disc), DVD (Digital Versatile Disc), SD card (Secure Digital card), or USB memory (Universal Serial Bus memory).

[0039] Furthermore, the auxiliary storage device 101 may be configured to be built into the casing of the controller 100, or it may be configured to be connected to the controller 100 in an external manner. In addition, the auxiliary storage device 101 may be configured as another computer or the like that is connected to the controller 100 in a communicative manner. As for the technology for recording various data, a distributed ledger technology such as blockchain may be used to avoid unauthorized data tampering. In this embodiment, the auxiliary storage device 101 stores the program 1011. The auxiliary storage device 101 may also hold a set of service information (not shown in Figure 3) similar to that of the LLM200. In that case, the set of service information may be, for example, the one that is set by default in the controller 100 as an existing smart speaker, and may be different from the set of service information 210 in the LLM200.

[0040] Furthermore, the main memory 102 may be composed of volatile semiconductor memory such as ROM (Read Only Memory) and RAM (Random Access Memory). In this embodiment, the main memory 102 holds the program 1011, which includes the OS (Operating System), read from the auxiliary memory 101.

[0041] The OS implements the control and basic functions of the controller 100 itself, and under its control, the processor 103 calls and executes program 1011 to implement each function corresponding to the integrated control method. The processor 103 may be composed of a CPU (Central Processing Unit), MPU (Micro-Processing Unit), MCU (Micro Controller Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), TPU (Tensor Processing Unit), or ASIC (Application Specific Integrated Circuit), etc.

[0042] Furthermore, the input device 104 mainly includes microphones and touch panels, but may also include other appropriate input devices such as mice and keyboards. On the other hand, the output device 105 includes speakers and displays, but may also include other output devices.

[0043] The communication device 106 is implemented using a communication chipset that supports, for example, 3G to 5G or later generations of mobile communication protocols, or the LTE (Long Term Evolution) protocol. However, it may be implemented using other devices depending on the type of network N. For example, if network N is composed of a wireless LAN, the communication device 107 will be implemented using a network interface card that supports the Wi-Fi®-based wireless LAN protocol.

[0044] <Service equipment hardware configuration> Next, the hardware configuration of the service device 400 in this embodiment will be described with reference to Figure 4. The service device 400 consists of a control server 410 connected to the network N and cooperating with the controller 100, and a controlled device 420 that operates in response to commands from the control server 410. Of course, a configuration in which the control server 410 and the controlled device 420 are integrated into a single device is also possible. The control server 410 has a configuration as a general server device and includes an auxiliary storage device 411, a main memory device 412, a processor 413, and a communication device 414.

[0045] In this embodiment, the control server 410 may consist of a single information processing device as shown in the figure, or it may consist of multiple parallel distributed information processing devices. Alternatively, the control server 410 may consist of information processing devices for ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service).

[0046] In this embodiment, assuming that the control server 410 is composed of a single information processing device, the diagram illustrates a configuration in which the auxiliary storage device 411, main storage device 412, processor 413, and communication device 414 are connected by a bus.

[0047] Of the above configurations, the auxiliary storage device 411 is implemented using non-volatile storage devices or storage media such as HDDs (Hard Disk Drives), SSDs (Solid State Drives), flash memory, FDs (Flexible Discs), MO disks (Magneto-Optical Discs), CDs (Compact Discs), DVDs (Digital Versatile Discs), SD cards (Secure Digital Cards), or USB memory (Universal Serial Bus memory).

[0048] Furthermore, the auxiliary storage device 411 may be configured to be built into the enclosure of the control server 410, or it may be configured to be connected to the control server 410 in an external manner. In addition, the auxiliary storage device 411 may be configured as another computer or the like that is connected to the control server 410 in a communicative manner. As for the technology for recording various data, a distributed ledger technology such as blockchain may be used to avoid unauthorized data tampering. In this embodiment, the auxiliary storage device 411 stores the control program 4111.

[0049] This control program 4111 sends commands to the controlled device 420 via the communication device 414 to control events such as the on / off state and operating status of the controlled device 420, and retrieves the execution results from the controlled device 420. The commands used by the control program 4111 may include those provided by the LLM200, as well as known commands that conform to the protocol of the controlled device 420 which was controlled by default.

[0050] Furthermore, the main memory 412 may be composed of volatile semiconductor memory such as ROM (Read Only Memory) and RAM (Random Access Memory). In this embodiment, the main memory 412 holds the program 4111, which includes the OS (Operating System) read from the auxiliary memory 411.

[0051] Of these, the OS implements the control and basic functions of the control server 410 itself, and under its control, the processor 413 calls and executes program 4111 to implement the control functions of the controlled device 420. The processor 413 may be composed of a CPU (Central Processing Unit), MPU (Micro-Processing Unit), MCU (Micro Controller Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), TPU (Tensor Processing Unit), or ASIC (Application Specific Integrated Circuit), etc.

[0052] Furthermore, the communication device 414 is implemented using a communication chipset that supports, for example, 3G to 5G or later generations of mobile communication protocols, or the LTE (Long Term Evolution) protocol. However, it may also be implemented using other devices depending on the communication path with the controlled device 420. For example, if the communication path with the controlled device 420 is configured as a wireless LAN, the communication device 414 will be implemented using a network interface card that supports the Wi-Fi®-based wireless LAN protocol.

[0053] On the other hand, as already mentioned, the controlled device 420 is, for example, a weather information service provider server, an IoT home appliance that can be remotely controlled via network N, or a server that provides membership information services to customers in a company. Such a controlled device 420 consists of a communication device 421, a control mechanism 422, and an equipment unit 423. Of these, the communication device 421 is connected to the control server 410 via a wireless or wired communication path and transmits and receives data, and is a communication chipset that corresponds to the protocol of the communication path. The control mechanism 422 is a mechanism that controls the power supply unit and various control units based on commands received via the communication device 421. This control mechanism 422 can take the form of a unit that controls the voltage and current values ​​applied to the equipment unit 423 as an example. The equipment unit 423 is the main component of the controlled device 420 and is the device itself, such as a lighting fixture, air conditioner, or security device.

[0054] <Flowchart of integrated control method: Initial setup> Next, a flowchart example of the integrated control method in this embodiment will be described. Figure 5 shows a flowchart example of the integrated control method in this embodiment. Here, a series of steps are shown in which the user operates the user terminal 300 to obtain service information for the service equipment 400 that the user wants to control, i.e., to link with the controller 100 or LLM200, and registers this information with the LLM200 or controller 100. However, in this embodiment, the explanation will focus on the LLM200 as an example of a registration destination for service information.

[0055] First, the user terminal 300 receives various specifications from the user as part of the service linkage application for the service equipment 400 (and the services provided by it) that the user wishes to control with the controller 100 (S1). At this time, the user terminal 300 outputs a screen G1, for example, as shown in Figure 6, and receives the user's specification regarding the service to be linked via this screen G1. In the example of screen G1 in Figure 6, services such as "weather service," "equipment control service," "home appliance control service," "IoT device service," "instruction manual service," and "member information service" are presented as options, and the user has selected "home appliance control service."

[0056] After specifying the target for integration on screen G1 above, if the user taps the "Next" button, the user terminal 300 displays screen G2 shown in Figure 7, and accepts the user's selection of the manufacturer and specific type of equipment for the "home appliance" service to be integrated. In the example of screen G2 in Figure 7, the user has selected an "air conditioner" product from a manufacturer called "X Precision". If the user taps the "Next" button on this screen G2, the user terminal 300 queries a management system (e.g., a server operated by the manufacturer) that manages various information about the "air conditioner" product of the aforementioned "X Precision" company for service information such as authentication information and commands necessary to control the "air conditioner" product.

[0057] Of course, the user terminal 300 is assumed to have prior information on the URLs for these inquiries, i.e., the URLs for executing service integration applications. Furthermore, these inquiries are not limited to systems operated by the manufacturer of the service equipment 400, but may also include, for example, servers that compile and publish specifications and control information for each company's products. These servers are operated by the provider of the controller 100 or the operator of the LLM200 and are made public to users of the controller 100 and LLM200. In addition, if the user has already obtained the service information through a separate route, the process in S1 (and S2 described later) can be skipped.

[0058] Meanwhile, the management system of the manufacturer receives a service linkage request from the user terminal 300 and responds to the user terminal 300 with various control information it holds regarding the target product according to the content of the request (S2). Figures 8A and 8B show examples of commands, which are one type of control information. Of these, command C1 shown in Figure 8A is a control command for the "air conditioner" and includes fields for setting authentication information such as ID and password (C1A), a field for entering the URL of the service provider (C1B), a field for setting the device ID (C1C), and a field for setting the control content (C1D). Command C2 shown in Figure 8B is a command for obtaining a weather forecast for a specific area and includes fields for entering the URL of the service provider (C2A), a field for setting authentication information such as ID and password (C2B), and a field for setting the target area for the weather forecast (C2C). In this case, the user terminal 300 receives the control information responded to from the management system as service information (S3) and stores it, for example, in main memory. Next, the user terminal 300 notifies the LLM200 (or controller 100) of a service registration request regarding the "air conditioner" product mentioned above (S4).

[0059] In this case, LLM200 receives the service registration request and sequentially responds to the user terminal 300 with the corresponding product or service registration screens G3 (see Figure 9) and G4 (see Figure 10) (S5). On registration screen G3, the user terminal 300 accepts user input regarding attributes of the target service equipment 400, such as the device ID, name, and installation location. The device ID and name of the service equipment 400 are the serial number, model number, and product name of the target home appliance, etc., and are values ​​known to the user. On registration screen G4, the user terminal 300 sets the information already held in S3, such as the endpoint URL, user ID, authentication password, and control commands for accessing the manufacturer's management system, and receives user confirmation (S6).

[0060] Furthermore, in S6, the user terminal 300 notifies the LLM200 of the setting request regarding the attributes and authentication information of the service device 400, which it obtained or confirmed on the registration screens G3 and G4 as described above. On the other hand, the LLM200 adds the attributes and authentication information regarding the service device 400 to be registered, as indicated by the setting request received from the user terminal 300, to the service information group 210 (S7). Note that the registration and storage of such service information may be performed similarly between the user terminal 300 and the controller 100.

[0061] <Flowchart of Integrated Control Method: Control of Service Equipment> Next, the procedure for the controller 100 to control the service equipment 400 will be explained based on the flow chart in Figure 11. Prior to or in conjunction with the start of this flow chart, the LLM 200 is assumed to have appropriate service information 210. Furthermore, the user is assumed to be standing in front of the controller 100 and intending to operate, for example, the air conditioner, which is the service equipment 400, by voice. The user then says to the controller 100, "Set the air conditioner on the first floor to 26 degrees and start heating." In this case, the controller 100 will: The utterance is detected by the microphone, which is the input device 104 (S10), and the utterance data is transmitted to the LLM200 as a user instruction (instruction information) (S11). At this time, if the controller 100 has the same speech recognition function as the LLM200, the controller 100 may perform the speech recognition processing for the utterance itself. In that case, the content transmitted in S11 will be text data which is the result of speech recognition of the utterance.

[0062] Meanwhile, LLM200 receives the user's utterance data from the controller 100 and analyzes the content of the user's instructions (S12). This analysis includes the speech recognition process and the determination of the execution mode based on the results of the speech recognition process. The speech recognition process is a process that uses general functions in a large-scale language model and converts the utterance into text. The determination of the execution mode, on the other hand, is a process that determines whether the user's utterance, i.e., the user's instructions, pertains to an event that has already been learned by LLM200. An event that has already been learned by LLM200 is, for example, information that is published at a certain frequency and in a certain quantity on the network N and can be collected by a predetermined crawler, such as source code of programs published by many engineers, literature and history described in textbooks, various laws and theories described in scientific papers, and past trends in exchange rates and stock prices, and is, for example, information that has been learned by a large-scale GPU-based machine learning. In other words, events indicated by information that is not publicly available or is only publicly available in very limited quantities on network N are unlikely to become learned events in LLM200.

[0063] Therefore, LLM200 determines whether the execution mode is normal or specific based on whether the user's utterance, i.e., user instruction, as shown by the results of the analysis in S12, has already been learned by LLM200 (S13). An execution mode of "normal" means that the response concerns an event that has already been learned by LLM200. An execution mode of "specific" means that the response concerns an event that has not been learned by LLM200. If the execution mode is "normal" as a result of the above determination (S13: normal), LLM200 creates a response to the user instruction (for example, "The relationship between quantum mechanics and general relativity is...") and responds to the controller 100 (S14). In this case, the controller 100 receives the response, i.e., the answer from LLM200, speaks it through the speaker, which is the output device 105 (or displays it on the display) (S15), and terminates the flow.

[0064] On the other hand, if the execution mode is "specific" as a result of the above determination (S13: specific), LLM200 identifies the linked service (target of instruction) and arguments corresponding to the user instruction in the service information group 210 and sends them to the controller 100 (S16). For example, if the user instruction is "set the air conditioner on the 1st floor to 26 degrees and start heating," LLM200 identifies a record in the service information group 210 held by the controller 100 that includes "air conditioner" as the "equipment name" and has "1st floor" as the "installation location," for example, a record with "service equipment ID" of "001" and device ID of "E12F23G34H45."

[0065] Therefore, LLM200 will obtain the endpoint URL "https: / / www.*****.co.jp / X0T / **** / ", authentication information, and the command "curl-x POST*****" for the "Air Conditioner ACDX0T" with device ID "E12F23G34H45" as a linked service, and respond to the controller 100 with these. In this response, LLM200 will send the above command with the endpoint URL and authentication information set to it to the controller 100. An example of such a set command C10 is shown in Figure 12. The command C10 shown in Figure 12 has authentication information C10A including user ID and authentication password, access destination information C10B including endpoint URL, identification information C10C for the service equipment 400 including device ID, and control value C10D including user instruction set.

[0066] Meanwhile, the controller 100 receives the command C10 from the LLM 200 and transmits it to the target service equipment 400 (S17). The control server 410 in the service equipment 400 receives the command C10 via the communication device 414 and executes control of the controlled equipment 420 (in this case, the air conditioner) based on the control program 4111 (S18). In this case, the controlled equipment 420 receives the command C10 via its communication device 421 and, based on this, performs operations such as turning the power on / off in the control mechanism 422 and controlling the voltage applied to the equipment unit 423, thereby executing user-specified operations. The control server 410 of the service equipment 400 also responds to the controller 100 with the result of the control (operation) performed in the controlled equipment 420. This result may be, for example, "1F, air conditioner, setting: 26 degrees, heating: ON".

[0067] The controller 100 receives the above execution result from the service device 400 and sends a conversion request including this execution result to the LLM 200 (S19). This conversion request asks that the execution result of control, etc., in response to user instructions be formatted into a natural conversational format. An example of a prompt for the conversion request would be, "You are the administrator who manages the air conditioner on behalf of the user. Please format the control result "1F, air conditioner, setting: 26 degrees, heating: ON" into a response or report in a natural conversational format directed at the user." For this reason, the controller 100 has a pre-existing prompt template, "You are the administrator who manages XX on behalf of the user. Please format the control result "***" into a response or report in a natural conversational format directed at the user," and generates the prompt by setting the name of the service device 400 in the "XX" part and the execution result of the control in the "***" part.

[0068] Meanwhile, upon receiving the above prompt, the LLM200 generates a response in a natural conversational format, such as, "The air conditioner on the first floor has been set to heating at 26 degrees and has started operation. Please enjoy the warm and comfortable space," and sends this response to the controller 100 (S14). The controller 100 receives the above response from the LLM200 and, based on this, performs speech output through the speaker or display on the display (S15), and ends the flow. Figure 13 shows an example of screen G10 displaying the above response on the display.

[0069] In addition to the above example, let's explain another example where the user instruction is, for example, "Tell me the weather forecast for Nara." In this case, in S16 above, LLM200 identifies a record in the service information group 210 held for the controller 100 that includes "weather forecast" as the "device / service name," for example, a record where the "service device ID" is "002" and the device / service name is "latest weather forecast."

[0070] Therefore, LLM200 will obtain the endpoint URL "http: / / www.weather***.co.jp / we8 / **** / ", authentication information, and the command "curl-GET*****" for the linked service "Latest Weather Forecast," and respond to the controller 100 with these. In this response, LLM200 will send the above command with the endpoint URL and authentication information set to it to the controller 100. An example of such a set command C20 is shown in Figure 14A. The command C20 shown in Figure 14A has the access destination information C20A including the endpoint URL, authentication information C20B including the user ID and authentication password, and the control value C20C including the user instruction (e.g., nara) set.

[0071] Meanwhile, the controller 100 receives the command C20 from the LLM 200 and transmits it to the service equipment 400, such as the web server of the target weather forecast site (S17). The control server 410 in the service equipment 400 receives the command C20 via the communication device 414 and executes control of the controlled equipment 420 (e.g., a DB server where weather forecast data is stored) based on the control program 4111 (S18).

[0072] In the above case, the controlled device 420 receives the command C20 via its communication device 421, executes a database search based on this command, and performs the user-specified search operation. The control server 410 of the service device 400 responds to the controller 100 with the result of the control (operation) performed by the controlled device 420. This result might be something like, "Tomorrow, Nara, Weather: Cloudy, mostly cloudy." Figure 14B shows an example of this result in command format. This command C21 includes the weather forecast values ​​C21A and C21B, which are the result of the execution, and the value C21C for the forecast area.

[0073] The controller 100 receives the above execution result from the service device 400 and sends a conversion request including this execution result to the LLM200 (S19). This conversion request asks the controller to format the execution result of control, etc., in response to user instructions into a natural conversational format. An example of a prompt for the conversion request would be, "You are a commentator who explains the weather forecast in an easy-to-understand way on behalf of the user. Please format the weather forecast result, "Tomorrow, Nara, Weather: Cloudy, mostly cloudy," into a response or report in a natural conversational format for the user." For this reason, the controller 100 has a pre-existing prompt template, "You are a commentator who explains XX in an easy-to-understand way on behalf of the user. Please format the XX prediction result, "***," into a response or report in a natural conversational format for the user," and generates the prompt by setting the service name in the "XX" part and the control execution result in the "***" part.

[0074] Meanwhile, upon receiving the above prompt, the LLM200 generates a response (response data) in a natural conversational format, such as, "The forecast for Nara tomorrow is mostly cloudy. It will likely be a cloudy day, but there is no expectation of any major weather changes," and sends this response to the controller 100 (S14). The controller 100 receives the above response from the LLM200 and, based on this, performs speech output through the speaker or display on the display (S15), and terminates the flow. Figure 15 shows an example of screen G11 with the above response displayed on the display.

[0075] In the integrated control system of this embodiment described above, the controller 100 works in cooperation with the LLM200, and within the scope that the LLM200 can handle (in other words, the scope that it can learn), services can be provided without installing any dedicated programs, and for areas that the LLM200 cannot handle, responses can be made based on a pre-registered set of service information 210. Furthermore, conventionally, it was necessary for the user to use different programs for each service, but with the technology of this embodiment, the user can issue instructions through the controller 100 only, regardless of the type of service. In addition, the LLM200 determines which service the instruction received by the controller 100 corresponds to, and can take appropriate action and obtain the execution result according to the determination result. Moreover, the user's requests for service execution and receipt of execution results are performed seamlessly in a natural conversational format by the LLM200, effectively reducing the effort and burden of understanding various aspects on the user's side.

[0076] Although one embodiment of the integrated control system and integrated control method of the present invention has been described above, the above embodiment is merely an example to facilitate understanding of the present invention and does not limit the present invention. In other words, the present invention can be modified and improved without departing from its spirit. Furthermore, it goes without saying that the present invention includes equivalents thereof. [Explanation of Symbols]

[0077] N Network 10. Integrated control system 100 controllers 101 Auxiliary storage 1011 Program 102 Main storage 103 Processors 104 Input device 105 Output device 106 Communication equipment 200 LLM 210 Service Information Group 300 user terminals 400 Service Equipment 410 Control Server 411 Auxiliary storage 4111 Control Program 412 Main storage 413 processors 414 Communication equipment 420 Controlled Devices 421 Communication equipment 422 Control mechanism 423 Equipment Unit

Claims

1. A controller that receives user instruction information, The system comprises a large-scale language model that analyzes the instruction information and performs processing according to the instruction information, If the content of the instruction information cannot be handled by the large-scale language model alone, the large-scale language model transmits service information corresponding to the instruction information from the pre-registered service information for using the service equipment to the controller. When the controller receives the service information from the large-scale language model, it communicates with the service equipment based on the service information and executes processing to use the service provided by the service equipment. An integrated control system characterized by the following:

2. The aforementioned large-scale language model, If the content of the instruction information can be handled solely by the large-scale language model, the response data generated by the large-scale language model in response to the instruction information is transmitted to the controller. The integrated control system according to claim 1.

3. The aforementioned service information is information for controlling in-home devices. The integrated control system according to claim 1.

4. The aforementioned service information is information for controlling the web service. The integrated control system according to claim 1.

5. The aforementioned large-scale language model, For each of the aforementioned service devices, service information, including at least command information among the authentication information and commands necessary for using the service device, is acquired in advance from a predetermined device and registered in a storage device. If the content of the instruction information cannot be handled by the large-scale language model alone, the controller shall receive at least the command information from among the authentication information and the command information, which are service information relating to the service equipment that is the target of the instruction information among the registered service information. The integrated control system according to claim 1.

6. In a system comprising a controller that receives user instruction information and a large-scale language model that analyzes the instruction information and executes processing according to the instruction information, If the content of the instruction information cannot be handled by the large-scale language model alone, the large-scale language model transmits the service information corresponding to the instruction information from the pre-registered service information for using the service equipment to the controller. When the controller receives the service information from the large-scale language model, it communicates with the service equipment based on the service information and executes processing to use the service provided by the service equipment. An integrated control method characterized by the following:

Images