Information processing method, information processing program, and information processing device

Abstract

In the present invention, a processor: uses a camera to capture a space in which an electronic apparatus embodying a predetermined function is to be disposed; displays the captured image of the space on a display; accepts a designation or operation input by a user on the image of the captured space; disposes a virtual object of the electronic apparatus designated by the user according to the operation of the user; superimposes and displays the virtual object of the disposed electronic apparatus on the captured image of the space; and displays a simulation executed by using the virtual object of the electronic apparatus on the captured image of the space.

Description

Information processing method, information processing program, and information processing device

[0001] The present disclosure relates to an information processing method, an information processing program, and an information processing device.

[0002] It is already common for multiple devices in a home to function in cooperation with one another. For example, Patent Literature 1 describes a technology that proposes a combination of individual devices to solve a problem.

[0003] Japanese Patent Application Laid-Open No. 2004-342113

[0004] The technology described in the above-mentioned Patent Document 1 combines devices and displays a proposal. However, the above-mentioned technology does not suggest devices that the user does not own or that are unknown. Therefore, in order to confirm whether a solution using a device that the user does not yet own is truly effective in solving the user's problem, the user has no choice but to actually purchase the device and check it on the actual device.

[0005] Therefore, the present disclosure proposes an information processing method, an information processing program, and an information processing device that allow the placement and operation of a device that is not owned to be checked in an actual environment without actually purchasing the device and checking it on the actual machine.

[0006] According to the present disclosure, an information processing method is provided in which a processor uses a camera to capture an image of a space in which an electronic device that realizes a predetermined function is to be placed, displays an image of the captured space on a display, accepts a user's input of a designation or operation on the image of the captured space, places a virtual object of the electronic device designated by the user in accordance with the user's operation, superimposes the placed virtual object of the electronic device on the image of the captured space, and displays a simulation performed using the virtual object of the electronic device on the image of the captured space.

[0007] 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. FIG. 1 is a diagram illustrating an example configuration of an electronic device according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example functional configuration of a terminal device according to an embodiment of the present disclosure. FIG. 3 is a block diagram illustrating an example functional configuration of a control device according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating an image of a recipe according to an embodiment of the present disclosure. FIG. 5 is a flowchart illustrating a processing procedure for posting a recipe according to an embodiment of the present disclosure. FIG. 6 is a flowchart illustrating a processing procedure for recipe analysis according to an embodiment of the present disclosure. FIG. 7 is a flowchart illustrating a processing procedure for preparation before recipe generation according to an embodiment of the present disclosure. FIG. 8 is a flowchart illustrating a processing procedure for recipe search according to an embodiment of the present disclosure. FIG. 9 is a diagram illustrating an image of a recipe search. FIG. 10 is a flowchart illustrating a processing procedure for improving recipe suggestions through continued use according to an embodiment of the present disclosure. FIG. 11 is a flowchart illustrating a processing procedure for improving recipe suggestions based on user information according to an embodiment of the present disclosure. FIG. 12 is a flowchart illustrating a processing procedure for improving recipe suggestions based on correlations between the real world and sensors according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an overview of recipe suggestions based on accumulated data and a program. FIG. 14 is a flowchart illustrating a processing procedure for improving recipe suggestions based on feedback according to an embodiment of the present disclosure. FIG. 14 is a flowchart illustrating a processing procedure for implementing a simulation according to an embodiment of the present disclosure. FIG. 15 is a diagram illustrating an example of a settings screen including a slider bar. FIG. 16 is a diagram illustrating an overview of a trial of a simulation including another workpiece. FIG. 17 is a diagram illustrating an example screen illustrating the placement of virtual objects of electronic blocks. FIG. 18 is a diagram illustrating an example screen illustrating behavior results of virtual objects of electronic blocks. FIG. 19 is a diagram illustrating an example screen illustrating interactions from on the screen. FIG. 1 is a diagram showing an example of a screen showing behavior that takes accessibility into consideration; FIG. 2 is a diagram showing an example of a video of an operation; FIG. 3 is a diagram showing an example of parameter settings on a screen; FIG. 4 is a diagram showing an example of data of a procedure manual; and FIG. 5 is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The present disclosure will be described in the following order of items. 1. Embodiments 1-1. Overview of information processing system according to embodiments 1-2. Example configuration of electronic device 1-3. Example functional configuration of terminal device 1-4. Example functional configuration of control device 2. Program sharing service via cloud service utilizing AI 2-1. Sharing and proposing solutions to problems 2-1-1. Image of recipe 2-2. Posting recipe 2-3. Recipe analysis 2-4. Preparation before recipe creation 2-5. Recipe search 2-5-1. Image of recipe search 2-6. Example 1: Data acquired by recipe 2-6-1. Improvement of recipe suggestions through continued use 2-7. Example 2: Information associated with user 2-7-1. Improvement of recipe suggestions through user information 2-8. Example 3: Association between real world and sensors 2-8-1. Improvement of recipe suggestions through association between real world and sensors 2-9. 2-10. Example 4: Recipe suggestions using accumulated data and programs 2-10. Example 5: Data acquired through feedback 2-10-1. Improvement of recipe suggestions through feedback 3. Simulation system not requiring electronic device itself 3-1. Realization of simulation system 3-2. Overview of simulation implementation 3-3. Placement of virtual electronic block objects 3-4. Behavior results of virtual electronic block objects 3-5. Interaction from the screen 3-6. Behavior taking accessibility into consideration 3-7. Simulation implemented from video of operation 3-8. Simulation implemented from text of desired action 4. Other embodiments 5. Effects of information processing method according to the present disclosure 6. Hardware configuration

[0010] 1. Embodiment 1-1. Overview of Information Processing System According to Embodiment First, an overview of an information processing system 10 according to an embodiment of the present disclosure will be described using Fig. 1. Fig. 1 is a diagram showing an overview of the information processing system 10 according to an embodiment of the present disclosure. Referring to Fig. 1, the information processing system 10 includes an electronic device 100, a terminal device 200, a relay device 300, and a control device 400.

[0011] The electronic device 100 and the terminal device 200, the electronic device 100 and the relay device 300, and the terminal device 200 and the relay device 300 can communicate with each other via a network NW. When the devices and equipment are located in close proximity, the network NW is a wireless communication network such as Bluetooth (registered trademark) or Wi-Fi (Wireless Local Area Network). When the devices and equipment are located far away or externally, the network NW may be the Internet, a mobile network, or the like. The relay device 300 and the control device 400 communicate with each other via the Internet, a mobile network, or the like. The mobile network may be, for example, Long Term Evolution (LTE), 4th Generation (4G), or 5th Generation (5G).

[0012] The electronic device 100 is a device or module that can be controlled by a program and realizes predetermined functions such as a sensor or actuator. The electronic device 100 has a shape, for example, a rectangular parallelepiped block. Hereinafter, the device as the electronic device 100 may also be referred to as an "electronic block." However, the rectangular parallelepiped block is merely an example. In reality, the shape of the electronic device 100 is arbitrary. In the example shown in FIG. 1 , the electronic devices 100 include electronic device 100A, electronic device 100B, and electronic device 100C. The electronic device 100B and the electronic device 100C can be combined to operate in cooperation with each other.

[0013] The terminal device 200 is an information processing device used by a user. For example, the terminal device 200 may be a smart device such as a smartphone or a tablet terminal, a desktop or laptop PC (Personal Computer), a wearable device such as a head-mounted display (HMD) or smart glasses, an information appliance or home appliance with a monitor compatible with the Internet of Things (IoT), a game console or AV device with a communication function, a car navigation system, or the like. In the present disclosure, the terminal device 200 is capable of monitoring the operation of the electronic devices 100A, 100B, and 100C as the electronic devices 100. The terminal device 200 may also create a program that controls the operation of the electronic devices 100A, 100B, and 100C as the electronic devices 100 in response to a user's operation.

[0014] The relay device 300 is an information processing device that relays communications between the electronic device 100 and the control device 400 and between the terminal device 200 and the control device 400. In practice, the relay device 300 may also relay communications between the electronic device 100 and the terminal device 200. For example, the relay device 300 is a gateway, a router, a repeater, a base station, or the like. In this embodiment, a gateway will be described as an example of the relay device 300.

[0015] The control device 400 is an information processing device that communicates with the electronic device 100 and the terminal device 200 via the relay device 300 to transmit and receive information and control the electronic device 100 and the terminal device 200. The control device 400 is implemented by a computer, a cloud system, or the like. The control device 400 is, for example, a computer such as a server device or a PC, or a mainframe or workstation. In this embodiment, the control device 400 is one or more server devices that constitute a cloud CL, which is a cloud system, and exists on the cloud CL. The control device 400 also cooperates with the terminal device 200 to provide the terminal device 200 with API (Application Programming Interface) services for applications (hereinafter, “apps”) and various data. The control device 400 also implements the mechanism according to the present disclosure using AI (Artificial Intelligence) such as a Generative Pre-trained Transformer (GPT). GPT is a text generation AI and a language model capable of generating sentences using natural language processing. For example, a user sends a prompt (prompt: instruction sentence) from the terminal device 200 to the control device 400 on the cloud CL via the relay device 300, thereby inputting the prompt into an AI such as GPT and obtaining a response.

[0016] The number of devices and equipment included in the information processing system 10 shown in Fig. 1 is not limited to those shown. For example, in Fig. 1, one terminal device 200, one relay device 300, and one control device 400 are shown for the sake of simplicity, but this is merely an example and is not limiting, and two or more of each may be used. In reality, other devices may also be combined.

[0017] 1-2. Example of the Configuration of an Electronic Device An example of the configuration of an electronic device will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of an electronic device according to an embodiment of the present disclosure.

[0018] 2, the electronic device 100 may be separable into a functional block 110 and a common block 120. In this case, the electronic device 100 may be configured such that only one of the functional block 110 and the common block 120 is replaceable.

[0019] The functional blocks 110 have predetermined functions such as sensors and actuators. The functional blocks 110 to be used can have any functions. The common block 120 has common functions such as power supply and wireless. By connecting to the functional blocks 110, the common block 120 provides power and communication functions to the functional blocks 110.

[0020] In reality, the electronic device 100 may not be separated into the functional block 110 and the common block 120, but may be a single block that includes all functions.

[0021] 1-3. Functional Configuration Example of Terminal Device] A functional configuration example of the terminal device 200 according to an embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a block diagram showing a functional configuration example of the terminal device 200 according to an embodiment of the present disclosure.

[0022] The terminal device 200 includes an operation unit 210, a sensor unit 220, a control unit 230, a storage unit 240, a communication unit 250, and a display unit 260. The terminal device 200 may be, for example, an information processing device, but the configuration for realizing general functions of an information processing device is not shown in the figure because it is common.

[0023] The operation unit 210 is an input device that accepts various operations from the user. The operation unit 210 acquires user operations on tags such as icons included in a console screen displayed on the display unit 260. The operation unit 210 may include, for example, a touch screen, a mouse, a touchpad, or other pointing device. The operation unit 210 may also include, for example, a camera for acquiring user gesture operations, a microphone for accepting voice input from the user, and buttons for inputting letters, numbers, and the like. For example, the operation unit 210 accepts input of actual device information of the electronic device 100 via an input device such as a user interface (UI) or keyboard that can be operated by the user.

[0024] The sensor unit 220 includes various sensors mounted on or connected to the terminal device 200. The sensor unit 220 may include, for example, an image sensor such as a camera, a sound sensor such as a microphone, a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), a ToF (Time of Flight) sensor, an IMU (Inertial Measurement Unit), and the like. The sensor unit 220 may also include an acceleration sensor, a gyro sensor, a motion sensor, a temperature sensor, a light sensor, a barometric pressure sensor, a magnetic sensor, and the like. The sensor unit 20 may be configured to include some of the sensors, or may include other sensors in addition to or instead of the sensors. The sensor unit 220 detects the environment surrounding the terminal device 200, the user's actions or circumstances, and the like, and outputs sensor values ​​indicating the detection results to the control unit 230. The sensor values ​​indicating the detection results may be output signals from the sensors. The sensor unit 20 may also be an external device.

[0025] The control unit 230 is realized by a processor such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit) executing a program (e.g., an information processing program according to the present disclosure) stored inside the terminal device 200 using a RAM (Random Access Memory) or the like as a working area. The control unit 230 is also a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0026] The control unit 230 controls the overall operation of the terminal device 200. For example, the control unit 230 acquires information indicating an operation acquired by the operation unit 210 and interprets it as a user operation on the electronic device 100 corresponding to an icon included in the console screen displayed on the display unit 260. The control unit 230 also acquires sensor values ​​indicating detection results from the sensor unit 220 and recognizes the environment surrounding the terminal device 200, the user's actions, or the situation. Furthermore, the control unit 230 transmits instructions to the electronic device 100 that is the target of the operation via the communication unit 250 and the network NW to operate in accordance with the user's operation. Alternatively, the control unit 230 adds, changes, and / or deletes operations set for the electronic device 100, more specifically, cooperative operations with other electronic devices 100, in accordance with the user's operation, and reflects this in the setting information for the electronic device 100 stored in the storage unit 240.

[0027] The control unit 230 is also connected to the relay device 300 via the communication unit 250 and the network NW, transmits various information or data to the control device 400 on the cloud CL via the relay device 300, and receives various information or data from the control device 400 on the cloud CL. The control unit 230 may also be connected to the relay device 300 via the communication unit 250 and the network NW, and receive control from the control device 400 on the cloud CL via the relay device 300. For example, the control unit 230 may receive display control from the control device 400 regarding the display mode of various information or data to be displayed on the display unit 260.

[0028] Furthermore, the control unit 230 receives signals transmitted from GPS (Global Positioning System) satellites via a GPS receiver, and acquires location information indicating the current location of the terminal device 200, such as latitude and longitude, based on the received signals. That is, the control unit 230 measures the location of the terminal device 200. Note that GPS is merely one example of a GNSS (Global Navigation Satellite System). The control unit 230 can also measure the location using various methods other than GPS.

[0029] The storage unit 240 is realized, for example, by various storage devices constituting a random access memory (RAM) or a read-only memory (ROM), and / or removable media and their drivers. The storage unit 240 may also be realized by a hard disk drive (HDD), a solid state drive (SSD), flash memory, or the like. The storage unit 240 temporarily or permanently stores various information or data handled by the control unit 230. For example, the storage unit 240 temporarily stores information indicating operations acquired by the operation unit 210, sensor values ​​indicating detection results from the sensor unit 220, operation information received by the communication unit 250 from the electronic device 100, and the like. Furthermore, the storage unit 240 may store, for example, a program executed by a processor that implements the control unit 230.

[0030] The communication unit 250 is realized by a transmitter and a receiver compatible with a wireless communication network such as Bluetooth (registered trademark) or Wi-Fi. The communication unit 250 may be a communication port, an antenna, or a network interface card (NIC). The communication unit 250 transmits and receives various information or data via the network NW.

[0031] The display unit 260 is realized by various display devices such as a liquid crystal display (LCD) or an organic electroluminescent display (OLED). The display device may be a touch panel. The display unit 260 is not limited to a display device, and may be an output port that outputs various information or data to an external device such as an external monitor or projector. The display unit 260 displays an icon representing the electronic device 100 controlled by the terminal device 200. The icon may have a shape similar to part or all of the actual electronic device 100. In reality, the icon may be displayed in any manner that allows visual recognition that it represents a specific electronic device 100.

[0032] 1-4. Example of Functional Configuration of Control Device An example of the functional configuration of the control device 400 according to an embodiment of the present disclosure will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the functional configuration of the control device 400 according to an embodiment of the present disclosure.

[0033] The control device 400 includes a control unit 410, a storage unit 420, and a communication unit 430. The control device 400 may be, for example, an information processing device, but the configuration for realizing general functions of an information processing device is not shown in the figure because it is common.

[0034] The control unit 410 is realized by a processor such as a CPU, MPU, or GPU executing a program (e.g., an information processing program according to the present disclosure) stored inside the control device 400 using a RAM or the like as a work area. The control unit 410 is also a controller, and may be realized by an integrated circuit such as an ASIC or FPGA.

[0035] The control unit 410 provides a UI (User Interface) to, for example, a display or a touch panel that serves as the display unit 260 of the terminal device 200 via the communication unit 430. Through this UI, the user can select, for example, a program for operating the electronic device 100 independently or a program for operating a plurality of electronic devices 100 in combination from among programs pre-installed in the electronic device 100 or the terminal device 200, programs provided by the control device 400, and the like.

[0036] The storage unit 420 is realized by an HDD, an SSD, or the like. The storage unit 420 temporarily or permanently stores various information or data handled by the control unit 410. The storage unit 420 also stores various databases. The storage unit 420 also stores programs for operating the electronic device 100 independently and programs for operating a plurality of electronic devices 100 in combination. Programs can be added as needed by, for example, a user, a developer, or a system administrator. The storage unit 420 may be a storage device or a server device external to the control device 400.

[0037] The communication unit 430 is realized by a network interface card (NIC), etc. The communication unit 250 transmits and receives various information or data to and from the electronic device 100 or the terminal device 200 via the relay device 300.

[0038] [2. Program sharing service via cloud services utilizing AI] [2-1. Sharing and proposing solutions to problems] Currently, with technology that proposes combinations of individual devices to solve problems, users have problems they want to solve themselves, but they have no idea what tools or services to use, and they don't know how to create them.

[0039] The reasons for this include the difficulty of presenting solutions that can comprehensively solve all problems due to the diversification and deepening of problems, and the fact that users are not able to sufficiently abstract solutions.

[0040] Therefore, in this disclosure, solutions to problems implemented using the electronic device 100 are abstracted as "recipe" so that they can be expressed as symbol strings, and a space for sharing solutions (recipes) is created. The recipes posted there are organized, classified, and abstracted using a proposed method, and feature quantities are generated. These feature quantities are then used to calculate the similarity between recipes, and a recipe desired by the user is selected and proposed (for example, presented). Alternatively, a new recipe is generated and proposed based on a shared recipe. This allows users to obtain solutions using sensors, etc., even without specialized knowledge.

[0041] Features include natural language information, image information, and more, ranging from the type and number of sensors. For example, "recipe descriptions" and "user comments" are examples of natural language information. Similarly, "photos of the location where the problem is to be solved" and "photos of the site where the equipment is installed" are examples of image information. Natural language information and image information are each converted into multidimensional vectors through separate processes. Natural language information is converted into vectors by mapping it to a latent space, such as a topic model. Image information is converted into vectors using a convolutional neural network (CNN) or similar. Finally, each multidimensional vector is combined to create a multidimensional vector. However, this cannot be used for sparseness as is, so dimensionality is reduced using techniques such as principal component analysis (PCA).

[0042] Furthermore, since the features are converted into multidimensional vectors, the similarity can be calculated simply from the distance of the multidimensional vectors. For example, the similarity can be found from the cosine similarity of the multidimensional vectors of each feature.

[0043] The solution (recipe) also includes abstracted properties and setting values ​​of the electronic device 100 that realizes the functions of a sensor or actuator. The control device 400 provides a place where recipes can be stored on the cloud CL, and defines the similarity between recipes by analyzing the stored recipes and converting them into features. In such a place, users can search for recipes using text, voice, and photos. Furthermore, if the control device 400 cannot find a recipe that matches the user's request, it automatically generates and suggests a recipe. In this case, the control device 400 utilizes the features of the recipe to automatically generate a recipe that meets the user's diverse needs.

[0044] If the data acquired from a recipe can be used as a recipe feature, the accuracy of suggestions will improve as each user continues to use the recipe, making it possible to infer the true purpose and make suggestions.

[0045] Furthermore, if information linked to a user can be used as a feature of a recipe, the state and situation of each user can be grasped using sensors, making it possible to make suggestions tailored to the number and type of sensors each user possesses. Hereinafter, information linked to a user may also be referred to as "user information."

[0046] Furthermore, if the correlation between the real world and sensors can be used as a feature of a recipe, it will be possible to reflect combinations with objects that exist in the real world in recipe suggestions.

[0047] If the recipe has few features (for example, if there is no recipe explanation), it may be possible to utilize data obtained by program operation.

[0048] In this embodiment, a solution (recipe) includes a program and accompanying information. The program is essential. The accompanying information is preferable but not essential.

[0049] The program is a dedicated program for controlling the electronic device 100, which is an electronic block. In this embodiment, the program is expressed as a combination of blocks in block programming. Note that block programming is merely an example. In reality, the program may be expressed using graphics in visual programming other than block programming, source code in text programming, or prompts from a program generation AI. The program also includes information such as block IDs, block types, block names, block settings, block locations (GUI (Graphical User Interface)), connections between blocks, and recipe names / descriptions. In reality, external linkage blocks other than electronic blocks exist, but the program does not include behavior outside the system, such as the behavior of a server to which data is sent.

[0050] The additional information includes a description of the solution (including examples of licenses, applications, recombinations, settings, etc.), additional information required for the HW (hardware) / SW (software) of external services or devices (materials, assembly methods, additional programs), raw data obtained using the recipe and its processing and analysis, and information related to the actual environment (photos, recordings, videos, etc. during installation).

[0051] (2-1-1. Recipe Image) An image of a recipe according to an embodiment of the present disclosure will be described with reference to Fig. 5. Fig. 5 is a diagram showing an image of a recipe according to an embodiment of the present disclosure.

[0052] For example, as shown in FIG. 5 , the search tag “Brightness Block” is displayed near the recipe title “Exhibition Stand that Automatically Starts Explanation.” Below that, an image of the exhibition stand and a description of the solution (recipe) are displayed. Next, the electronic blocks to be used and other materials to be used are displayed as hardware materials. In this example, “Brightness Block” is displayed as the electronic block to be used. Other materials to be used, such as wood (4mm MDF), glue, a power cable, and a power AC adapter, are also displayed. Next, the instructions are displayed as information on how to assemble and install the hardware. For each step of the instructions, “images of the actual environment (examples)” are displayed as examples of the actual environment, along with an explanation of the steps involved. Note that the images of the actual environment (examples) may be videos. Next, a dedicated program is displayed. In practice, additional information such as setting parameters, options, comments, and actual data during operation associated with the program may also be displayed. Information about the poster (recipe creator) and ratings and comments on the recipe may also be displayed.

[0053] (2-2. Recipe Posting) An overview of recipe posting will be described with reference to Fig. 6. Fig. 6 is a flowchart showing a processing procedure for posting a recipe according to an embodiment of the present disclosure.

[0054] 6 , for example, a user creates a recipe on the terminal device 200 (step S101). Then, the user posts the recipe on a sharing site (step S102). At this time, the terminal device 200 generates a recipe in response to the user's operation and transmits the recipe to a sharing site prepared by the control device 400 on the cloud CL via the relay device 300.

[0055] Next, the control device 400 on the cloud CL determines whether the posted recipe is genuine and correct (step S103). That is, the control device 400 on the cloud CL checks whether the posted recipe is valid as a recipe. At this time, if the recipe is malicious or incorrect, the control device 400 on the cloud CL discards the recipe without storing (registering) it in the database DB of the sharing site. In this way, the control device 400 on the cloud CL eliminates recipes including harmful or fraudulent programs, recipes including inoperable programs, recipes unrelated to the information processing system 10, and useless information disguised as recipes. Conversely, if the recipe is genuine and correct, the control device 400 on the cloud CL stores the recipe in the database DB of the sharing site and notifies the user that the recipe has been posted on the sharing site (step S104). Note that recipes stored in the database DB of the sharing site are assumed to be posted on the sharing site. The database DB of the shared site may be the control device 400 on the cloud CL (or a storage device mounted on or connected to the control device 400), or may be a device that exists independently from the control device 400.

[0056] Next, the user checks the recipe posted on the sharing site on the terminal device 200 (step S105). At this time, the terminal device 200 receives a notification from the control device 400 on the cloud CL that the recipe has been posted on the sharing site, and notifies the user by displaying the recipe on the screen, etc., and displays the recipe posted on the sharing site on the screen in response to an operation by the user.

[0057] In this way, users can post recipes on the sharing site. Users can also view recipes posted by themselves and other users. Users can also easily execute posted recipes at home. At this time, users can also perform simulations based on posted recipes.

[0058] Furthermore, when a user takes a predetermined action on the sharing site, a recipe template is provided. For example, when a user answers questions from the sharing site, a recipe template is provided that corresponds to the content of the answer. When a user sends a photo taken with a camera, a recipe template is provided that corresponds to the content of the photo. When a user sends content written in natural language, a recipe template is provided that corresponds to the content of the description. When a user sends instructions by voice, a recipe template that corresponds to the instructions is provided. The recipe templates provided are personalized for each user.

[0059] (2-3. Recipe Analysis) An overview of recipe analysis will be described with reference to Fig. 7. Fig. 7 is a flowchart showing a processing procedure for recipe analysis according to an embodiment of the present disclosure.

[0060] 7 , for example, the control device 400 on the cloud CL analyzes a recipe and extracts features (step S201). Subsequently, the control device 400 on the cloud CL calculates each parameter using the extracted features (step S202). The control device 400 on the cloud CL then stores the calculated parameters in a database DB of the shared site (step S203). The control device 400 on the cloud CL may associate the calculated parameters with the recipe and store them in the same database DB as the recipe, or in a database DB separate from the recipe.

[0061] 2-4. Preparation before Recipe Creation An overview of preparation before recipe creation will be described with reference to Fig. 8. Fig. 8 is a flowchart showing a processing procedure of preparation before recipe creation according to an embodiment of the present disclosure.

[0062] For example, as shown in Fig. 8, the control device 400 on the cloud CL acquires an arbitrary recipe from the database DB of the sharing site (step S301). In the example shown in Fig. 8, the control device 400 on the cloud CL acquires the original recipe "org" from the database DB of the sharing site. Then, the control device 400 on the cloud CL sets the acquired recipe as the initial original recipe (step S302).

[0063] Next, the control device 400 on the cloud CL randomly changes or adds to a portion of the original recipe to create a processing recipe (step S303). In the example shown in FIG. 8, the control device 400 on the cloud CL randomly changes or adds to a portion of the original recipe "org" to create a processing recipe "oag". Then, the control device 400 on the cloud CL learns how to restore the original recipe from the processing recipe (step S304). In the example shown in FIG. 8, the control device 400 on the cloud CL learns how to restore the original recipe "org" from the processing recipe "oag". Then, the control device 400 on the cloud CL determines whether the above learning has been repeated a specified number of times (step S305). If the above learning has not been repeated a specified number of times (step S305; No), the control device 400 on the cloud CL determines whether the recipe has been left blank (step S306). If the recipe is not blank (step S306; No), the control device 400 on the cloud CL sets the processing recipe as the next new original recipe (step S307), and then returns to step S303.

[0064] For example, the control device 400 on the cloud CL randomly changes or adds to a part of the original recipe to create processing recipe 1, and then learns how to restore the original recipe from processing recipe 1. Next, the control device 400 on the cloud CL randomly changes or adds to a part of processing recipe 1 to create processing recipe 2, and then learns how to restore processing recipe 1 from processing recipe 2. Next, the control device 400 on the cloud CL randomly changes or adds to a part of processing recipe 2 to create processing recipe 3, and then learns how to restore processing recipe 2 from processing recipe 3. Thereafter, this process is repeated a specified number of times.

[0065] Next, when the control device 400 on the cloud CL has repeated the above learning a specified number of times (step S305; Yes), or when the recipe has been blanked before that (step S306; Yes), the control device 400 on the cloud CL determines whether the series of processes has been repeated a specified number of times (step S308). If the series of processes has not been repeated a specified number of times (step S308; No), the control device 400 on the cloud CL returns to step S301. That is, the control device 400 on the cloud CL obtains another recipe from the database DB of the shared site and repeats the series of processes up to the specified number of times.

[0066] Then, when the control device 400 on the cloud CL repeats the series of processes a specified number of times (step S308; Yes), it obtains a recipe generation rule (step S309). As a result, the control device 400 on the cloud CL becomes able to generate a new recipe based on the recipe generation rule. In this embodiment, the control device 400 on the cloud CL trains a diffusion model (e.g., U-Net) using a technique such as stable diffusion as described above, and generates a new recipe using this diffusion model. However, stable diffusion is merely an example. In practice, the control device 400 on the cloud CL may train a generative model using generative adversarial networks (GAN) and generate a new recipe using this generative model.

[0067] 2-5. Recipe Search An overview of recipe search will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the procedure for recipe search according to an embodiment of the present disclosure.

[0068] 9, the terminal device 200 determines whether or not there is input information such as a photo, audio, video, or text information, or any combination thereof (step S401). For example, when the terminal device 200 receives input information such as a photo, audio, video, or text information, or any combination thereof, related to a problem to be solved (or a location or situation where the problem exists) from the user, the terminal device 200 determines that there is input information.

[0069] When the terminal device 200 has input information such as a photo, audio, video, or text information, or any combination thereof (step S401; Yes), the terminal device 200 searches for a recipe using the input information (step S402). For example, the terminal device 200 transmits a recipe search request based on the query to the control device 400 on the cloud CL via the relay device 300, using the input information as a query.

[0070] On the other hand, if there is no input information such as a photo, audio, video, or text information, or any combination thereof (step S401; No), the terminal device 200 determines whether the user has selected a recipe category (step S403). For example, when the terminal device 200 receives a selection of a recipe category from the user, it determines that the user has selected a recipe category.

[0071] When the user selects a recipe category (step S403; Yes), the terminal device 200 searches for recipes from the category (step S404). For example, the terminal device 200 transmits a recipe search request based on the query, using the category as a query, to the control device 400 on the cloud CL via the relay device 300. Conversely, when there is no input information and the user has not selected a recipe category (step S403; No), the terminal device 200 does nothing. Note that in practice, different processing may be added.

[0072] Next, when a recipe search is performed from the terminal device 200, the control device 400 on the cloud CL extracts feature quantities of the recipe desired by the user from the input information or category (step S405). Note that the feature quantities of the recipe desired by the user may be feature quantities of a problem and its solution estimated from the results of analyzing the input information. Then, the control device 400 on the cloud CL searches the database DB of the sharing site for highly similar recipes based on the extracted feature quantities of the recipe (step S406). Note that a highly similar recipe refers to a recipe whose similarity is equal to or greater than a threshold value.

[0073] At this time, the control device 400 on the cloud CL checks whether there is a highly similar recipe (step S407). For example, the control device 400 on the cloud CL vectorizes the feature quantities of the extracted recipe and the feature quantities of the recipes stored in the database DB, determines whether the cosine similarity of each vectorized data is equal to or greater than a threshold, and if the cosine similarity is equal to or greater than the threshold, determines that the recipe stored in the database DB is a highly similar recipe. Note that, although the above describes a case where the determination is made using cosine similarity, in practice, the determination may be made using the similarity or distance of the vectorized data without using cosine similarity. In other words, the method for calculating the similarity is not limited, and any method, including known methods, may be used.

[0074] If there are highly similar recipes (step S407; Yes), the control device 400 on the cloud CL selects the highly similar recipes as suggested recipes to be proposed to the user (step S408). At this time, the control device 400 on the cloud CL may select all highly similar recipes obtained as a result of the search as suggested recipes, or may select a predetermined number of them (for example, the top 10) as suggested recipes.

[0075] Conversely, if there is no highly similar recipe (step S407; No), the control device 400 on the cloud CL generates a new highly similar recipe (step S409). For example, the control device 400 on the cloud CL generates a new recipe using the diffusion model trained in the preparation before recipe generation shown in Fig. 8 above. Then, the control device 400 on the cloud CL stores the generated recipe in the database DB of the shared site (step S410).

[0076] In practice, the control device 400 on the cloud CL may generate a new highly similar recipe regardless of whether or not there is a highly similar recipe. For example, if there is a highly similar recipe (step S407; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a proposed recipe (step S408) and then generate a new highly similar recipe (step S409). In other words, the control device 400 on the cloud CL may set both the selected highly similar recipe and the newly generated highly similar recipe as proposed recipes.

[0077] Alternatively, if there is a highly similar recipe (step S407; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a basic recipe (step S408) and generate a more similar recipe based on the selected basic recipe (step S409). For example, the control device 400 on the cloud CL may generate a recipe customized from a highly similar recipe to suit the user's problem or situation.

[0078] Next, the control device 400 on the cloud CL proposes recipes with high similarity to the user (step S411). That is, the control device 400 on the cloud CL notifies the terminal device 200 of the recipes with high similarity via the relay device 300.

[0079] The terminal device 200 uses the recipe proposed by the control device 400 on the cloud CL as is, either in response to a user operation or automatically (step S412). For example, the terminal device 200 installs a program included in the proposed recipe into the electronic device 100. Alternatively, the terminal device 200 controls the electronic device 100 based on the proposed recipe. In practice, the terminal device 200 may process and use the recipe proposed by the control device 400 on the cloud CL, as necessary. In other words, the terminal device 200 may customize or arrange the recipe to suit the user's wishes or environment.

[0080] 2-5-1. Recipe Search Concept An image of a recipe search will be described with reference to FIG. 10. FIG. 10 is a diagram showing an image of a recipe search. As shown in FIG. 10 (A), a user can search for (or create) a recipe using photos, audio, text, chat, etc. In this case, the user can search by problem or assignment, rather than by which electronic block to use. Alternatively, the user can search for (or create) a recipe by selecting a thumbnail or icon that indicates a recipe category.

[0081] For example, when searching for a recipe using a photo, as shown in FIG. 10B, the user takes a photo of the scene related to the "problem" and attaches the photo, which results in a list of similar recipes and generated recipes. At this time, the control device 400 on the cloud CL searches for or generates recipes with high similarity based on the taken photo, lists them, and proposes them to the user. The user selects and acquires the desired recipe from the listed recipes. Note that if the recipe requires payment, the user purchases it. At this time, the user can also obtain the recipe as visual information or audio information.

[0082] 2-6. Example 1: Data Obtained from a Recipe The control device 400 on the cloud CL analyzes data obtained from a recipe, enabling it to propose a recipe based on an inferred true purpose. Furthermore, the more the system is used, the more it fits the user's purpose, and the more accurate the inferences and recipe proposals become.

[0083] Also, the XY problem can be partially solved by "asking about a method for solving a secondary problem Y, rather than directly asking about the problem X that the questioner really wants to solve." For example, it can be partially solved to ask about a method for getting the last three characters (Y) of a file name in order to get the file extension (X) (the extension is not necessarily three characters).

[0084] In this embodiment, the control device 400 on the cloud CL infers a new objective X' from the acquired data of recipe A corresponding to objective X, and generates a corresponding recipe proposal. For example, the control device 400 on the cloud CL analyzes the data of recipe A generated for objective X "to measure vibration." Then, the control device 400 on the cloud CL infers a new objective X' "abnormal value detection" from the data trend, and generates a recipe proposal.

[0085] 2-6-1. Recipe Suggestion Improvement Through Continued Use An overview of recipe suggestion improvement through continued use will be described with reference to Fig. 11. Fig. 11 is a flowchart showing a processing procedure for recipe suggestion improvement through continued use according to an embodiment of the present disclosure.

[0086] For example, as shown in FIG. 11 , the terminal device 200 searches for a recipe using recipe data registered by a user (step S501). For example, the terminal device 200 sends a recipe search request based on the query to the control device 400 on the cloud CL via the relay device 300, using the recipe data registered by the user as a query. The terminal device 200 searches for a recipe not only when the user explicitly uploads recipe data, but also when the terminal device 200 collects recipes with the user's permission. Recipes also include the results of program operation. Recipes may include (1) a blank recipe program but with a title, description, comments, etc., or (2) a recipe may be created to acquire raw data, or the acquired raw data may be processed within or outside the recipe. Furthermore, in addition to actually running a program to acquire raw data, a simulator may be run to acquire virtual data.

[0087] Next, when a recipe is searched for from the terminal device 200, the control device 400 on the cloud CL stores and extracts recipe features based on the recipe data registered by the user (step S502). The recipe data is stored and updated in the database DB on the sharing site as recipe features. Examples of recipe data include sensor measurement data indicating environmental conditions and changes, measurement data used to control output, etc., descriptions, audio, photos, and videos of the recipe usage environment, and metrics such as recipe scheduling / trigger execution.

[0088] The control device 400 on the cloud CL then searches the database DB of the sharing site for recipes with high similarity based on the extracted recipe features (step S503). Both "purpose" and "true purpose" are features linked to recipes. The similarity determination logic is not determined by a human but by a set of parameters derived from data.

[0089] At this time, the control device 400 on the cloud CL checks whether there is a highly similar recipe (step S504). If there is a highly similar recipe (step S504; Yes), the control device 400 on the cloud CL selects the highly similar recipe as a suggested recipe to be proposed to the user (step S505). Conversely, if there is no highly similar recipe (step S504; No), the control device 400 on the cloud CL generates a new highly similar recipe (step S506). Then, the control device 400 on the cloud CL stores the generated recipe in the database DB of the shared site (step S507).

[0090] In practice, the control device 400 on the cloud CL may generate a new highly similar recipe regardless of whether or not there is a highly similar recipe. For example, if there is a highly similar recipe (step S504; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a proposed recipe (step S505) and then generate a new highly similar recipe (step S506). In other words, the control device 400 on the cloud CL may set both the selected highly similar recipe and the newly generated highly similar recipe as proposed recipes.

[0091] Alternatively, if there is a highly similar recipe (step S504; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a basic recipe (step S505) and generate a more similar recipe based on the selected basic recipe (step S506). For example, the control device 400 on the cloud CL may generate a recipe customized from a highly similar recipe to suit the user's problem or situation.

[0092] Next, the control device 400 on the cloud CL proposes recipes with high similarity to the user (step S508). That is, the control device 400 on the cloud CL notifies the terminal device 200 of the recipes with high similarity via the relay device 300.

[0093] The terminal device 200 uses the recipe proposed by the control device 400 on the cloud CL as is, in response to a user operation or automatically (step S509).

[0094] 2-7. Example 2: Information linked to a user The control device 400 on the cloud CL can propose recipes that are tailored to information linked to a user. For example, the control device 400 on the cloud CL takes into account the electronic blocks (electronic devices 100) owned by the user and proposes recipes that can be realized with the electronic blocks on hand.

[0095] For example, if the control device 400 on the cloud CL has a recipe that uses a play / stop button, but the user does not have a button block, the control device 400 proposes a recipe that substitutes a brightness block for the button block. The button block is an electronic block with a button function. The brightness block is an electronic block with a function that turns on / off when the user holds their hand over it.

[0096] Furthermore, the control device 400 on the cloud CL edits the purpose field of the door open / close detection recipe information to "for refrigerator." For example, a motion block configured to capture only the rising edge of the door when it is opened is proposed. The motion block is an electronic block with the function of detecting motion. Furthermore, when using a brightness sensor in a room that detects changes in brightness due to the opening and closing of the refrigerator door, specifying the situation may be meaningful because the brightness threshold varies depending on the location, sunlight / lighting conditions, etc.

[0097] Furthermore, the control device 400 on the cloud CL registers disabilities such as paralysis as physical characteristics of the user in the user's profile. For example, the control device 400 on the cloud CL proposes recipes using movement blocks configured for the user, with input operations being movements that the user is capable of, such as setting input to a specific movement. Furthermore, something like "personal setting values" can be separately registered and made adjustable on the adjustment screen. Furthermore, when not registered, general values, including those of other users, are temporarily reflected.

[0098] 2-7-1. Improving Recipe Suggestions Using User Information An overview of improving recipe suggestions using user information will be described with reference to Fig. 12. Fig. 12 is a flowchart showing a processing procedure for improving recipe suggestions using user information according to an embodiment of the present disclosure.

[0099] For example, as shown in FIG. 12 , the terminal device 200 determines whether or not there is user information registered by the user (step S601). For example, when the terminal device 200 accepts registration of user information from the user, it determines that there is user information registered by the user. The user information also includes usage block information specified by the user. The usage block information is, for example, information about electronic blocks (electronic devices 100) owned (used) by the user. In addition to automatically taking into account the user's personal information based on purchase history / product registration, usage block information can also be specified using search criteria, such as electronic blocks available to the user but owned by someone other than the user (family members, colleagues, etc.), or electronic blocks used by someone other than the user.

[0100] If the terminal device 200 has user information registered by the user (step S601; Yes), it searches for a recipe based on the user information (step S602). For example, the terminal device 200 transmits a recipe search request based on the query to the control device 400 on the cloud CL via the relay device 300, using the user information as a query. Conversely, if the terminal device 200 has not registered user information by the user (step S601; No), it does nothing. In practice, a different process may be added.

[0101] Next, when a recipe is searched for from the terminal device 200, the control device 400 on the cloud CL stores and extracts recipe features based on the user information (step S603). For example, the user information is anonymized and becomes one of the features of the recipe data. Examples of user information include the number and type of blocks owned (sensors, actuators, etc.), items for self-optimization such as physical limitations (height, disabilities), and favorite settings (such as "exterior color"). Then, the control device 400 on the cloud CL searches the database DB of the sharing site for recipes with high similarity based on the extracted recipe features (step S604).

[0102] At this time, the control device 400 on the cloud CL checks whether there is a highly similar recipe (step S605). If there is a highly similar recipe (step S605; Yes), the control device 400 on the cloud CL selects the highly similar recipe as a suggested recipe to be proposed to the user (step S606). Conversely, if there is no highly similar recipe (step S605; No), the control device 400 on the cloud CL generates a new highly similar recipe (step S607). Then, the control device 400 on the cloud CL stores the generated recipe in the database DB of the shared site (step S608).

[0103] In practice, the control device 400 on the cloud CL may generate a new highly similar recipe regardless of whether or not there is a highly similar recipe. For example, if there is a highly similar recipe (step S605; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a proposed recipe (step S606) and then generate a new highly similar recipe (step S607). In other words, the control device 400 on the cloud CL may set both the selected highly similar recipe and the newly generated highly similar recipe as proposed recipes.

[0104] Alternatively, if there is a highly similar recipe (step S605; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a basic recipe (step S606) and generate a more similar recipe based on the selected basic recipe (step S607). For example, the control device 400 on the cloud CL may generate a recipe customized from a highly similar recipe to suit the user's problem or situation.

[0105] Next, the control device 400 on the cloud CL proposes recipes with high similarity to the user (step S609). That is, the control device 400 on the cloud CL notifies the terminal device 200 of the recipes with high similarity via the relay device 300.

[0106] The above example corresponds to recipe generation when no highly similar recipes are found (i.e., no highly similar recipes exist). For example, the control device 400 on the cloud CL has a recipe that uses a play / stop button, but if the user does not have a button block, it proposes a recipe that substitutes a brightness block that reacts when the user holds their hand over it instead of a button. Also, in the information for a door opening / closing detection recipe, the purpose field is edited to "for refrigerator" and a movement block configured to capture only the rising edge of the door when it is opened is proposed. Furthermore, by registering a disability such as paralysis in the user's profile, movement blocks tailored to that person are proposed so that input can be made into specific movements.

[0107] The terminal device 200 uses the recipe proposed by the control device 400 on the cloud CL as is, in response to a user operation or automatically (step S610).

[0108] [2-8. Example 3: Association between the Real World and Sensors] The control device 400 on the cloud CL utilizes the association between the real world and sensors to reflect the combination of sensors and objects in recipe suggestions. A sensor is an electronic block (electronic device 100) that has a sensor function. Note that the sensor is merely an example. In reality, the electronic block may have a function other than a sensor, such as an actuator. An "object" is an object that exists in the real world and is detected by a sensor mounted on or connected to the terminal device 200.

[0109] For example, when the terminal device 200 transmits a photo of a door to the control device 400 on the cloud CL via the relay device 300, the control device 400 on the cloud CL proposes a door open / close detection recipe and suggests setting values ​​in the recipe. For example, when the terminal device 200 transmits a request to the control device 400 on the cloud CL via the relay device 300 to "install this sensor," sending a link and model number of a new sensor, the control device 400 on the cloud CL also proposes the placement and connection method of the new sensor. It is assumed that unknown information will be confirmed interactively by AI or cited from an internet search.

[0110] 2-8-1. Recipe proposal improvement based on the association between the real world and sensors An overview of recipe proposal improvement based on the association between the real world and sensors will be described with reference to Fig. 13. Fig. 13 is a flowchart showing a processing procedure for improving recipe proposal based on the association between the real world and sensors according to an embodiment of the present disclosure.

[0111] 13 , the user adds information describing the association (i.e., relationship) between the real world and the sensor on the terminal device 200 and searches for a recipe (step S701). For example, the terminal device 200 uses the information describing the association between the real world and the sensor added by the user as a query, and transmits a recipe search request based on the query to the control device 400 on the cloud CL via the relay device 300. The information describing the association between the real world and the sensor is expressed in the form of text, audio, images (photographs, diagrams, etc.), video, etc.

[0112] Next, when a recipe search is performed from the terminal device 200, the control device 400 on the cloud CL stores and extracts features of the recipe based on information added by the user that explains the relationship between the real world and the sensor (step S702). For example, the control device 400 on the cloud CL obtains, as the analysis results of the input image, information such as "detection target: person, shoes, passing," "installation: wall, floor, corner of room," "sensor: human presence sensor," and "environment: wide hallway with heavy pedestrian traffic, with flow in multiple lanes," and converts the input information into features based on this information.

[0113] Then, the control device 400 on the cloud CL searches the database DB of the sharing site for recipes with high similarity based on the extracted recipe feature quantities (step S703). In this embodiment, since it is possible to make a judgment taking into account signals from electronic blocks of sensors and the like as feature quantities, it is possible to make a judgment that incorporates more real-world information in real time than a simple LLM (Large Scale Language Model).

[0114] At this time, the control device 400 on the cloud CL checks whether there is a highly similar recipe (step S704). If there is a highly similar recipe (step S704; Yes), the control device 400 on the cloud CL selects the highly similar recipe as a suggested recipe to be proposed to the user (step S705). Conversely, if there is no highly similar recipe (step S704; No), the control device 400 on the cloud CL generates a new highly similar recipe (step S706). Then, the control device 400 on the cloud CL stores the generated recipe in the database DB of the shared site (step S707).

[0115] In practice, the control device 400 on the cloud CL may generate a new highly similar recipe regardless of whether or not there is a highly similar recipe. For example, if there is a highly similar recipe (step S704; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a proposed recipe (step S705) and then generate a new highly similar recipe (step S706). In other words, the control device 400 on the cloud CL may set both the selected highly similar recipe and the newly generated highly similar recipe as proposed recipes.

[0116] Alternatively, if there is a highly similar recipe (step S704; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a basic recipe (step S705) and generate a more similar recipe based on the selected basic recipe (step S706). For example, the control device 400 on the cloud CL may generate a recipe customized from a highly similar recipe to suit the user's problem or situation.

[0117] Next, the control device 400 on the cloud CL proposes recipes with high similarity to the user (step S708). That is, the control device 400 on the cloud CL notifies the terminal device 200 of the recipes with high similarity via the relay device 300.

[0118] The terminal device 200 uses the recipe proposed by the control device 400 on the cloud CL as is, in response to a user operation or automatically (step S709).

[0119] 2-9. Example 4: Recipe proposal using accumulated data and a program An overview of recipe proposal using accumulated data and a program will be described with reference to Fig. 14. Fig. 14 is a diagram showing an overview of recipe proposal using accumulated data and a program.

[0120] For example, as shown in FIG. 14 , a user may have program A and data A that they created but have not published as recipes. Furthermore, other users may have published recipes B, C, and D. Recipe B includes program B and data B. Recipe C includes program C and data C. Recipe D includes program D and data D. When program A, which controls an electronic block having a sensor or other function, is executed in a user's terminal device 200, data A acquired from the sensor or other function is accumulated. The control device 400 on the cloud CL acquires program A created by the user and the accumulated data A and calculates the similarity between the program and data combinations based on the feature values. For example, the control device 400 on the cloud CL extracts the feature values ​​of the combination of program A and data A, and calculates the similarity between the feature values ​​of the combination of program B and data B, the feature values ​​of the combination of program C and data C, and the feature values ​​of the combination of program D and data D. The control device 400 on the cloud CL then suggests recipes with high similarity to the user.

[0121] The acquired data A is handled only within the system and is not made public between users. In other words, a user's data A remains undisclosed and cannot be referenced or used by other users, unlike data B, C, and D that are made public as recipes. The same applies to program A.

[0122] In this way, the control device 400 on the cloud CL suggests other recipes that a user is likely to use based on the program created by that user and the accumulated data. In a preliminary stage, the programs and data created by each user who is a recipe creator are stored in the control device 400 on the cloud CL of the information processing system 10. In addition, some users are recipe creators who publish "recipes" that add installation methods, photos of the installation, and explanations of what can be done to the programs they created, and the information on these recipes is also stored in the control device 400 on the cloud CL.

[0123] The control device 400 on the cloud CL calculates the similarity between a set of a user's program and accumulated data and those of other users, and proposes public recipes with programs and data that have similarities to the user as recommended recipes. Alternatively, it identifies another user (a user with similar tastes) whose recipes are highly similar to the user's program and data from multiple programs and data, and proposes recipes created by that other user to the user.

[0124] Typically, users do not necessarily create detailed instructions or recipes, such as installation methods, for the programs they create. Many users simply create and use their own programs, but do not go as far as creating and publishing recipes. In such cases, the program alone cannot obtain information related to the installation situation or real-world physical information, such as how the program is installed. Since the system does not know the user's situation or environment, the accuracy of recipe suggestions cannot be improved. On the other hand, because data accumulated from sensors and other sources is related to real-world information, the accuracy of recipe suggestions can be improved by calculating the similarity between the program and accumulated data as a set.

[0125] If there is a similar recipe while the user is creating a program, the user can receive a recipe suggestion from the control device 400 on the cloud CL. For example, the user is notified of a suggested recipe with similar features to the program in a modal / pop-up / dialog on the screen while creating the program.

[0126] 2-10. Example 5: Data Obtained by Feedback In any of the above examples, after receiving a recipe proposal from the control device 400 on the cloud CL, the user can provide feedback on the proposed recipe. For example, the user provides feedback to the control device 400 on the cloud CL about the results of implementing the proposed recipe or the on-site environment or situation by text, audio, images (photographs, diagrams, etc.), video, etc.

[0127] The control device 400 on the cloud CL recognizes user feedback, explanations, and accompanying information such as on-site images, audio, and video as features and generates improvement proposals in the form of recipes and explanations. For example, the control device 400 on the cloud CL recognizes the environment in which the electronic sensor blocks are placed and provides advice on appropriate installation methods. The control device 400 on the cloud CL also warns if communication conditions are unstable based on the radio wave strength received by the electronic sensor blocks. The control device 400 on the cloud CL also provides advice on installation methods and locations and troubleshooting based on images of the installed state of the electronic sensor blocks. When the terminal device 200 specifies the desired installation location using an image and sends the image to the control device 400 on the cloud CL via the relay device 300, the control device 400 on the cloud CL proposes an on-demand block with a just-right exterior. When the terminal device 200 displays a video of the electronic block and the object or location to be fixed, the control device 400 on the cloud CL proposes a fixing method.

[0128] 2-10-1. Improving Recipe Suggestions Through Feedback An overview of improving recipe suggestions through feedback will be described with reference to Fig. 15. Fig. 15 is a flowchart showing a processing procedure for improving recipe suggestions through feedback according to an embodiment of the present disclosure.

[0129] 15 , in the terminal device 200, a user provides feedback on a recipe proposed by the control device 400 on the cloud CL (step S801). For example, the terminal device 200 transmits information regarding the feedback on the proposed recipe as a query to the control device 400 on the cloud CL via the relay device 300, and transmits a recipe search request based on the query. Then, the control device 400 on the cloud CL stores and extracts feature quantities of the recipe based on the information regarding the feedback on the recipe from the terminal device 200.

[0130] Next, when feedback on the recipe is provided from the terminal device 200, the control device 400 on the cloud CL stores and extracts the feature quantities of the recipe based on information on the feedback on the recipe from the user (step S802).

[0131] Then, the control device 400 on the cloud CL searches the database DB of the sharing site for recipes with high similarity based on the extracted recipe feature quantities (step S803). In this embodiment, since it is possible to make a judgment taking into account signals from electronic blocks of sensors and the like as feature quantities, it is possible to make a judgment that incorporates more real-world information in real time than a simple LLM (Large Scale Language Model).

[0132] At this time, the control device 400 on the cloud CL checks whether there is a highly similar recipe (step S804). If there is a highly similar recipe (step S804; Yes), the control device 400 on the cloud CL selects the highly similar recipe as a suggested recipe to be proposed to the user (step S805). Conversely, if there is no highly similar recipe (step S804; No), the control device 400 on the cloud CL generates a new highly similar recipe (step S806). Then, the control device 400 on the cloud CL stores the generated recipe in the database DB of the shared site (step S807).

[0133] In practice, the control device 400 on the cloud CL may generate a new highly similar recipe regardless of whether or not there is a highly similar recipe. For example, if there is a highly similar recipe (step S804; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a proposed recipe (step S805) and then generate a new highly similar recipe (step S806). In other words, the control device 400 on the cloud CL may set both the selected highly similar recipe and the newly generated highly similar recipe as proposed recipes.

[0134] Alternatively, if there is a highly similar recipe (step S804; Yes), the control device 400 on the cloud CL may select the highly similar recipe as a basic recipe (step S805) and generate a more similar recipe based on the selected basic recipe (step S806). For example, the control device 400 on the cloud CL may generate a recipe customized from a highly similar recipe to suit the user's problem or situation.

[0135] Next, the control device 400 on the cloud CL proposes recipes with high similarity to the user (step S808). That is, the control device 400 on the cloud CL notifies the terminal device 200 of the recipes with high similarity via the relay device 300.

[0136] The terminal device 200 uses the recipe proposed by the control device 400 on the cloud CL as is, in response to a user operation or automatically (step S809).

[0137] In this way, the control device 400 on the cloud CL recognizes user feedback and accompanying information such as explanations, on-site images, audio, and video as features, and generates improvement suggestions in the form of recipes and explanations. For example, the control device 400 on the cloud CL recognizes the environment in which the electronic sensor blocks are placed and provides advice on appropriate installation methods. It also warns users if the communication status is unstable based on the radio wave strength received by the electronic sensor blocks. It also provides advice on installation methods and locations and troubleshoots based on images of the installed state of the electronic sensor blocks. It also suggests on-demand blocks with exteriors that are just the right size by specifying the desired installation location in an image. It also suggests installation methods based on images of the electronic block and the object or location to be fixed.

[0138] 3. Simulation System Not Requiring Electronic Device Main Unit 3-1. Realization of Simulation System Until now, before purchasing an electronic block (electronic device 100) with functions such as a sensor or actuator, a user had to select an electronic block that was suitable for their intended use based on past experience and specification information such as a data sheet. As a result, users who did not have sufficient knowledge of the types and functions of electronic blocks and had little experience could take a considerable amount of time to select an electronic block. Furthermore, when users actually used the electronic block they purchased, it sometimes turned out to be operating differently from what they had expected.

[0139] Therefore, this disclosure realizes a system configuration that allows the system configuration between the cloud and electronic blocks with functions such as sensors and actuators to be mirrored within an app that can launch a simulation mode and used for simulation. For example, the terminal device 200 running the app acquires sensor values ​​from cameras, LiDAR, ToF sensors, IMUs, etc., and then pseudo-replaces the acquired sensor values ​​with data from electronic blocks with functions such as sensors and actuators. This data can then be used to simulate a program included in a recipe within the app. Furthermore, a spatial UI representation is devised to enable intuitive understanding of the data.

[0140] Specifically, by combining with AR (Augmented Reality) and devising UI display in space, even if the electronic blocks themselves, which have functions such as sensors and actuators, are not at hand, it is possible to arrange blocks in AR according to the usage scenario and perform a simulation as if operating the real thing. Note that AR is just one example. In practice, VR (Virtual Reality) or MR (Mixed Reality) may be used instead of AR. Furthermore, if an actual electronic block (real block) having functions such as a sensor or actuator is paired, turned on, and active, the information obtained in the simulation can be linked and driven.

[0141] In this embodiment, the terminal device 200 is assumed to be a smart device such as a smartphone or tablet terminal capable of AR / VR display, or an AR goggle such as a head-mounted display (HMD) or smart glasses capable of AR / VR display.

[0142] 3-2. Overview of Simulation Implementation An overview of simulation implementation will be described with reference to Fig. 16. Fig. 16 is a flowchart showing the processing procedure for implementing a simulation according to an embodiment of the present disclosure.

[0143] For example, as shown in FIG. 16 , the terminal device 200 determines whether a recipe has been proposed by the control device 400 on the cloud CL (step S901). If a recipe has been proposed by the control device 400 on the cloud CL (step S901; Yes), the terminal device 200 uses the proposed recipe (step S902). Conversely, if a recipe has not been proposed by the control device 400 on the cloud CL (step S901; No), the terminal device 200 uses a recipe (which may be a program) created by the user (step S903). At this time, the user may create a recipe, or may select a recipe that the user desires to simulate from among multiple recipes created by the user. Furthermore, the terminal device 200 may use only a program, rather than a recipe.

[0144] Next, the terminal device 200 starts the simulation mode using the recipe (step S904). At this time, the user may press a button to start the simulation mode. The button may be an ON-OFF switch or a slide switch. The start of the simulation mode can be recognized by a trigger on the UI, such as the user turning on a switch in the app, clicking an AR view icon, or starting by voice. The simulation mode can also be started by receiving a request from on-demand or recipe sharing.

[0145] Next, the user photographs the space using the in-app camera on the terminal device 200 and performs space recognition processing (step S905). At this time, the terminal device 200 may photograph the space using the camera of the sensor unit 220. Note that the space is the location where the electronic block is to be placed.

[0146] Next, the user places the desired virtual object of the electronic block in the space through the screen of the terminal device 200 (step S906). At this time, the terminal device 200 displays the space on the display unit 260, and also uses AR technology to superimpose the virtual object of the electronic block designated / selected by the user onto the space.

[0147] In this way, by activating the simulation mode within the app, the user can use the in-app camera to superimpose the virtual electronic block object onto the real world. For example, the user points the camera of the terminal device 200 containing the app at the plane on which they want to place the electronic block to recognize the space. Then, when the user taps or pinches where they want to place the electronic block, the virtual electronic block object is placed through the camera. The virtual electronic block object can also be rotated or moved on the screen.

[0148] Next, the terminal device 200 determines whether the virtual object of the electronic block is placed in a recognizable location (step S907). If the virtual object of the electronic block is not placed in a recognizable location (step S907; No), the terminal device 200 requests (or prompts) the user to correct the placement so that the virtual object of the electronic block is placed in a recognizable location (return to step S906).

[0149] On the other hand, if the virtual electronic block object is located in a recognizable location (step S907; Yes), the terminal device 200 acquires the size, position, and depth of the object (real object) existing in the space in the real coordinate system using a camera image or LiDAR or millimeter-wave radar (step S908). Objects existing in the space include, for example, the walls, floor, and ceiling of a room, doors and stairs, furniture and fixtures, and people, animals, plants, and other living things existing in the space. Note that if the terminal device 200 cannot acquire the size, position, or depth of an object existing in the space, it calculates or estimates them as needed. Estimation also includes cases where it is estimated based on taps on the screen, etc.

[0150] Next, the terminal device 200 performs calculations based on the size, position, and depth of the objects present in the space and automatically adjusts the size of the virtual objects of the electronic blocks (step S909). For example, the terminal device 200 adjusts (changes) the size of the virtual objects of the placed electronic blocks according to the size, position, and depth of the objects present in the space, and displays (draws) the virtual objects of the electronic blocks whose size has been adjusted on the screen. Note that the terminal device 200 automatically adjusts the size of the virtual objects of the electronic blocks each time a change occurs in the size, position, or depth of an object present in the space, for example, when the user changes the position or orientation of a camera, optical magnification, or distance from the object in order to view from a different position or angle. In other words, the virtual objects of the electronic blocks also reflect the perspective and changes in perspective.

[0151] Next, the terminal device 200 estimates the state of the electronic block in the space from the placed virtual object of the electronic block and configures settings for the electronic block (step S910). For example, the terminal device 200 configures settings for the electronic block based on the content and procedure of the settings included in the recipe. Alternatively, the user may configure any settings for the operation of the electronic block on the terminal device 200. Note that the settings for the electronic block are not limited to settings related to the functions of the electronic block, but may also include fine adjustments to the position and attitude (angle) of the virtual object of the electronic block, changes to the connections between the virtual objects of the electronic block, and the like. The terminal device 200 may also configure settings related to the environment in the space, such as brightness and temperature. This allows simulations to be performed assuming any environment.

[0152] Next, the terminal device 200 executes the program included in the recipe based on the settings and performs a simulation (step S911).The terminal device 200 then displays the simulation results on the screen (step S912).That is, as a result of executing the program included in the recipe based on the settings, the terminal device 200 renders on the screen the behavior of the virtual object of the electronic block, which moves in the same way as the actual electronic block.The behavior of the virtual object of the electronic block is the same as the behavior of the actual electronic block.

[0153] For example, in the case of a virtual object of a sensor-type electronic block, as shown in Figure 17, the user can adjust the sensor value by sliding the slider bar or by directly entering a value within the app. Figure 17 shows an example of a settings screen including a slider bar. In the example of Figure 17, values ​​are set to "1" through "3," and the slider bar displays a slider at the "1" and "3" positions. That is, the slider bar has a lower limit and an upper limit, indicating the range of values. Furthermore, when a user connects a sensor-type electronic block to a node of another electronic block within the app, they can check the output behavior simply by adjusting the sensor value, which indicates the numerical value on the sensor side. The app also includes features such as OpenSound Control (OSC) communication, enabling integration with external services. In the case of a virtual object of an actuator-type electronic block, adjusting the numerical value displays a response on the screen.

[0154] In addition, users can simulate actual movements by actually moving the superimposed object and its surroundings on the AR screen. For more intuitive operation, simulations can also be performed by placing virtual objects that mimic the shape of electronic blocks, such as rectangular parallelepiped objects.

[0155] The terminal device 200 can also measure the server load and delay caused by the use of multiple electronic blocks. The terminal device 200 can then notify the user of the expected load and delay. The app can also present specific numerical values. It can also present the load and delay levels categorized into low, medium, and high levels based on a load test that can be easily performed using information within the device. The presentation can be done using a visual UI, an audio UI, or other methods.

[0156] As shown in FIG. 18, users can simulate the operation of existing programs registered on a shared site, a user forum on the cloud CL. FIG. 18 illustrates an overview of a simulation trial involving a separate workpiece. For example, as shown in FIG. 18A, the terminal device 200 may display a "Try" button for a recipe. When a user presses the "Try" button, the program included in the recipe is executed and a simulation is performed. In this case, the "Try" button serves as a button for starting the simulation mode. As shown in FIG. 18B, a separate workpiece to be combined with an electronic block, such as a 3D-printed workpiece, can also be used in the simulation if the corresponding data is linked to the program. This allows users to check the size and actual operation of the electronic block and the separate workpiece, as shown in FIG. 18C. In this way, users can easily run a simulation of a sample program and understand its operation, similar to reading a sample e-book. Users can also gain a firsthand understanding of the "mechanism" and operation that spans the real world and the computer, and confirm whether an existing program can be used in the user's environment (rather than the creator's environment).

[0157] Next, the terminal device 200 determines whether a button for ending the simulation has been pressed (step S913). For example, if the user is satisfied with the simulation results, the user presses a button for ending the simulation on the terminal device 200, since there is no need to perform any further simulations. Note that pressing a button is merely an example. In practice, the terminal device 200 may determine whether any action other than pressing a button has been performed as an action for ending the simulation.

[0158] If the button for ending the simulation has not been pressed (step S913; No), the terminal device 200 restarts the simulation from the settings (returns to step S910). For example, if the user is not satisfied with the simulation results on the terminal device 200, the user fine-tunes / modifies the settings to achieve the desired results and runs the simulation again. In practice, the user may restart the simulation on the terminal device 200, not only by changing the settings but also by starting over with the placement of the virtual objects of the electronic blocks (returns to step S906).

[0159] If the button for ending the simulation is pressed (step S913; Yes), the terminal device 200 ends the simulation mode (step S914). Thereafter, the terminal device 200 may perform a simulation for another recipe (return to step S901).

[0160] At this time, the terminal device 200 can operate the actual device based on the results of the simulation. For example, if the simulation is successful, the user purchases electronic blocks that the user does not actually own and places them in the actual space in the same way as the simulation. Then, the terminal device 200 provides the program used in the simulation to the electronic blocks placed in the actual space. The electronic blocks load and execute the program used in the simulation to perform the same operations as in the simulation. Alternatively, the terminal device 200 executes the program used in the simulation to control the operations of the electronic blocks placed in the actual space in the same way as in the simulation.

[0161] In addition, in this embodiment, since it is possible to create and modify a program while the simulation mode is started, the program may be created in advance, or the program creation or modification may be started while the simulation mode is started.

[0162] In the above description, the terminal device 200 may complete the processing in a stand-alone manner, but in reality, some or all of the processing executed by the terminal device 200 may be executed by the control device 400. For example, the control device 400 may receive an image of a space captured by a camera from the terminal device 200 and perform a simulation on the control device 400. In this case, the terminal device 200 is assumed to be linked to the control device 400 via the relay device 300. Furthermore, the control device 400 is assumed to have the functions of the terminal device 200 in the above description.

[0163] 3-3. Arrangement of Virtual Objects of Electronic Blocks The arrangement of virtual objects of electronic blocks will be described with reference to Fig. 19. Fig. 19 is a diagram showing an example screen displaying the arrangement of virtual objects of electronic blocks. For example, as shown in Fig. 19, a space photographed by a camera is displayed on the screen of the terminal device 200. In addition, virtual objects of each electronic block are displayed on the screen of the terminal device 200 as a list display of block types.

[0164] The list of possible block types displays any or all of the following block types: microphone sensor, motion sensor, brightness sensor, human presence sensor, button input, temperature / humidity sensor, LED (actuator), speaker (actuator), and other sensors / actuators that can be expanded using GPIO (e.g., weight sensor, distance sensor, infrared sensor, motor, camera, etc.).

[0165] The user specifies / selects any virtual object of an electronic block from this list of block types and places it on the screen. Note that the virtual objects displayed in the list of block types may be only virtual objects of electronic blocks related to the recipe. Furthermore, the type and number of electronic blocks displayed in the list can be changed by user operation. For example, it is also possible to add new electronic blocks later.

[0166] For example, in a recipe where a motion sensor is attached to a door and an LED (light-emitting diode) is turned on and displayed in red when a person is detected, the user specifies / selects a virtual object of an electronic block with motion sensor functionality and a virtual object of an electronic block with LED functionality from a list of block types, and attaches them to an object (real object) such as a door displayed on the camera screen using drag-and-drop or other methods.

[0167] If the object to which the virtual electronic block object is attached fits entirely within the camera screen and object recognition, depth information, and ground surface coordinates are acquired, the size of the virtual electronic block object is automatically adjusted to match the overall image of the camera and object. In other words, when attaching a virtual object on the screen, the size magnification of the virtual object is automatically changed to match the size of the object on the screen.

[0168] Furthermore, if an object goes off-screen due to a change in the camera's position, orientation, or angle of view, the virtual electronic block object attached to that object will also go off-screen and be lost. Alternatively, a change in the camera's position, orientation, or angle of view may cause the virtual electronic block object to be cast in the shadow of another object, becoming hidden behind that object and becoming lost.

[0169] In the case of a virtual object of an electronic block of sensors, even if the virtual object of the electronic block is lost on the screen, if the position of the virtual object of the electronic block can be estimated from the position of the recognized object on the screen, the estimated behavior result is calculated using the basic algorithm and drawn. This is because even if the electronic block of sensors itself is off-screen, its behavior result may appear on-screen.

[0170] In the case of virtual objects of electronic blocks such as actuators, if the virtual object of the electronic block is lost on the screen, the predicted behavior results will not be drawn. This is because if the actuator block itself is off-screen, the behavior results will also be completed off-screen and will not appear on-screen.

[0171] 3-4. Behavior Results of Virtual Objects of Electronic Blocks The behavior results of virtual objects of electronic blocks will be described with reference to FIG. 20. FIG. 20 is a diagram showing an example of a screen displaying the behavior results of virtual objects of electronic blocks. For example, in the case of a virtual object of an electronic block with a motion sensor function, if the sensor's response range, which indicates the range that the sensor can detect, can be visualized, it is rendered as a semi-transparent circle as shown in FIG. 20. Note that the semi-transparent circle is merely an example. In practice, the sensor's response range may be rendered and visualized in any shape or display mode. Furthermore, although the sensor's response range appears as a semi-transparent circle when viewed from the front, it can also be displayed in three dimensions when viewed from the side.

[0172] Furthermore, when a person passes within the camera screen and is determined to be a person through object recognition, the LED portion of a virtual object of an electronic block having an LED function placed on the screen is illuminated and displayed in red. The person passing within the camera screen does not have to be a real person, but may be a virtual object representing a person placed on the screen. The size and behavior of the virtual object representing a person can be set arbitrarily. Alternatively, the user may be able to specify / select a virtual object representing a desired person from among virtual objects representing people based on attributes such as age, gender, etc.

[0173] Furthermore, although it differs from the actual behavior, it is also possible to make the entire screen display red to make the behavior result easier to recognize. For example, when a virtual object of an electronic block having a human presence sensor function detects a person, the entire screen display may be colored reddish, or the entire screen may light up or flash red. Note that red is just one example. In reality, other colors or display modes may be used. In other words, it is sufficient if the behavior result, such as the fact that the human presence sensor has detected a person, is displayed in a visible manner.

[0174] 3-5. On-Screen Interactions We will now explain on-screen interactions with reference to FIG. 21. FIG. 21 shows an example of an on-screen interaction screen. For example, let us consider a case where a user wants to attach a motion sensor to a door and create a program. Because the camera must be fixed when moving an object in the real world, it may be difficult to see the on-screen response when working alone. For example, when reviewing a recipe while looking at the motion sensor's values, the user may want to see the response through on-screen interaction. In this case, as shown in FIG. 21, when the user touches the motion sensor's virtual object on the screen, the terminal device 200 estimates the force and magnitude applied to the motion sensor based on the pressure on the display and the position coordinates, and changes the sensor value indicating the motion sensor's value. At this time, it is also possible to output a graph or other data on a time axis.

[0175] 3-6. Behavior that takes accessibility into consideration Behavior that takes accessibility into consideration will be described with reference to Fig. 22. Fig. 22 is a diagram showing an example screen of behavior that takes accessibility into consideration. If the user's characteristics and attributes are known to the system, notifications can be sent even in modal / pop-up / dialog boxes that are different from those intentionally created in the recipe.

[0176] For example, when simulating a recipe that involves a behavior such as playing a sound, the terminal device 200 displays a sound icon "♪" on the screen as / along with the behavior result so that people with hearing impairments can understand.

[0177] Furthermore, when simulating a recipe that involves behavior such as lighting an LED, the terminal device 200 makes a sound so that visually impaired people can understand.

[0178] 3-7. Implementing a Simulation from a Video of an Action A case where a simulation is implemented from a video of an action will be described. In this case, as shown in FIG. 23 , the user moves an object to be detected while capturing an image using the camera function of the terminal device 200 such as a smartphone, tablet, or AR goggles. In this way, the user captures a video of the action using the terminal device 200. FIG. 23 is a diagram showing an example of a video of the action.

[0179] The control device 400 on the cloud CL as a proposal system works in cooperation with the terminal device 200 to recognize what is moving from a video of the operation, recognize how it is moving, and propose a recipe using a sensor that captures that movement. For example, if the control device 400 recognizes that the moving object is a door, it proposes a recipe using sensors such as a motion sensor and an open / close sensor.

[0180] Based on the recipe, the user can perform a simulation of the placement and operation of the sensor on the screen of the terminal device 200. At this time, the control device 400 on the cloud CL may propose a marker to indicate the sensor installation position as printable data, and the user may print out the marker and attach it to the object. Alternatively, instead of attaching a physical marker, the terminal device 200 may simply superimpose the marker on the AR screen.

[0181] 24, the user can set the parameters of the sensor on the screen of the terminal device 200 while viewing the simulation results. Fig. 24 is a diagram showing an example of setting parameters on the screen.

[0182] 3-8. Implementing a Simulation from a Desired Action Description A case where a simulation is implemented from a desired action description will be described. In this case, the user writes down what they want to do using text, voice, handwritten notes, or the like on the terminal device 200, such as a smartphone, tablet, or AR goggles. For example, the user inputs a statement such as "I want to record the door opening and closing speed and frequency as data" into the terminal device 200.

[0183] The control device 400 on the cloud CL as a proposal system reads the text, voice, handwritten notes, etc. as input sentences and proposes necessary sensors and installation methods based on the input sentences. For example, the control device 400 proposes sensors such as motion sensors and open / close sensors and their installation methods. At this time, the control device 400 may propose a recipe for the sensors and their installation methods.

[0184] The proposal includes information such as the sensors required and where and how to install the sensors. At this time, as shown in Fig. 25, the control device 400 on the cloud CL as the proposal system automatically generates PDF data such as a procedure manual on how to install the sensors and provides it to the terminal device 200. Fig. 25 is a diagram showing an example of the data of the procedure manual. Note that the data of the procedure manual may be included in the recipe.

[0185] Based on the proposal, the user can perform a simulation of the placement and operation of the sensor on the screen of the terminal device 200. At this time, the control device 400 on the cloud CL may propose a marker to indicate the sensor installation position as printable data, and the user may print out the marker and attach it to the object. Alternatively, instead of attaching a physical marker, the terminal device 200 may simply superimpose the marker on the AR screen.

[0186] As shown in FIG. 24, the user can set the parameters of the sensor on the screen of the terminal device 200 while viewing the results of the simulation.

[0187] In this way, users can use sensors and perform simulations even if they have no prior knowledge of what types of sensors can detect what. Also, even if they do not have the actual sensor on hand, they can try out different sensor variations, making it easier to find the appropriate solution.

[0188] It is also possible to register patterns of the objects and user characteristics used during the simulation (such as the person's skeletal information from bones). It is also possible to accumulate sensor value data for specific objects in a database for learning and registration by situation. For example, if a user wants to register a specific object as "when tilted slightly," it is possible to register the sensor value at that time and context information such as "when tilted slightly." Furthermore, the sensor value at that time can be learned by AI such as GPT as the sensor value for "when tilted slightly."

[0189] 4. Other Embodiments The processing according to the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

[0190] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. Furthermore, the information, including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0191] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0192] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0193] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0194] 5. Effects of the Information Processing Method According to the Present Disclosure As described above, in the information processing method according to the present disclosure, a processor (in an embodiment, the control unit 230 of the terminal device 200) uses a camera (in an embodiment, the sensor unit 220) to capture an image of a space in which an electronic device (in an embodiment, the electronic device 100) that realizes a predetermined function is to be placed, displays an image of the captured space on a display (in an embodiment, the display unit 260), accepts a user's input of a designation or operation on the captured image of the space, places a virtual object of the electronic device designated by the user in accordance with the user's operation, superimposes the virtual object of the placed electronic device on the captured image of the space, and displays a simulation performed using the virtual object of the electronic device on the captured image of the space.

[0195] In this way, users can check the placement and operation of electronic blocks that they do not own in an actual environment without actually purchasing the electronic blocks and checking them on a physical device.

[0196] The processor also displays the performed simulation reflecting information describing the relationship between the objects present in the photographed space and the virtual object of the electronic device.

[0197] In this way, the user can check the placement and operation of electronic blocks that he or she does not own, taking into account the relationship between objects that exist in the actual space and the electronic blocks that he or she does not own, without actually purchasing the electronic blocks.

[0198] The processor also acquires data on a solution to the problem desired by the user from a sharing site (in this embodiment, the control device 400) for sharing data on solutions to problems (recipes in this embodiment) posted and input by creators via the network, and displays a simulation performed using a virtual object of the electronic device in accordance with a program for operating an electronic device that realizes a specified function, which is included in the acquired solution data.

[0199] In this way, users can obtain solutions from a sharing site where solutions to problems are posted, and check the placement and operation of electronic blocks that they do not own based on that solution.

[0200] The processor also receives, via the network, from a server (in this embodiment, the control device 400) that has learned the data on solutions to problems posted and input by the creator, a proposal for a solution selected by the server or a newly generated solution to the user's problem, and displays a simulation performed using a virtual object of the electronic device in accordance with a program for operating an electronic device that realizes a specified function, which is included in the solution to the problem proposed by the server.

[0201] In this way, users can receive solutions tailored to their goals from a server that has trained AI such as GPT to solve problems, and can check the placement and operation of electronic blocks that they do not own based on those solutions.

[0202] Furthermore, when a user creates a program for operating an electronic device that realizes a predetermined function, the processor executes a simulation using a virtual object of the electronic device in accordance with the program.

[0203] In this way, when a user creates a program to operate an electronic block that the user does not own, the user can check the operation of the electronic block that the user does not own based on the program in an actual environment without actually purchasing the electronic block.

[0204] The processor also presents the user with a list of virtual objects of multiple electronic devices on the image of the captured space, arranges the virtual object of the electronic device designated by the user from the presented list of virtual objects of the multiple electronic devices in accordance with the user's operation, and superimposes the virtual object of the arranged electronic device on the image of the captured space.

[0205] In this way, the user can place a virtual object of an electronic block that the user does not possess on the image of the actual space and check the operation of the placed electronic block. At this time, the electronic block that the user does not possess is displayed in AR on the image of the actual space.

[0206] The processor also presents virtual objects of electronic devices included in the solution to the problem obtained via the network on the image of the captured space, arranges the presented virtual objects of the electronic devices according to the user's operation, and superimposes the arranged virtual objects of the electronic devices on the image of the captured space.

[0207] In this way, when a user obtains a solution, the user can place virtual objects of the electronic blocks that the user does not have on a video of the actual space according to the solution, thereby confirming the placement and operation of the electronic blocks that the user does not have based on the solution in the actual environment. This allows the user to confirm whether the solution is effective in the actual environment without actually purchasing the electronic blocks.

[0208] The processor also determines whether or not the virtual object of the electronic device is placed in a recognizable location on the captured image of the space, and if the virtual object of the electronic device is not placed in a recognizable location, requests the user to correct the placement of the virtual object of the electronic device.

[0209] In this way, if the virtual objects of electronic blocks are not properly placed on the image of the space, the user is prompted to correct the placement, and can place them properly.

[0210] The processor also recognizes objects present in the photographed space, performs calculations based on the positions and depths of the objects present in the photographed space, and adjusts the size of the virtual object on the electronic device.

[0211] In this way, the virtual objects of electronic blocks placed on the image of the space are displayed at the same size as the electronic blocks in the actual environment, allowing the user to see the placement and operation of the electronic blocks at the same size as in the actual environment.

[0212] The processor also operates the virtual object of the electronic device on the captured image of the space in accordance with the execution of the simulation, and displays the behavior of the virtual object of the electronic device.

[0213] In this way, the user can see on the display the behavior of the electronic blocks, which operate in the same way as in the actual environment. At this time, the behavior of the electronic blocks is also displayed in AR.

[0214] In addition, the processor recognizes objects present in the photographed space, and if the virtual object of the electronic device is a virtual object of a sensor-type electronic device, even when the virtual object of the electronic device is lost on the display screen, if the position of the virtual object of the electronic device can be estimated from the position of the recognized object, the processor calculates an estimated behavior result and draws the estimated behavior result.

[0215] In this way, for electronic blocks of sensors, the user can check the behavior of the electronic block even if the electronic block is no longer visible on the display.

[0216] Furthermore, when the virtual object of the electronic device is a virtual object of an electronic device of sensors, the processor visualizes the reaction range of the sensors by drawing them in a predetermined shape as the behavior of the virtual object of the electronic device.

[0217] In this way, for electronic blocks of sensors, the user can visually check the response range of the sensors on the display.

[0218] Furthermore, the processor displays the virtual object of the electronic device in a display mode different from the actual behavior to indicate that the virtual object has moved.

[0219] In this way, even if the actual behavior of the electronic block is difficult to understand on the display, it can be replaced with an easily understandable behavior and displayed, allowing the user to confirm that the electronic block is operating normally by seeing behavior different from the actual behavior on the display.

[0220] The processor also changes the entire screen of the display to a predetermined color to indicate that the virtual object of the electronic device has moved.

[0221] In this way, even if the actual behavior of the electronic block is difficult to understand on the display, the user can be informed that the electronic block has operated by, for example, turning the entire display screen red when the electronic block operates.

[0222] Furthermore, when the processor executes a simulation involving a behavior of making a sound as the behavior of the virtual object of the electronic device, a sound icon is displayed on the display together with the behavior of making a sound.

[0223] In this way, if the behavior of the electronic block is related to sound, the user can be informed that the electronic block has operated by displaying a sound icon on the display so that even hearing-impaired users can visually recognize that the electronic block has operated.

[0224] Furthermore, when the processor executes a simulation involving a behavior of emitting light as the behavior of a virtual object of the electronic device, a sound is generated along with the behavior of emitting light.

[0225] In this way, if the behavior of the electronic block is visual, the user can be notified that the electronic block has operated by making a sound so that even a visually impaired user can auditorily recognize that the electronic block has operated.

[0226] Furthermore, when a user touches a virtual object of an electronic device on the display screen, the processor estimates the point and magnitude of force applied to the electronic device based on the pressing force on the display and the position coordinates, and changes values ​​related to the behavior of the electronic device.

[0227] In this way, by touching the virtual object of the electronic block on the display with a finger or the like, the user can estimate the point and magnitude of force acting on the electronic block in the actual environment and change values ​​related to the behavior of the electronic block.

[0228] Furthermore, if the user adjusts the settings related to the electronic device based on the results of the simulation, the processor displays the simulation performed again using the virtual object of the electronic device on the image of the captured space in accordance with the adjusted settings.

[0229] In this way, the user can immediately change the settings based on the results of the simulation and feed this back into the next simulation. By repeating this process, the user can obtain results that are closer to their ideals.

[0230] 6. Hardware Configuration An information processing device such as the terminal device 200 or the control device 400 according to each of the above-described embodiments is realized by a computer 1000 having a configuration as shown in FIG. 26, for example. FIG. 26 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device. The information processing device will be described below using the terminal device 200 according to the embodiment as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0231] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0232] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0233] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a conversion program according to the present disclosure, which is an example of program data 1450.

[0234] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0235] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0236] For example, when the computer 1000 functions as the terminal device 200 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the control unit 230, etc. The information processing program according to the present disclosure and data in the storage unit 240 are stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0237] The above shows an example of the hardware configuration of the computer 1000. Each of the above components may be configured using general-purpose components, or may be configured using hardware specialized for the function of each component. This configuration may be modified as appropriate depending on the technical level at the time of implementation.

[0238] The technology disclosed herein may also be configured as follows: (1) An information processing method in which a processor uses a camera to capture an image of a space in which an electronic device that realizes a predetermined function is to be placed, displays an image of the captured space on a display, accepts a user's input of a designation or operation on the captured image of the space, places a virtual object of the electronic device designated by the user in accordance with the user's operation, superimposes the placed virtual object of the electronic device on the captured image of the space, and displays a simulation performed using the virtual object of the electronic device on the captured image of the space. (2) The information processing method described in (1), in which the processor recognizes an object present in the captured space, and displays the simulation performed by reflecting information describing a relationship between the object present in the captured space and the virtual object of the electronic device. (3) The information processing method according to (1) or (2), wherein the processor acquires, via a network, solution data for a problem desired by the user from a sharing site for sharing solution data for problems posted and input by creators, and displays the simulation performed using a virtual object of the electronic device in accordance with a program for operating an electronic device that realizes a predetermined function, which is included in the acquired solution data. (4) The information processing method according to any one of (1) to (3), wherein the processor receives, via a network, from a server that has learned the solution data for the problem posted and input by the creator, a proposal of solution data selected by the server for the user's problem or newly generated solution data, and displays the simulation performed using a virtual object of the electronic device in accordance with a program for operating an electronic device that realizes a predetermined function, which is included in the solution data for the problem proposed by the server.(5) The information processing method according to any one of (1) to (4), wherein, when a user creates a program for operating an electronic device that realizes a predetermined function, the processor displays the simulation performed using a virtual object of the electronic device in accordance with the program. (6) The information processing method according to any one of (1) to (5), wherein the processor presents to the user a list of virtual objects of multiple electronic devices on the captured image of the space, arranges a virtual object of the electronic device specified by the user from the presented list of virtual objects of the multiple electronic devices in accordance with an operation of the user, and superimposes and displays the arranged virtual object of the electronic device on the captured image of the space. (7) The information processing method according to any one of (1) to (6), wherein the processor presents, on the captured image of the space, a virtual object of an electronic device included in data on a solution to a problem obtained via a network, arranges the presented virtual object of the electronic device in accordance with an operation of the user, and superimposes and displays the arranged virtual object of the electronic device on the captured image of the space. (8) The information processing method according to any one of (1) to (7), wherein the processor determines whether the virtual object of the electronic device is placed in a recognizable location on the captured image of the space, and if the virtual object of the electronic device is not placed in a recognizable location, requests the user to correct the location of the virtual object of the electronic device. (9) The information processing method according to any one of (1) to (8), wherein the processor recognizes an object present in the captured space, performs calculation processing based on the position and depth of the object present in the captured space, and adjusts the size of the virtual object of the electronic device. (10) The information processing method according to any one of (1) to (9), wherein the processor operates the virtual object of the electronic device on the captured image of the space in conjunction with execution of the simulation, and displays the behavior of the virtual object of the electronic device.(11) The information processing method according to any one of (1) to (10), wherein the processor recognizes an object present in the captured space, and, if the virtual object of the electronic device is a virtual object of a sensor-type electronic device, calculates an estimated behavior result and draws the estimated behavior result even when the virtual object of the electronic device is lost on the screen of the display if the position of the virtual object of the electronic device can be estimated from the position of the recognized object. (12) The information processing method according to any one of (1) to (11), wherein, if the virtual object of the electronic device is a virtual object of a sensor-type electronic device, the processor draws and visualizes the response range of the sensors in a predetermined shape as the behavior of the virtual object of the electronic device. (13) The information processing method according to any one of (1) to (12), wherein the processor displays the virtual object of the electronic device in a display mode different from its actual behavior to indicate that it has operated. (14) The information processing method according to any one of (1) to (13), wherein the processor changes the entire screen of the display to a predetermined color to indicate that the virtual object of the electronic device has operated. (15) The information processing method according to any one of (1) to (14), wherein the processor, when executing a simulation involving sound generation as a behavior of the virtual object of the electronic device, displays a sound icon on the display along with the sound generation behavior. (16) The information processing method according to any one of (1) to (15), wherein the processor, when executing a simulation involving light emission as a behavior of the virtual object of the electronic device, generates a sound along with the light emission behavior. (17) The information processing method according to any one of (1) to (16), wherein, when the user touches the virtual object of the electronic device on the screen of the display, the processor estimates the point and magnitude of force applied to the electronic device based on the pressure on the display and position coordinates, and changes a value related to the behavior of the electronic device.(18) The information processing method according to any one of (1) to (17), wherein, when the user adjusts settings related to the electronic device based on the results of the simulation, the processor again displays the simulation performed using the virtual object of the electronic device on the captured image of the space in accordance with the adjusted settings. (19) An information processing program causing a processor to execute the following steps: using a camera to capture an image of a space in which an electronic device that realizes a predetermined function is to be placed; displaying the captured image of the space on a display; accepting input of a designation or operation by a user on the captured image of the space; placing the virtual object of the electronic device designated by the user in accordance with the user's operation; superimposing and displaying the placed virtual object of the electronic device on the captured image of the space; and displaying the simulation performed using the virtual object of the electronic device on the captured image of the space. (20) An information processing device comprising: a camera that photographs a space in which an electronic device that realizes a predetermined function is to be placed; a display that displays an image of the photographed space; and a processor that accepts input of a designation or operation by a user on the image of the photographed space, places a virtual object of the electronic device designated by the user in accordance with the user's operation, superimposes the placed virtual object of the electronic device on the image of the photographed space, and displays a simulation performed using the virtual object of the electronic device on the image of the photographed space.

[0239] REFERENCE SIGNS LIST 10 Information processing system 100 Electronic device 110 Functional block 120 Common block 200 Terminal device 210 Operation unit 220 Sensor unit 230 Control unit 240 Storage unit 250 Communication unit 260 Display unit 300 Relay device 400 Control device 410 Control unit 420 Storage unit 430 Communication unit

Claims

1. An information processing method in which a processor uses a camera to capture an image of a space in which an electronic device that realizes a predetermined function is to be placed, displays the captured image of the space on a display, accepts user input of specifications or operations on the captured image of the space, places a virtual object of the electronic device designated by the user in accordance with the user's operation, superimposes the placed virtual object of the electronic device on the captured image of the space, and displays a simulation performed using the virtual object of the electronic device on the captured image of the space.

2. The information processing method according to claim 1, wherein the processor recognizes objects present in the photographed space, and displays the simulation performed by reflecting information explaining the relationship between the objects present in the photographed space and the virtual object of the electronic device.

3. The information processing method according to claim 1, wherein the processor acquires, via a network, data on a solution to a problem desired by the user from a sharing site for sharing data on solutions to problems posted and input by creators, and displays the simulation performed using a virtual object of the electronic device in accordance with a program for operating an electronic device that realizes a specified function, the program being included in the acquired solution data.

4. The information processing method of claim 1, wherein the processor receives, via a network, from a server that has learned the data of solutions to problems posted and input by a creator, a proposal of data of solutions selected by the server for the user's problem or data of newly generated solutions, and displays the simulation performed using a virtual object of the electronic device in accordance with a program for operating an electronic device that realizes a specified function, which program is included in the data of solutions to the problem proposed by the server.

5. The information processing method according to claim 1, wherein, when a user creates a program for operating an electronic device that realizes a predetermined function, the processor displays the simulation performed using a virtual object of the electronic device in accordance with the program.

6. The information processing method according to claim 1, wherein the processor presents to the user a list of virtual objects of the plurality of electronic devices on the image of the captured space, arranges the virtual object of the electronic device specified by the user from the presented list of virtual objects of the plurality of electronic devices in accordance with the user's operation, and superimposes the arranged virtual object of the electronic device on the image of the captured space.

7. The information processing method according to claim 1, wherein the processor presents a virtual object of an electronic device included in the data on the solution to the problem obtained via a network on the image of the captured space, arranges the presented virtual object of the electronic device in accordance with the user's operation, and superimposes the arranged virtual object of the electronic device on the image of the captured space.

8. The information processing method according to claim 1, wherein the processor determines whether or not the virtual object of the electronic device is placed in a recognizable location on the captured image of the space, and if the virtual object of the electronic device is not placed in a recognizable location, requests the user to correct the placement of the virtual object of the electronic device.

9. The information processing method according to claim 1, wherein the processor recognizes objects present in the photographed space, performs calculations based on the positions and depths of the objects present in the photographed space, and adjusts the size of the virtual object on the electronic device.

10. The information processing method according to claim 1, wherein the processor operates a virtual object of the electronic device on the captured image of the space as the simulation is performed, and displays the behavior of the virtual object of the electronic device.

11. The information processing method according to claim 1, wherein the processor recognizes an object present in the photographed space, and, if the virtual object of the electronic device is a virtual object of an electronic device of a sensor type, calculates an inferred behavior result and draws the inferred behavior result even when the virtual object of the electronic device is lost on the screen of the display, if the position of the virtual object of the electronic device can be estimated from the position of the recognized object.

12. The information processing method according to claim 1, wherein, when the virtual object of the electronic device is a virtual object of an electronic device of sensors, the processor visualizes the reaction range of the sensors by drawing them in a predetermined shape as the behavior of the virtual object of the electronic device.

13. The information processing method according to claim 1, wherein the processor displays the virtual object of the electronic device in a display mode different from its actual behavior to indicate that the virtual object has moved.

14. The information processing method according to claim 1, wherein the processor changes the entire screen of the display to a predetermined color to indicate that the virtual object of the electronic device has moved.

15. The information processing method according to claim 1, wherein when the processor performs a simulation involving a behavior of making a sound as a behavior of the virtual object of the electronic device, it displays a sound icon on the display together with the behavior of making a sound.

16. The information processing method according to claim 1, wherein when the processor executes a simulation involving a light-emitting behavior as the behavior of a virtual object of the electronic device, the processor plays a sound along with the light-emitting behavior.

17. The information processing method according to claim 1, wherein when the user touches a virtual object of the electronic device on the screen of the display, the processor estimates the point and magnitude of force applied to the electronic device based on the pressing force on the display and position coordinates, and changes a value related to the behavior of the electronic device.

18. The information processing method of claim 1, wherein, when the user adjusts settings related to the electronic device based on the results of the simulation, the processor again displays the simulation performed using the virtual object of the electronic device on the image of the captured space in accordance with the adjusted settings.

19. An information processing program that causes a processor to execute the following steps: using a camera to photograph a space in which an electronic device that realizes a specified function is to be placed; displaying the photographed image of the space on a display; accepting user input of specifications or operations on the photographed image of the space; placing a virtual object of the electronic device designated by the user in accordance with the user's operation; superimposing the placed virtual object of the electronic device on the photographed image of the space; and displaying a simulation performed using the virtual object of the electronic device on the photographed image of the space.

20. An information processing device comprising: a camera that photographs a space in which an electronic device that realizes a specified function is to be placed; a display that displays an image of the photographed space; and a processor that accepts input of a user's designation or operation on the photographed image of the space, places a virtual object of the electronic device designated by the user in accordance with the user's operation, superimposes the placed virtual object of the electronic device on the photographed image of the space, and displays a simulation performed using the virtual object of the electronic device on the photographed image of the space.

Images