Assistance device and assistance method

Abstract

An assistance device for assisting creation of a program comprises: a display unit that displays, on a display device, a graphic object representing a program component that a user has instructed is to be added to a program; an acquisition unit that acquires settings related to the program component from the user; and a creation unit that creates a program on the basis of settings related to a plurality of program components acquired from the user. The program component is of any type among a plurality of types including a first type that defines an activation condition for activating the program and a second type that defines individual processing of the program. For the first type of program component, a second type of program component that is activated in response to the activation condition being satisfied can be set. For the second type of program component, the details of the individual processing and another program component of the second type activated after the execution of the individual processing can be set. For at least one program component of the second type, the details of the individual processing can be set in a natural language.

Description

Support device and support method

[0001] The present invention relates to an assistance device and an assistance method.

[0002] There are various known methods for supporting a user in creating a program. Patent Document 1 proposes a technique for supporting the creation of a workflow using a GUI.

[0003] US Patent Publication No. 2024 / 0134874

[0004] It is difficult to create a program efficiently with the technique of Patent Document 1. An object of some aspects of the present invention is to provide a technique that enables a program to be created efficiently.

[0005] According to some embodiments, there is provided an assistance device for assisting in the creation of a program, the assistance device comprising: a display means for displaying on a display device graphic objects representing program parts instructed by a user to be added to the program; an acquisition means for acquiring settings for the program parts from the user; and a creation means for creating the program based on the settings for a plurality of program parts acquired from the user, wherein the program parts are of one of a plurality of types including a first type that specifies a start condition for starting the program and a second type that specifies individual processing of the program, wherein for a program part of the first type, it is possible to set a program part of the second type that is started in response to the start condition being satisfied, and for a program part of the second type, it is possible to set the content of the individual processing and another program part of the second type that is started after the individual processing is executed, and for at least one program part of the second type, it is possible to set the content of the individual processing in natural language.

[0006] Some embodiments allow for efficient creation of programs.

[0007] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0008] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and together with the description, are used to explain the principles of the present invention. A block diagram illustrating an example configuration of a support device according to some embodiments. A block diagram illustrating an example hardware configuration of a support device according to some embodiments. A flow diagram illustrating an example operation of a support device according to some embodiments. A schematic diagram illustrating an example creation screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example setting screen according to some embodiments. A schematic diagram illustrating an example creation screen according to some embodiments. A schematic diagram illustrating an example creation screen according to some embodiments. A flow diagram illustrating an example operation of a support device according to some embodiments.

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0010] An example configuration of a system including an assistance device 101 according to some embodiments will be described with reference to Fig. 1. The connection between two components in Fig. 1 indicates that these two components can communicate over a network. The network may be a cellular network, a local area network, the Internet, another network, or any combination thereof.

[0011] The support device 101 is a device that provides a service to support a user in creating a program. In the following description, a service that supports the creation of a program is referred to as a creation support service. Furthermore, a program created by a user using the creation support service is referred to as a user-created program. The support device 101 may be capable of executing a user-created program.

[0012] The user terminal 102 is a device used by a user to use the creation support service. The user terminal 102 may be any device such as a smartphone, a tablet computer, a desktop computer, or a laptop computer. The support device 101 may provide the creation support service as a web service, and the user may use the creation support service using a browser on the user terminal 102. Alternatively, the user may use the creation support service using a dedicated application installed on the user terminal 102.

[0013] Although only one user terminal 102 is shown in Figure 1, the support device 101 may be capable of providing creation support services to multiple users, and multiple users may use the creation support services using their own user terminals 102.

[0014] When executing a user-created program, the assistance device 101 may cooperate with at least one of a data collection server 103, a file server 105, an image analysis server 106, and a natural language processing server 107.

[0015] The data collection server 103 is a device that collects data from one or more Internet of Things (IoT) devices 104. The IoT devices 104 are devices that can be connected to the Internet. The IoT devices 104 may be used while connected to the Internet or in a closed network. For example, the IoT devices 104 may include sensors for measuring specific physical quantities. The sensors may be any type of sensors, such as a temperature sensor, a humidity sensor, an acceleration sensor, an illuminance sensor, a vibration sensor, a positioning sensor, a pressure sensor, an acoustic sensor, or an image sensor. The IoT devices 104 may transmit data representing the physical quantities measured by the sensors to the data collection server 103 periodically or when a specific condition is met. The data collection server 103 may store the data received from the IoT devices 104 or transfer it to another device.

[0016] The file server 105 is a device that provides a file service. The file service is a service that stores files of users of the file service. In response to a request from a user, the file server 105 stores a new file in a storage device, reads an existing file stored in the storage device, or deletes the file. The file service may have a function that allows multiple users to share stored files.

[0017] The image analysis server 106 is a device that provides an image analysis service. The image analysis service is a service that analyzes images. The image analysis service may analyze still images or moving images.

[0018] The natural language processing server 107 is a device that provides a natural language processing service. The natural language processing service returns a response to an input made in a natural language. The natural language processing service may generate a response using a large-scale language model trained by machine learning. For example, the natural language processing service may be ChatGPT by OpenAI.

[0019] The support device 101 and the various servers described above may be configured by devices in an on-premise environment or by computing resources in a cloud computing environment. The support device 101 and the various servers may be configured as separate devices or as the same device.

[0020] An example of the hardware configuration of a computer 200 according to some embodiments will be described with reference to Fig. 2. The computer 200 may be used as any of the assistance device 101, the user terminal 102, and various servers shown in Fig. 1. The computer 200 may have the hardware components shown in Fig. 2. The computer 200 may have hardware components not shown in Fig. 2, or may not have some of the hardware components shown in Fig. 2.

[0021] The processor 201 controls the overall operation of the computer 200. The processor 201 may be configured, for example, by a central processing unit (CPU), a graphics processing unit (GPU), or a combination of these. The processor 201 may be a single processor or a collection of multiple processors connected to each other so that they can communicate with each other.

[0022] The memory 202 stores programs and data used in the processing of the computer 200. The memory 202 may be configured, for example, by a combination of random access memory (RAM) and read-only memory (ROM).

[0023] The input device 203 is a device for obtaining instructions from a user of the computer 200. The input device 203 may be configured, for example, by a combination of one or more of a keyboard, buttons, a touchpad, and a microphone. The display device 204 is a device for visually presenting information to the user of the computer 200. The display device 204 may be, for example, a dot-matrix display such as a liquid crystal display. The computer 200 may have a device (e.g., a touch screen) in which the input device 203 and the display device 204 are integrated. The input device 203 and the display device 204 may be external to the computer. In this case, the computer 200 may have an interface for communicating with the external input device 203 and display device 204.

[0024] The communication device 205 is a device for communicating with devices external to the computer 200. When the computer 200 performs wired communication, the communication device 205 may be a network interface card (NIC) having a connector for connecting a cable. When the computer 200 performs wireless communication, the communication device 205 may be a wireless communication module including an antenna and a baseband processing circuit.

[0025] The secondary storage device 206 is a device for non-volatilely storing programs and data used in the processing of the computer 200. The secondary storage device 206 is configured by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0026] In the following description, a component (eg, processor 201) of computer 200 that functions as a particular device (eg, support device 101) will be referred to as a component of this particular device (eg, processor 201 of support device 101).

[0027] An example of the operation of the assistance device 101 for providing a creation assistance service will be described with reference to Fig. 3 . The operation of Fig. 3 may be performed by the processor 201 of the assistance device 101 executing a program read into the memory 202 of the assistance device 101. Alternatively, some or all of the steps of the operation of Fig. 3 may be performed by a dedicated integrated circuit such as an application specific integrated circuit (ASIC). The operation of Fig. 3 may be started in response to a request from a user of the assistance device 101 for assistance in program creation from the assistance device 101. The following description will be given of the case where a user starts creating a new program. As will be described later, the user may be able to edit an existing user-created program.

[0028] In S301, the assistance device 101 displays a creation screen 400 on the display device 204 of the user terminal 102 for receiving instructions from the user regarding program creation. An example of the creation screen 400 will be described with reference to FIG. 4 . The creation screen 400 is a graphic user interface (GUI) screen. The GUI screen is a screen that includes one or more graphic objects (hereinafter simply referred to as objects) for receiving input from the user. In the following description, a specific object for receiving a specific input is described as an example, but another object capable of receiving the same input may be used instead.

[0029] Creation screen 400 includes the objects shown in Fig. 4. Creation screen 400 may include objects not shown in Fig. 4, or may not include some of the objects shown in Fig. 4. The same applies to other screens described below.

[0030] The field 401 is an area where objects representing one or more program parts are displayed. A program part is a set of instructions that defines the operation of a user-created program. As will be described later, the assistance device 101 creates a program based on the multiple program parts displayed in the field 401.

[0031] Program parts are classified into one of a plurality of types. In this embodiment, program parts are one of three types: "event source", "action", and "channel".

[0032] An "event source" is a type of program component that specifies the conditions for starting a user-created program. The conditions for starting a user-created program are referred to as start conditions. An "action" is a type of program component that specifies individual processing of a user-created program. A "channel" is a type of program component that specifies the connection between an event source and an action, or the connection between two actions.

[0033] The button 402 is an object for acquiring from the user an instruction to add an icon representing an event source to the field 401. The button 403 is an object for acquiring from the user an instruction to add an icon representing an action to the field 401. The button 404 is an object for acquiring from the user an instruction to complete the creation of the program. The button 405 is an object for acquiring from the user an instruction to stop the creation of the program. In response to the button 405 being pressed, the assistance device 101 ends the operation of FIG. 3 .

[0034] 3, in step S302, the assistance device 101 updates the program setting and creation screen 400 in accordance with an instruction from the user. The following describes an instruction from the user and the operation of the assistance device 101 in response to that instruction.

[0035] In response to the button 402 being pressed, the support device 101 displays a setting screen 500 for acquiring settings related to the event source from the user on the display device 204 of the user terminal 102. The setting screen 500 may be displayed superimposed on the creation screen 400, or may be displayed in place of the creation screen 400. An example of the setting screen 500 will be described with reference to FIGS. 5A to 5D. The event source for which settings are made using the setting screen 500 is referred to as a target event source.

[0036] The text box 501 is an object for setting the name of the target event source. The name set in the text box 501 may be displayed on an icon representing the target event source in the field 401 on the creation screen 400. The text box 502 is an object for setting a description of the target event source.

[0037] The field 503 is an area in which objects for setting activation conditions are arranged. Details of the field 503 will be described later. The button 504 is an object for acquiring an instruction from the user to cancel the setting of the target event source. The assistance device 101 cancels the addition of the target event source in response to the button 504 being pressed. Specifically, the assistance device 101 closes the setting screen 500 and displays the creation screen 400 again. The button 505 is an object for acquiring an instruction from the user to create the target event source. In response to the button 505 being pressed, the assistance device 101 creates the target event source in accordance with the settings specified by the user on the setting screen 500. Specifically, the assistance device 101 closes the setting screen 500, displays the creation screen 400 again, and adds an icon representing the target event source to the field 401 of the creation screen 400. Furthermore, the support device 101 associates the settings specified by the user on the setting screen 500 with the target event source and stores them in the memory 202 of the support device 101. At this time, the support device 101 may assign an individual component ID to the target event source.

[0038] The toggle switch 506 is an object for setting whether a target event source is enabled or disabled. When the toggle switch 506 is set to enabled, the target event source is enabled. In this case, the assistance device 101 activates the target event source when executing a user-created program. When the toggle switch 506 is set to disabled, the target event source is disabled. In this case, the assistance device 101 does not activate the user-created program even if the activation conditions are met.

[0039] Next, details of field 503 will be described. Field 503 has multiple tabs (four in the example of FIGS. 5A to 5D), and a different activation condition can be set for each tab. Field 503 may include tabs for setting other activation conditions in addition to the four tabs shown in FIGS. 5A to 5D, or may not include one or more of the four tabs shown in FIGS. 5A to 5D.

[0040] FIG. 5A shows a field 503 in a state where a tab ("IoT Device") for setting activation conditions for IoT devices is selected. When this tab is selected, data transmission from a specific device is set as an activation condition. When this activation condition is set, the assistance device 101 requests the data collection server 103 to transmit data to the assistance device 101 when data is transmitted from a specific IoT device 104. In response to this request, the data collection server 103 transmits data to the assistance device 101 when data is transmitted from the specific IoT device 104. At this time, the data collection server 103 may attach information about the IoT device 104 to the data and transmit it to the assistance device 101. The assistance device 101 launches a user-created program in response to data transmission from a specific IoT device. Instead of the data collection server 103 proactively transmitting data to the assistance device 101 as described above, the assistance device 101 may periodically inquire of the data collection server 103.

[0041] The text box 507 is an object for setting the IoT device 104 that is the activation condition. The IoT devices 104 may be set individually or in groups. In the above example, the activation condition is set to be that data is transmitted from an IoT device, but the activation condition may also be set to be that data is transmitted from a device other than an IoT device.

[0042] FIG. 5B shows the field 503 with the tab for setting a startup condition for the file server ("File") selected. When this tab is selected, the startup condition is set to be at least one of adding, updating, and deleting a file in the file server 105. When this startup condition is set, the support device 101 requests the file server 105 to notify the support device 101 that a specific operation (at least one of adding, updating, and deleting a file) has occurred in a specific folder. In response to this request, the file server 105 transmits a notification to the support device 101 when a specific operation has occurred in the specific folder. In this case, if the specific operation is adding or updating a file, the file server 105 may transmit the added or updated file to the support device 101. In response to receiving the notification from the file server 105, the support device 101 launches a user-created program. Instead of the file server 105 proactively sending a notification to the support device 101 as described above, the support device 101 may periodically inquire of the file server 105.

[0043] The text box 511 is an object for setting the path of a folder for which at least one of file addition, update, and deletion is monitored. The check box 512 is an object for setting that file addition is a startup condition. The check box 513 is an object for setting that file update is a startup condition. The check box 514 is an object for setting that file deletion is a startup condition.

[0044] 5C shows the field 503 in a state where a tab ("Timer") for setting a time-related activation condition is selected. When this tab is selected, a time-related condition is set as the activation condition. When this activation condition is set, the assistance device 101 activates the user-created program in response to the time-related condition being satisfied.

[0045] The text box 521 is an object for setting the name of the timer. The radio button 522 is an object for setting a method for defining a schedule. The text box 523 is an object for setting a time-related value. When "Periodic" is set using the radio button 522, the assistance device 101 starts the user-created program at the period set in the text box 523. When "Once" is set using the radio button 522, the assistance device 101 starts the user-created program at the time set in the text box 523. When "CRON" is set using the radio button 522, the assistance device 101 starts the user-created program using CRON at the time set in the text box 523. The assistance device 101 may display the specific most recent time set using CRON, or may display an explanation of the content set using CRON in natural language. This makes it easier for the user to recognize whether the setting was correct.

[0046] 5D shows the field 503 in a state where a tab ("API") for setting a start condition related to an API (Application Programming Interface) is selected. When this tab is selected, a condition related to the API is set as a start condition. When this start condition is set, the assistance device 101 starts a user-created program in response to a request from an external device via the API. Data may be provided to the user-created program from an external device via the API. Individual settings may not be made for the conditions related to the API.

[0047] In response to pressing the button 403, the assistance device 101 displays a setting screen 600 for acquiring settings related to an action from the user on the display device 204 of the user terminal 102. The setting screen 600 may be displayed superimposed on the creation screen 400, or may be displayed in place of the creation screen 400. An example of the setting screen 600 will be described with reference to Fig. 6. An action for which settings are made using the setting screen 600 is referred to as a target action.

[0048] The pull-down list 601 is an object for setting the type of the target action. In this embodiment, the type of the target action can be selected from five types: "AI," "data conversion," "object detection," "web hook," and "notification." The type of the target action may be selected from types other than these, or one or more of these five types may not be selectable. The individual processes that can be executed by the target action differ depending on the type of the target action. Details of each type will be described later.

[0049] The text box 602 is an object for setting the name of the target action. The name set in the text box 602 may be displayed on the icon representing the target action in the field 401 on the creation screen 400. The text box 603 is an object for setting a description of the target action.

[0050] The toggle switch 604 is an object for setting whether a target action is enabled or disabled. When the toggle switch 604 is set to enabled, the target action is enabled. In this case, the assistance device 101 operates the target action when the user-created program is executed. When the toggle switch 604 is set to disabled, the target action is disabled. In this case, the assistance device 101 does not operate the target action when the user-created program is executed.

[0051] The field 605 is an area where an object for setting a condition for executing an individual process of the target action is placed. The text box 606 is an object for setting a condition for executing an individual process of the target action. The condition for executing an individual process of the target action is represented as an execution condition. The execution condition may be written in an artificial language. The execution condition may be set to use data provided to the target action from an event source or another action. For example, the execution condition may be written as "payload.temperature>24". The data provided to the target action from another action may include the execution result of the individual process of this other action. "payload" represents data provided from an event source or another action. "payload.temperature" represents the value of a variable named "temperature" of this data. If an execution condition is not set, it may be determined that the execution condition is always met.

[0052] The field 607 is an area where objects for setting the content of individual processing of the target action are placed. The objects included in the field 607 may differ depending on the type selected in the pull-down list 601. The field 607 will be described in detail later.

[0053] The field 613 is an area where objects for setting the output of the execution results of individual processes of the target action are placed. The toggle switch 614 is an object for setting whether or not to output the execution results of individual processes of the target action to a channel. When the toggle switch 614 is set to enabled, the assistance device 101 outputs the execution results of individual processes of the target action to a channel when executing a user-created program. When the toggle switch 614 is set to disabled, the assistance device 101 does not output the execution results of individual processes of the target action to a channel when executing a user-created program.

[0054] The radio button 615 is an object for instructing whether to create a new output destination channel or select from existing channels. The text box 616 is an object for setting a channel name. When the radio button 615 is used to instruct creation of a new output destination channel, the name set in the text box 616 is assigned to this new channel. When the radio button 615 is used to instruct selection from existing channels, the channel with the name set in the text box 616 is selected. The radio button 615 and the text box 616 do not need to be displayed when the toggle switch 614 is set to disabled. The icon 619 is an object for displaying a field for making additional settings related to output. Details of the additional settings will be described later.

[0055] The button 617 is an object for acquiring an instruction from the user to cancel the setting of the target action. In response to the button 617 being pressed, the support device 101 cancels the addition of the target action. Specifically, the support device 101 closes the setting screen 600 and displays the creation screen 400 again. The button 618 is an object for acquiring an instruction from the user to create the target action. In response to the button 618 being pressed, the support device 101 creates the target action according to the settings specified by the user on the setting screen 600. Specifically, the support device 101 closes the setting screen 500, displays the creation screen 400 again, and adds an icon representing the target action to the field 401 of the creation screen 400. Furthermore, the support device 101 associates the settings specified by the user on the setting screen 600 with the target action and stores them in the memory 202 of the support device 101. At this time, the support device 101 may assign an individual component ID to the target action.

[0056] 6 shows the field 607 in a state where "AI" is selected as the action type in the pull-down list 601. When this type is selected, it is set to execute processing using the natural language processing service provided by the natural language processing server 107.

[0057] The pull-down list 608 is an object for setting the type of natural language processing service to be used. When executing a user-created program, the assistance device 101 requests processing from the natural language processing service set in the pull-down list 608. The response from the natural language processing service becomes the execution result of the target action. The check box 609 is an object for setting whether to instruct the natural language processing service to respond in a specific format (JSON format in the example of FIG. 6 ).

[0058] The check box 610 is an object for setting whether to use a system prompt in a request (called a prompt) to the natural language processing service. When the use of a system prompt is set, a text box for inputting the system prompt is displayed. The text box 611 is an object for setting a user prompt for the natural language processing service. The text box 611 can be set in natural language. Furthermore, the user prompt may be set to use data provided to the target action from an event source or another action. The data provided to the target action from another action may include the execution result of an individual process of this other action. For example, the user prompt input into the text box 611 may include the character string "${payload.temperature}". In this case, when executing the user-created program, the assistance device 101 replaces "${payload.temperature}" with the data provided from the event source or another action and then sends the user prompt to the natural language processing service.

[0059] The check box 612 is an object for setting whether to include an image in the prompt. If the check box 612 is set to include an image, the assistance device 101 sends a prompt including an image to the natural language processing service when executing a user-created program. In this way, when the type is "AI," multimodal data (in this example, text data and image data) can be set as targets for individual processing. Alternatively, other multimodal data (for example, voice data and text data, or voice data, image data, and text data) may be set.

[0060] 7A shows the field 607 in a state where "data conversion" is selected as the action type in the pull-down list 601. When this type is selected, it is set to execute data conversion processing.

[0061] The text box 701 is an object for setting the data format after conversion. The text box 702 is an object for setting the data to be converted. The data to be converted may be set to use data provided to the target action from an event source or another action. When executing a user-created program, the assistance device 101 converts the data set in the text box 702 into the format set in the text box 701. The converted data becomes the execution result of the target action. The text box 702 may be set to use data provided to the target action from an event source or another action.

[0062] 7B shows field 607 in a state where "object detection" is selected as the action type in pull-down list 601. When this type is selected, it is set to execute processing using the image analysis service provided by image analysis server 106.

[0063] The text box 711 is an object for setting image data to be analyzed. The data to be analyzed may be set to use data provided to the target action from an event source or another action. The data to be analyzed may be video or still images. The image analysis service may respond with the number of specific objects (e.g., people, cars, buses, trucks, cats, and dogs) contained in the image data to be analyzed. When executing a user-created program, the assistance device 101 requests the image analysis service to analyze the image data set in the text box 711. The response from the image analysis service is the execution result of the target action. Options for preprocessing the image analysis service, such as distortion correction and color correction, before performing image analysis may be set. Based on the results of the image analysis, for example, based on the type and location information of the detected object, the assistance device 101 can visually highlight noteworthy objects or locations by drawing marks such as rectangles or circles on the image (e.g., surrounding the target area with a red frame). The assistance device 101 may assign different marks or labels to multiple objects. In addition to being based on the results of image analysis, the marking or labeling may be based on a predefined area such as a target object or location that requires attention. The assistance device 101 can also analyze the state of the target in that area.

[0064] 7C shows the field 607 in a state where "Web Hook" is selected as the action type in the pull-down list 601. When this type is selected, it is set to execute processing using a web service.

[0065] The pull-down list 721 is an object for setting the method of the request. The text box 722 is an object for setting the destination of the request. The text box 723 is an object for setting the header of the request. The radio button 724 is an object for setting the format of the body of the request. The text box 725 is an object for determining the contents of the body of the request.

[0066] When the support device 101 executes a user-created program, it creates a request in accordance with the settings in the pull-down list 721, text box 723, radio button 724, and text box 725, and sends the request to the destination specified in the text box 722. The response to this request becomes the execution result of the target action.

[0067] 7D shows the field 607 in a state where "Notify" is selected as the action type in the pull-down list 601. When this type is selected, it is set to notify other devices.

[0068] The text box 731 is an object for setting the notification destination. A URL (Uniform Resource Locator), an email address, or the like may be set as the notification destination. The text box 732 is an object for setting data to be included in the notification. The data to be included in the notification may be set to use data provided to the target action from an event source or another action.

[0069] When the assistance device 101 executes the user-created program, it sends a notification including the data specified in the text box 732 to the destination specified in the text box 731. The response to this notification becomes the execution result of the target action.

[0070] 7E shows a field 740 that is displayed when icon 619 is pressed. Objects for configuring settings related to output are arranged in field 740. Check box 741 is an object for configuring settings for converting data to be output. When check box 741 is checked, a window for configuring data conversion rules is displayed. Examples of conversion rules that can be configured include extracting only a specific portion of the data to be output, or adding input parameters for an action to the output.

[0071] The text box 742 is an object for setting the delay time before the execution result of the action is output. For example, there may be a time lag of several seconds between receiving an event notification from a surveillance camera and actually uploading the recorded data to the cloud. In such a case, by setting the output delay time for the download action to, for example, 10 seconds, the download of the recorded data will begin 10 seconds later. By setting a time that is empirically known to be the time required for the upload to be completed, the recorded data will be downloaded after the upload is complete.

[0072] Check box 743 is an object for setting whether the array data to be output is divided into elements. If check box 743 is not checked, the array data to be output is output as is. If check box 743 is checked, a new individual event is generated for each element of the array data to be output, and the resulting event is output as the next action. For example, if an action to reset a surveillance camera is specified as the output destination of an action to obtain a list of surveillance cameras, checking the check box 743 for the former action allows resetting each individual surveillance camera.

[0073] When the check box 743 is checked, the delay time set in the text box 742 may be applied only to the output of the first element. The delay time may be set separately for the output of the second and subsequent elements. This makes it possible to avoid violating any limit on the number of times the API can be used per unit time.

[0074] Next, channel settings will be described with reference to Figures 8A and 8B. Figure 8A shows field 401 to which five icons 801 to 805 representing program parts have been added. Icon 801 represents an event source for which a start condition related to a file server has been set. Icon 802 represents an "object detection" type action. Icon 803 represents an "AI" type action. Icon 804 represents a "notification" type action. Icon 805 represents a "notification" type action. Each icon displays the name set in text box 501 or 602.

[0075] The assistance device 101 displays an icon 806 representing a channel in association with an icon 801 representing an event source (in the example of FIG. 8A , so as to be in contact with the right side of the icon 801). Furthermore, when the toggle switch 614 is set to enable output of the execution results of an action to a channel, the assistance device 101 displays an icon representing a channel in association with the icon representing an action. In the example of FIG. 8A , it is assumed that output to a channel is enabled for the actions "object detection" and "object analysis." Therefore, the assistance device 101 displays an icon 807 representing a channel in association with the icon 802, and an icon 808 representing a channel in association with the icon 803. On the other hand, in the example of FIG. 8A , it is assumed that output to a channel is disabled for the actions "report" and "alert." Therefore, the assistance device 101 does not display icons representing channels in association with the icons 804 and 805.

[0076] In response to a user's selection of any of the icons representing channels, the assistance device 101 acquires from the user a setting for an action for outputting data requested to that channel. For example, assume that the output destination of icon 806 is set to the action "object detection," the output destination of icon 807 is set to the action "report," and the output destination of icon 808 is set to the action "alert." In response to this, the assistance device 101 updates the display of field 401 as shown in FIG. 8B . In FIG. 8B , icon 806 is displayed so as to connect icon 801 and icon 802. Data output sources and destinations are similarly connected for the other icons 807 and 808.

[0077] Furthermore, the assistance device 101 may be able to acquire an instruction from a user to add an output destination channel for an event source and an action. In response to acquiring this instruction, the assistance device 101 adds and displays an icon representing a channel in association with the icon representing the event source and the action. The data output destination can also be set for this added channel. For example, suppose a channel is added for the action "object detection" and the output destination of this channel is set to the action "object analysis." In this case, as shown in FIG. 8B , the assistance device 101 adds an icon 809 connecting icon 802 and icon 803.

[0078] By configuring the channel as described above, it is possible to set an action that is activated when the activation condition for the event source is satisfied. For example, in the example of FIG. 8B , it is set that the action "Object Detection" is activated when the activation condition set for the event source "Image File" is satisfied. Furthermore, it is possible to set an action that is activated after individual processing of the action is executed. For example, in the example of FIG. 8B , it is set that the action "Object Analysis" and the action "Report" are activated after individual processing of the action "Object Detection" is executed.

[0079] The user may be able to change the settings of the program part represented by an icon by selecting the icon displayed in field 401. The user may also be able to delete the icon displayed in field 401.

[0080] 3, in S303, the support device 101 determines whether a completion instruction has been received from the user. For example, the support device 101 determines that a completion instruction has been received from the user in response to pressing of button 404. If the support device 101 determines that a completion instruction has been received ("YES" in S303), the process proceeds to S304; otherwise ("NO" in S303), the process proceeds to S302.

[0081] In S304, the assistance device 101 creates a program based on the settings for the multiple program components represented by the icons displayed in field 401. Specifically, the assistance device 101 may manage the settings for the multiple program components as a single program. For example, the assistance device 101 may assign a common program ID to the multiple program components, or may manage the settings for the multiple program components in a single file. The assistance device 101 may store the created program in the secondary storage device 206 of the assistance device 101 for subsequent processing, or may transmit the created program to another device (e.g., the user terminal 102).

[0082] The user may load the program created as described above onto the creation screen 400. In this case, the assistance device 101 displays icons representing each program part in the field 401 based on the settings of the program parts of the stored program.

[0083] By using the program creation service provided as described above, users can create programs efficiently. For example, because the text box 611 allows users to set the processing details in natural language, appropriate execution results can be obtained even without proficiency in artificial languages. Furthermore, because it is possible to individually set whether program components are enabled or disabled, it is possible to easily change the behavior of a program while maintaining the structure between multiple program components. Furthermore, because users can easily select the external services (natural language processing services and image analysis services) to use, it is possible to gradually set the processing details according to the processing load.

[0084] An example of the operation of the assistance device 101 for executing a user-created program will be described with reference to Fig. 9. The operation of Fig. 9 may be performed by the processor 201 of the assistance device 101 executing a program read into the memory 202 of the assistance device 101. Alternatively, some or all of the steps of the operation of Fig. 9 may be performed by a dedicated integrated circuit such as an ASIC. The operation of Fig. 9 may be started in response to the activation of the assistance device 101.

[0085] In S901, the assistance device 101 determines whether the activation condition set for any valid event source is satisfied. If the assistance device 101 determines that the activation condition is satisfied ("YES" in S901), the process proceeds to S902. Otherwise ("NO" in S901), the assistance device 101 repeats S901. In this way, the assistance device 101 waits until the activation condition set for any valid event source is satisfied. The assistance device 101 does not determine the activation condition for an invalid event source.

[0086] In S902, the assistance device 101 activates an action connected through a channel to an event source whose activation condition is satisfied. If multiple actions are connected to the event source, the assistance device 101 activates each of these multiple actions. The activation of an action may be performed by sending a message from the event source through the channel. This message may include data set in the activation condition.

[0087] In S903, the assistance device 101 determines whether the activated action is valid and the execution conditions are satisfied. If the assistance device 101 determines that the activated action is valid and the execution conditions are satisfied (YES in S903), it transitions the process to S904, and otherwise (NO in S903), it terminates the process for this action. Whether the activated action is valid or invalid is set by the toggle switch 604.

[0088] In S904, the support device 101 executes the individual processing set for the activated action.

[0089] In S905, the assistance device 101 determines whether the output of the activated action is valid. If the assistance device 101 determines that the output of the activated action is valid ("YES" in S905), it transitions the process to S902; otherwise ("NO" in S905), it terminates the process for this action. Whether the output of the activated action is valid or invalid is set by the toggle switch 614. When the process transitions to S902, the assistance device 101 initiates another action connected through a channel to the action that has completed execution of an individual process. If multiple actions are connected to the action that has completed execution of an individual process, the assistance device 101 initiates each of these multiple actions. The activation of an action may be performed by sending a message through a channel. This message may include the execution result of the individual process.

[0090] In the above example, the support device 101 provides both the creation support service and the execution of the user-created program. However, the provision of the creation support service and the execution of the user-created program may be performed by separate devices. The user may also cause a separate device to execute the user-created program.

[0091] In the above example, the user uses the user terminal 102 to use the creation support service provided by the support device 101. Alternatively, application software that provides the creation support service may be installed on the user terminal 102, and the user may generate a program using this application software. In this case, the user terminal 102 executes the operations shown in FIG. 3 above, and the user terminal 102 functions as the support device.

[0092] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0093] This application claims priority based on Japanese Patent Application No. 2024-112620, filed July 12, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An assistance device for assisting in the creation of a program, comprising: a display means for displaying on a display device graphic objects representing program parts instructed by a user to be added to the program; an acquisition means for acquiring settings related to the program parts from the user; and a creation means for creating the program based on the settings related to a plurality of program parts acquired from the user, wherein the program parts are of a plurality of types including a first type that specifies a start condition for starting the program, and a second type that specifies individual processing of the program, wherein the assistance device is capable of setting, for the program parts of the first type, a program part of the second type that is started in response to the start condition being satisfied, and for the program parts of the second type, the content of the individual processing and another program part of the second type that is started after the individual processing is executed, and wherein, for at least one program part of the second type, the content of the individual processing can be set in natural language.

2. The support device according to claim 1, wherein multimodal data can be set as the target of said individual processing for at least one program part of said second type.

3. The support device according to claim 1 or 2, further capable of setting whether or not to launch another program component of the second type after the individual process is executed, with respect to the program component of the second type.

4. The support device according to any one of claims 1 to 3, further capable of being set to use data provided from another program part in the individual processing of the second type program part.

5. The support device according to any one of claims 1 to 4, wherein conditions for executing the individual processes can be further set for the second type program parts.

6. The support device according to any one of claims 1 to 5, wherein the activation conditions include at least one of: data being sent from a specific device; at least one of a file being added, updated, or deleted on a file server; a time-related condition; and a request being made through an API.

7. The support device according to any one of claims 1 to 6, wherein whether the operation of a program segment is enabled or disabled can be set individually for each program segment.

8. The support device according to any one of claims 1 to 7, further comprising an execution means for executing the program in response to the start condition being satisfied.

9. A program for causing a computer to function as each means of the support device according to any one of claims 1 to 8.

10. A method for assisting in the creation of a program, comprising: a display step in which a display means displays on a display device graphic objects representing program parts instructed by a user to be added to the program; an acquisition step in which an acquisition means acquires settings related to the program parts from the user; and a creation step in which a creation means creates the program based on the settings related to a plurality of program parts acquired from the user, wherein the program parts are of a plurality of types including a first type that specifies a start condition for starting the program, and a second type that specifies individual processing of the program, and wherein, for the program part of the first type, it is possible to set a program part of the second type that is started in response to the start condition being satisfied, and for the program part of the second type, it is possible to set the content of the individual processing and another program part of the second type that is started after the individual processing is executed, and for at least one program part of the second type, it is possible to set the content of the individual processing in natural language.

Images