Equipment motion generation support method, equipment motion generation support device, and equipment motion generation system

Abstract

To support the generation of equipment operations from action generation commands given in natural language. [Solution] The equipment operation generation support device 10 includes a user dialogue unit 12 that receives user instructions written in natural language to generate equipment operations, and an operation generation instruction unit 13 that reads an extraction prompt from the extraction item DB 17 for each extraction item of operation generation parameters to extract the extraction item from the user instructions, inputs the user instructions and the extraction prompt into a large-scale language model to obtain operation generation parameters included in the user instructions, and outputs a generation instruction to generate operation parameters based on the obtained operation generation parameters.

Description

Technical Field

[0001] The present invention relates to a facility operation generation support method, a facility operation generation support device, and a facility operation generation system.

Background Art

[0002] For labor saving, automated equipment such as robots and conveyors is utilized in various operations. These facilities perform operations such as movement according to input operation parameters such as target positions. Patent Document 1 describes providing a control method and a control system for a vacuum cleaner that allows a user to easily understand a planned movement route of the vacuum cleaner on a terminal device at hand and also easily change the route, providing high convenience. Patent Document 2 describes that a tidying robot can store any daily item at an arbitrary position and tidy it in an arbitrary empty space in response to a human instruction or spontaneously.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] To generate operation parameters that define the operation of a facility (facility operation), in many cases, the generation instruction is described in a unique programming language different for each facility. Therefore, the user needs to be familiar with the unique programming language, and the large user load has become an obstacle to the introduction of automated equipment. Thus, a mechanism that allows the user to set the facility operation without directly touching the programming language is required.

[0005] Patent Document 1 describes a method of instructing the destination using voice commands. However, the method described in Patent Document 1 requires registering a voice command to be recognized for each action, and it cannot accommodate changes in the user's phrasing. Patent Document 2 describes a system that interprets human voice commands and executes commands to tidy up, but, similar to Patent Document 1, it requires registering voice commands to be recognized for each action, and it cannot adapt to changes in the user's phrasing.

[0006] To accommodate changes in user phrasing, we are considering using large language models (LLMs) such as ChatGPT (registered trademark) to provide interactive programming and data processing instructions. In this case, registration of actions becomes unnecessary, and natural language commands can be interpreted even if the user's phrasing changes. However, a problem inherent to large-scale language models is that, due to the characteristics of their model structure, they cannot handle spatial information (information in three dimensions). Specifically, large-scale language models, due to the relationship between the training data and the training model structure, can only handle one-dimensional information such as the contextual relationships between different parts of a situation. Therefore, they cannot handle spatial information such as action generation, and thus cannot generate equipment actions.

[0007] Therefore, we consider a case where a large-scale language model is linked with a motion generation processing device that has the function of generating motion parameters to generate equipment movements. However, since the large-scale language model cannot handle spatial information, it is not possible to extract generation instructions (such as start and end point coordinates and velocity) to the motion generation processing device from the user's instructions in natural language.

[0008] This invention was made in view of the above circumstances, and aims to support the generation of equipment operations from operation generation commands instructed in natural language. [Means for solving the problem]

[0009] To solve the above problems, the equipment operation generation support device of the present invention has the following features. The present invention relates to an equipment operation generation support device having a user input unit and an operation generation instruction unit. The user input unit receives user instructions written in natural language in order to generate the operation of the equipment. The aforementioned motion generation instruction unit, For each extraction item of the parameters for generating the operation, an extraction prompt for extracting the extraction item from the user instruction content is read from the storage unit, and the user instruction content and the extraction prompt are input to the large-scale language model to obtain the parameters for generating the operation included in the user instruction content. The system is characterized by outputting a generation instruction to generate operation parameters that indicate the operation of the equipment, based on the acquired operation generation parameters. Other features will be described later. [Effects of the Invention]

[0010] According to the present invention, it is possible to support the generation of equipment operations from operation generation commands instructed in natural language. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram illustrating the configuration of the equipment operation generation system according to this embodiment. [Figure 2] This is a state transition diagram of the model control unit according to this embodiment. [Figure 3] This is a screen diagram showing an example of the message content of the user interaction unit and the operation generation instruction unit according to this embodiment. [Figure 4] This table shows an example of an extraction item database related to this embodiment. [Figure 5] This is a table showing an example of a processing instruction database related to this embodiment. [Figure 6] This is a table showing large-scale language model information related to this embodiment. [Figure 7] This is a table showing equipment information related to this embodiment. [Figure 8]It is a flowchart showing the processing of the equipment operation generation support device according to the present embodiment. [Figure 9] It is a first screen diagram showing the state before the additional instruction among the processing results of the equipment operation generation support device according to the present embodiment. [Figure 10] It is a second screen diagram showing the state after the additional instruction among the processing results of the equipment operation generation support device according to the present embodiment. [Figure 11] It is a hardware configuration diagram of each device of the equipment operation generation system according to the present embodiment.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] FIG. 1 is a configuration diagram of an equipment operation generation system 100. The equipment operation generation system 100 is configured by connecting an equipment operation generation support device 10 and an operation generation processing device 20 via a network. The operation generation processing device 20 reads generation parameters for equipment operation (hereinafter referred to as "operation generation parameters") from a generation instruction from the equipment operation generation support device 10. Then, the operation generation processing device 20 generates operation parameters of the equipment operation that satisfy the operation generation parameters, and returns the operation parameters to the equipment operation generation support device 10. Therefore, the operation generation processing device 20 operates, for example, a physical simulator such as a robot simulator or analysis software such as numerical analysis software.

[0014] Here, the "equipment operation" means, for example, when the equipment is an industrial robot, the tip of its arm moves. Depending on the equipment operation, the geometric shape of the equipment may change. Furthermore, motion generation parameters are parameters used in calculations during the process of generating equipment motion, such as physical constraints like the start point, end point, and velocity of the arm tip. Motion parameters are parameters that indicate equipment motion that conforms to the physical constraints of the motion generation parameters (such as the movement path of the arm tip).

[0015] When the equipment operation generation support device 10 receives instructions from the user (user instructions), it transmits the operation generation parameters generated from those user instructions to the operation generation processing device 20. Furthermore, the equipment operation generation support device 10 may accept user instructions from the user in a natural language dialogue format that can be handled by large-scale language models such as ChatGPT (registered trademark). In this case, the equipment operation generation support device 10 converts the user instructions into parameters for operation generation and outputs them to the operation generation processing device 20. In other words, the equipment operation generation support device 10 is a device that mediates between the user and the operation generation processing device 20. This allows users to have the motion generation processing device 20 generate motion parameters without needing to know advanced technical knowledge such as programming languages ​​or input formats for motion generation parameters.

[0016] Furthermore, the equipment operation generation support device 10 may use either text input via a keyboard or voice input via a microphone as a means of inputting user instructions in natural language. In the case of voice input, the equipment operation generation support device 10 converts the voice signal into text, thereby obtaining text data in the same way as with text input. Similarly, the equipment operation generation support device 10 may use screen display via a display or audio output via a speaker as a means of outputting text data in natural language. Even in the case of audio output, the equipment operation generation support device 10 can convert the text data into an audio signal through a text-to-speech process.

[0017] The equipment operation generation support device 10 has a model control unit 11, a user interaction unit 12, and an operation generation instruction unit 13 as processing units, and also has a storage unit that stores a processing instruction DB 16 and an extraction item DB 17. This storage unit may be configured as a storage device that can be accessed from outside the equipment operation generation support device 10. The user interaction unit 12 receives user instructions written in natural language to generate the operation of the equipment. Therefore, the user interaction unit 12 receives user instructions from, for example, the content of the interaction with the user using a large-scale language model.

[0018] The motion generation instruction unit 13 reads an extraction prompt from the extraction item DB 17 for each extraction item of motion generation parameters, and inputs the user instruction content and the extraction prompt into the large-scale language model to obtain the motion generation parameters included in the user instruction content. The motion generation instruction unit 13 then outputs a generation instruction to generate motion parameters that indicate the operation of the equipment, based on the acquired motion generation parameters.

[0019] The model control unit 11 manages the state transitions shown in Figure 2 and controls each process for generating equipment operation by assigning roles to the large-scale language model (user interaction unit 12, operation generation instruction unit 13). The Extraction Item DB17 (details in Figure 4) is a database of extraction item information referenced by the Operation Generation Instruction Unit 13. The Processing Instruction DB16 (details in Figure 5) is a database that manages information on processing instructions referenced by the Model Control Unit 11.

[0020] Figure 2 is a state transition diagram of the model control unit 11. The model control unit 11 selects a prompt to use from the processing instruction DB 16 according to the transition of each state it holds internally, and assigns roles to the user interaction unit 12 and the operation generation instruction unit 13. For this reason, the model control unit 11 internally holds the user interaction state 111, the extraction processing state 112, the operation generation processing state 113, and the missing information request processing state 114 as separate states. The following clarifies each state in Figure 2, referring to Figure 3.

[0021] Figure 3 is a screen diagram showing an example of the message content of the user interaction unit 12 and the operation generation instruction unit 13. The screen of the user interaction unit 12 displays input statements 121 and 123 from the user to the user interaction unit 12, and output statement 122 from the user interaction unit 12 to the user, displayed in order from top to bottom. The screen of the motion generation instruction unit 13 displays, from top to bottom, the input statement 131 to the motion generation instruction unit 13 and the output statement 132 from the motion generation processing device 20. Furthermore, the arrow from input statement 121 to input statement 131 indicates that input statement 121 was notified from the user interaction unit 12 to the operation generation instruction unit 13. Similarly, the arrow from output statement 132 to output statement 122 indicates that output statement 132 was notified from the operation generation instruction unit 13 to the user interaction unit 12.

[0022] First, in the user dialogue state 111, the user dialogue unit 12 interprets the user instructions contained in the input statement 121 and transmits them to the operation generation instruction unit 13. Next, the model control unit 11 transitions the state from the user dialogue state 111 to the extraction processing state 112. The operation generation instruction unit 13 selects the prompt to be used from the extraction item DB 17 as the input statement 131. The motion generation instruction unit 13 inputs the input statement 131 and the prompts from the extracted item DB 17 into a large-scale language model to extract motion generation parameters from the user's instructions.

[0023] Next, the model control unit 11 transitions from the extraction processing state 112 to the motion generation processing state 113. The motion generation instruction unit 13 generates a function call statement that takes the extracted motion generation parameters as arguments and notifies the motion generation processing device 20 of the function call statement. The operation generation instruction unit 13 may also generate arguments by performing data conversion processing (such as format conversion processing) on ​​the extracted operation generation parameters to make them suitable for various function call statements, such as converting meters to millimeters. The motion generation instruction unit 13 receives motion parameters as shown in the output statement 132 as a result of the motion generation processing of the motion generation processing device 20, which was executed by the function call statement.

[0024] Next, the model control unit 11 transitions from the motion generation processing state 113 to the user interaction state 111. The user interaction unit 12 displays the processing result received by the motion generation instruction unit 13 in a user-friendly format, such as an output statement 122. Furthermore, if there are additional instructions from the user, such as input statement 123, the model control unit 11 will execute the series of state transitions shown above again, repeating the state transitions until the user obtains the desired result.

[0025] In this state, if the input information for the motion generation processing device 20 is insufficient in the extraction processing state 112, the model control unit 11 transitions to the insufficient information request processing state 114. The user interaction unit 12 requests the user to input additional information until there is no more insufficient input information. Once all the input information is complete, the model control unit 11 transitions to the motion generation processing state 113.

[0026] Figure 4 shows a table illustrating an example of the extracted items DB17. Extraction item DB17 associates equipment type 171 with extraction item 172 and prompt 173. Equipment type 171's reference point is determined by the equipment that generates the operation described in the user's instructions. Thus, the extraction prompt (prompt 173) in the extraction item DB17 is defined for each equipment type 171. The operation generation instruction unit 13 then retrieves the extraction prompt corresponding to the equipment type 171 described in the user instruction from the extraction item DB17.

[0027] Extraction item 172 is an item extracted from user instructions for use in the process of generating actions. Note that Extraction Item DB17 may also contain information indicating the degree of necessity for each Extraction Item 172 (such as the [Required] notation for the starting position and orientation). Then, if the user interaction unit 12 is required to obtain the necessary parameters for generating the operation included in the user's instructions, the operation generation instruction unit 13 prompts the user to input additional parameters for the extracted items that could not be obtained.

[0028] Furthermore, item 172 is identified from equipment type 171. For example, in the case of equipment type "mobile robot," the movement range of the device is planar (X axis, Y axis), but in the case of equipment type "vertical articulated robot," the movement range of the tip of the robot arm is three-dimensional (X axis, Y axis, Z axis). Then, prompt 173 is input to the large-scale language model so that the action generation instruction unit 13 can extract information about the extracted item 172 from the user instruction content.

[0029] The first record 174, which extracts the starting position and orientation as item 172, has the role name "You are a robot teacher" as the prompt 173. This makes it easier for the motion generation instruction unit 13 to provide a response specific to the role name. For example, as preparation, teaching history data for each role name can be provided to the motion generation instruction unit 13 as a prompt through additional learning or RAG (Retrieval-Augmented Generation). The motion generation instruction unit 13 then refers to the teaching history data for the role name included in the prompt 173. However, the role name can be omitted. Furthermore, for role names such as "inspector," the operation generation instruction unit 13 is instructed to perform a more rigorous check.

[0030] Furthermore, the prompt 173 of the first record 174 contains the phrase "(User instruction) from". This indicates that the user instruction contained in the input statement 121 in Figure 3 will be directly substituted into prompt 173. Furthermore, from the prompt 173 of the first record 174, it can be seen that a total of six motion generation parameters will be extracted: three parameters (X, Y, Z) indicating the starting point position at the tip of the robot arm, and three parameters (roll, pitch, yaw) indicating the posture information of the robot arm. Furthermore, the "m units, degree units" in prompt 173 of the first record 174 indicates that data conversion processing is required to match the units.

[0031] Thus, the extraction prompt (prompt 173) of the extracted item DB17 includes at least one of the following: the role name when the large-scale language model is operated as the action generation instruction unit 13, the name of the extracted item, and the unit of the extracted item. The action generation instruction unit 13 then converts the data format (units, etc.) of the acquired action generation parameters into a data format that can be accepted as a generation instruction. For the second record 175 for extracting the endpoint position and orientation, the third record 176 for extracting the maximum transport speed, and the fourth record 177 for extracting the transport quality, each prompt 173 also specifies the role name and parameters for generating the motion.

[0032] Furthermore, the transport quality in the fourth record 177 is a motion generation parameter that indicates the state of the object being transported by the operation of the equipment. For example, consider the case where a robot arm lifts a box of objects and generates motion parameters to move the box. Even if the starting and ending points of a box's movement are in the same position, a movement that carefully moves the box with minimal vibration and rotation is considered to have higher transport quality than a rough movement that applies strong vibration and rotation to the box, because the contents of the box are less likely to collapse. By extracting transport quality in this way and including it in the motion generation parameters, it is possible to generate motion parameters suitable for objects that are sensitive to vibration, such as precision equipment. Thus, the extraction prompts for the extraction item DB17 include prompts for extracting transport quality, which indicates the state of the object being transported by the operation of the equipment.

[0033] Furthermore, the role name and motion generation parameters are the same in the first record 174 for the starting position and orientation and the second record 175 for the ending position and orientation. In this case, the motion generation instruction unit 13, which has read the prompt 173, also refers to the following pre-learned inference rules to distinguish between the starting position and orientation and the ending position and extracts both. Inference rule = When two location pieces are extracted, "(1st location) to (2nd location)", the location piece written before the particle "kara" (1st location) is considered the starting point, and the location piece written after "kara" (2nd location) is considered the ending point. Thus, the extraction prompt for the extraction item DB17 includes instructions for extracting at least one of the following extraction items: the starting position and orientation of the equipment in operation, the ending position and orientation of the equipment in operation, and the operating speed of the equipment.

[0034] Figure 5 shows an example of a table in the processing instruction DB16. The processing instruction DB16 associates the processing content 161 corresponding to each state in Figure 2 (user interaction state 111 to missing information request processing state 114) with the prompts 162 to be input to the user interaction unit 12 or the operation generation instruction unit 13. Note that the state transition content of the processing instruction DB16 table changes depending on the equipment generating the operation, and may include items other than the processing content described. The first record 163 is input to the user interaction unit 12 in accordance with the user interaction state 111. Similar to Figure 4, in Figure 5, the role name "Operator" is assigned, and the "(User Instructions)" part is replaced with the user instructions contained in the input statement 121 in Figure 3. The second record 164 is input to the operation generation instruction unit 13 in accordance with the extraction processing state 112. The third record 165 is input to the operation generation instruction unit 13 in accordance with the operation generation processing state 113. The fourth record 166 is input to the user interaction unit 12 in accordance with the missing information request processing state 114.

[0035] Figure 6 is Table 400, which shows information about large-scale language models. Table 400 associates the processing content 401 corresponding to each state in Figure 2 (user interaction state 111 to missing information request processing state 114), the name of the large-scale language model to be used 402, and the parameters 403 to be set in the large-scale language model. The model control unit 11 sets the name of the large-scale language model to be used 402 and the parameters 403 in the user interaction unit 12 and the operation generation instruction unit 13 according to the processing content 401. Parameter 403 specifies the degree of response, which can take a value between 0 and 1. The degree of response is closer to 0 when an accurate answer is required, such as in extraction processing state 112, and closer to 1 when an answer from a new perspective, even if uncertain, is required. In user interaction state 111, the person interacting is a user who can understand to some extent, and an answer from a new perspective is required, so the degree of response is set to 1.0.

[0036] Figure 7 is Table 500, which shows the equipment information. Table 500 associates the equipment name 501 with the equipment type 502 and the equipment's maximum speed 503. Table 500 may also include other items such as maximum acceleration and maximum gripping force. For example, if the user gives instructions other than numerical values, such as "Generate the action at maximum speed," the large-scale language model (user interaction unit 12 and action generation instruction unit 13) will refer to the equipment's maximum speed of 503.

[0037] Figure 8 is a flowchart showing the processing of the equipment operation generation support device 10. In the input processing of instruction information from the user (S11), the model control unit 11 transitions to the user dialogue state 111 as an initial state. At this time, the model control unit 11 inputs the prompt 162 of the first record 163 in Figure 5 to the user dialogue unit 12, assigning it the role of an operator to interact with the user. Subsequently, the user interaction unit 12 receives the instruction information entered by the user and transmits that instruction information (user instruction content) to the operation generation instruction unit 13. The user interaction unit 12 then transmits information to the model control unit 11 that it has transmitted the user instruction content to the operation generation instruction unit 13, causing the model control unit 11 to transition to the extraction processing state 112.

[0038] In the process of reading extracted item information (S12), the model control unit 11 inputs the prompt 162 of the second record 164 in Figure 5 to the operation generation instruction unit 13, giving it the role of an inspector to check if there is any missing information in the user instruction. Next, the operation generation instruction unit 13 refers to the extracted item DB 17 in Figure 4 and reads all the prompts 173 of the extracted items for the relevant equipment.

[0039] In the process of extracting information necessary for generating an action and converting it to an input format (S13), the action generation instruction unit 13 executes the read prompt 173 to extract action generation parameters from the user instruction content and converts them to the input format of the action generation processing device 20, such as by generating a function call statement that takes the extracted action generation parameters as arguments.

[0040] In the determination process (S14) to determine whether all necessary items have been extracted, the operation generation instruction unit 13 reads the prompt for the second record 164 in Figure 5 and checks if there is any missing information. For example, in S13, which involves a vertical articulated robot, if the starting position and orientation information (= missing information) cannot be extracted from the user's instructions despite the execution of prompt 173 (resulting in No in S14), the process proceeds to instruct the user to add information (S15). At this time, the motion generation instruction unit 13 transmits to the model control unit 11 that it has instructed the user to input the missing information, causing the model control unit 11 to transition to the missing information request processing state 114.

[0041] In S15, the model control unit 11 first inputs the prompt for the fourth record 166 in Figure 5 to the user interaction unit 12, giving the user interaction unit 12 the role of instructing the user to input the missing information. Next, the user interaction unit 12 receives the item name of the missing information from the operation generation instruction unit 13 and shows the item name to the user, instructing them to input it. Then, the user interaction unit 12 transmits the missing information (additional information) that the user has additionally entered to the operation generation instruction unit 13. By transmitting information that this transmission has been made from the user interaction unit 12 to the model control unit 11, the model control unit 11 transitions to the extraction processing state 112 (S12).

[0042] If all necessary items have been extracted (Yes in S14), the motion generation processing unit 20 is used to perform motion generation (S16). At this time, the motion generation instruction unit 13 transmits to the model control unit 11 that there is no missing information, and the model control unit 11 transitions to the motion generation processing state 113. The model control unit 11 inputs the prompt for the third record 165 in Figure 5 to the motion generation instruction unit 13, assigning it the role of a teacher to generate equipment operations. Next, the motion generation parameters generated in S13 are input to the motion generation processing device 20, which generates the operation parameters.

[0043] In the output processing of the processing result (S17), the motion generation instruction unit 13 notifies the user interaction unit 12 of the motion parameters received from the motion generation processing device 20. By transmitting information that this notification has been made from the motion generation instruction unit 13 to the model control unit 11, the model control unit 11 transitions to the user interaction state 111. The model control unit 11 gives the user interaction unit 12 the prompt of the first record 163 in Figure 5, returns to the role of an operator that interacts with the user, and presents the processing result to the user.

[0044] In the confirmation process (S18) to determine whether there are any additional instructions from the user, the user interaction unit 12 responds to any additional instructions from the user. If there are additional instructions (Yes in S18), the model control unit 11 reconcile the previous user instructions with the additional instructions and then restarts the process from S12. Here, consistency of instructions means, for example, if there is an additional instruction to reduce the maximum speed, only the value of the maximum speed from the previous instructions is changed. If there are no additional instructions (No in S18), the process ends.

[0045] Figure 9 is a first screen view showing the state of the equipment operation generation support device 10 before the additional instruction in S18, which is the processing result of the device. The output screen 600 is a screen displayed to generate equipment operations interactively based on user instructions, and includes an operation generation result 606, an operation performance table 607, a chat screen 603, a text box 601, and a send button 602. Text box 601 is an input field that accepts text input from the user, and the content entered in text box 601 is treated as user instruction. The send button 602 is a button that determines whether to transmit the text information entered in text box 601 to the large-scale language model (user interaction unit 12).

[0046] The chat screen 603 is a screen that displays the message content from the user interaction unit 12 (input sentences 121, 123 and output sentence 122 in Figure 3), showing the user's instructions and the interaction with the large-scale language model. The speech bubble on the right side of the chat screen 603 (e.g., speech bubble 605) is user input, and the speech bubble on the left side (e.g., speech bubble 604) is the response from the user dialogue unit 12. The callout 604 displays that because two extraction items (transport speed and transport quality) could not be extracted from the user's instructions, the user interaction unit 12 is requesting the user to provide the missing items (the two extraction items that could not be extracted) based on the fourth record 166 in Figure 5.

[0047] The callout 605 displays the information that the user has filled in to complete the missing items requested in callout 604. Even if instructions are given in language other than numerical values, such as "transport speed is maximum speed," the operation generation instruction unit 13 can extract the numerical value of the transport speed by referring to the maximum speed 503 of the equipment in Figure 7. Furthermore, the user instruction "I don't care about transport quality" is an instruction to not include the missing item in the parameters for generating the operation, instead of adding the missing item. As a result, the operation generation instruction unit 13 can omit the extraction process for the missing item "transport quality".

[0048] The motion generation result 606 is a screen that displays the motion parameters created by the motion generation processing device 20 as an animated video of the equipment (robot arm). This allows the user to check, through the animation, whether the motion generated matches the instructions entered in the chat screen 603. The performance table 607 shows the performance when the equipment is operated using the operation parameters created by the operation generation processing device 20. This allows the user to confirm, as a numerical value for each attribute, whether or not the operation was generated according to the instructions entered on the chat screen 603.

[0049] Figure 10 is a second screen view showing the state after the additional instruction S18 in the processing results of the equipment operation generation support device 10. The motion generation instruction unit 13 prompts the user to input additional information (additional instruction) for any extraction items for which it could not obtain the motion generation parameters included in the user instruction. If the additional input indicates that the extraction item is not essential, the unit omits the extraction of the extraction item that could not be obtained. For example, in the first screen diagram of Figure 9, the user gives additional instructions via the chat screen 701, resulting in the action generation result 606 in Figure 9 being modified to the action generation result 703 in Figure 10. Blower 701A indicates that, in terms of conveying quality, instructions are given to minimize the amount of change in posture.

[0050] The speech bubble 702 is an example of how the user interaction unit 12 presents the results of the operation generated based on additional instructions from the user in a format that is easy for the user to understand. In this example, the amount of change in posture during transport is minimized, which shortens the transport time, but the calculation time is longer. The motion generation result 703 presents the result of generating motion based on the additional instructions in animation. Compared to the motion generation result 606 in Figure 9, the user can see that the amount of change in the posture of the robot arm tip is smaller in the motion generation result 703 in Figure 10.

[0051] Figure 11 is a hardware configuration diagram of each device in the equipment operation generation system 100. Each device of the equipment motion generation system 100 (equipment motion generation support device 10, motion generation processing device 20) is configured as a computer 900 having a CPU 901, RAM 902, ROM 903, HDD 904, communication I / F 905, input / output I / F 906, and media I / F 907. The communication interface 905 is connected to an external communication device 915. The input / output interface 906 is connected to the input / output device 916. The media interface 907 reads and writes data to the recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (also called an application or app) loaded into the RAM 902. This program can also be distributed via a communication line or by recording it on a recording medium 917 such as a CD-ROM.

[0052] In the embodiment described above, the equipment operation generation support device 10 extracts operation generation parameters from user instructions entered in a natural language dialogue format. Therefore, the extraction item DB 17 of the equipment operation generation support device 10 prepares the start and end point coordinates to be extracted as individual prompts as extraction item information. As a result, the equipment operation generation support device 10 can extract the parameters for generating operations to be input to the operation generation processing device 20 simply by executing prompts according to the extracted item DB 17, without having to perform semantic interpretation of spatial information (three-dimensional information) included in the user's instructions. Therefore, the user can generate equipment operations in a natural language dialogue format without having to learn a programming language. Furthermore, since the equipment operation generation support device 10 accepts user instructions via voice input, there is no need to learn to input text via a keyboard.

[0053] Furthermore, the present invention is not limited to the embodiments described above, and it goes without saying that various other applications and modifications can be taken as long as they do not depart from the gist of the invention as described in the claims. For example, the embodiments described above describe the configuration of the equipment operation generation system 100 in detail and specifically in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those that include all the components described. Also, it is possible to replace a part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, replace, or delete other components for a part of the configuration of each embodiment.

[0054] Furthermore, some or all of the above configurations, functions, and processing units may be implemented in hardware, for example, by designing them as integrated circuits. Broadly defined processor devices such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits) may be used as hardware. Furthermore, each component of the equipment operation generation system 100 according to the above-described embodiment may be implemented on any hardware, as long as the respective hardware can send and receive information from each other via a network. Also, the processing performed by a certain processing unit may be implemented by a single piece of hardware, or by distributed processing by multiple pieces of hardware. [Explanation of Symbols]

[0055] 10 Equipment operation generation support device 11 Model Control Unit 12. User Interaction Section (User Input Section) 13 Motion generation instruction section 16 Processing Instruction DB 17 Extraction item DB (storage part) 20 Operation Generation Processing Unit 100 Equipment Operation Generation System 111 User interaction state 112 Extraction Processing Status 113 Action generation processing state 114 Processing status of missing information request

Claims

1. The equipment motion generation support device has a user input unit and a motion generation instruction unit. The user input unit receives user instructions written in natural language in order to generate the operation of the equipment. The aforementioned motion generation instruction unit, For each extraction item of the parameters for generating the operation, an extraction prompt for extracting the extraction item from the user instruction content is read from the storage unit, and the user instruction content and the extraction prompt are input to the large-scale language model to obtain the parameters for generating the operation included in the user instruction content. This feature is characterized by outputting a generation instruction to generate operation parameters that indicate the operation of the equipment, based on the acquired operation generation parameters. Equipment operation generation support method.

2. The aforementioned storage unit contains information indicating the degree of necessity for each extraction item, showing whether or not extraction is essential. The operation generation instruction unit is characterized in that, if an extracted item for which the operation generation parameters included in the user instruction content could not be obtained is deemed essential, the unit prompts the user to input additional extracted items for which the parameters could not be obtained. The equipment operation generation support method according to claim 1.

3. The operation generation instruction unit is characterized in that, for extraction items from which the operation generation parameters included in the user instruction content could not be obtained, it prompts the user to input additional extraction items that could not be obtained, and if the input indicates that the extraction item is not essential, it omits the extraction of the extraction item that could not be obtained. The equipment operation generation support method according to claim 1.

4. The extraction prompt of the storage unit is defined for each type of equipment, The operation generation instruction unit is characterized by obtaining the extraction prompt corresponding to the type of equipment described in the user instruction content from the storage unit. The equipment operation generation support method according to claim 1.

5. The extraction prompt of the memory unit is characterized in that it includes at least one of the following: the role name when the large-scale language model is operated as the operation generation instruction unit, the name of the extraction item, and the unit of the extraction item. The equipment operation generation support method according to claim 1.

6. The extraction prompt of the storage unit is characterized in that it includes an instruction for extracting at least one of the following extraction items: the starting position and orientation of the equipment in operation, the ending position and orientation of the equipment in operation, and the operating speed of the equipment. The equipment operation generation support method according to claim 1.

7. The extraction prompt in the storage unit is characterized in that it includes a prompt for extracting transport quality, which indicates the state of the object being transported by the operation of the equipment. The equipment operation generation support method according to claim 1.

8. The user input unit displays a video in which the previous operation parameters, generated using the operation generation parameters obtained from the previous user instruction, are reflected in the operation of the equipment, and also accepts the current user instruction as additional input. The operation generation instruction unit is characterized in that it generates the current operation parameters by modifying the previous operation parameters based on the operation generation parameters obtained from the user instruction content of the current operation. The equipment operation generation support method according to claim 1.

9. The motion generation instruction unit is characterized by converting the acquired data format of the motion generation parameters into a data format that can be accepted as a generation instruction. The equipment operation generation support method according to claim 1.

10. A user input unit that accepts user instructions written in natural language to generate the operation of the equipment, For each extraction item of the parameters for generating the operation, an extraction prompt for extracting the extraction item from the user instruction content is read from the storage unit, and the user instruction content and the extraction prompt are input to the large-scale language model to obtain the parameters for generating the operation included in the user instruction content. It is characterized by having an operation generation instruction unit that outputs a generation instruction to generate operation parameters based on the acquired operation generation parameters. Equipment operation generation support device.

11. A system for generating equipment motion, comprising an equipment motion generation support device and a motion generation processing device as described in claim 10, The motion generation processing device is characterized by generating motion parameters that satisfy the motion generation parameters of the generation instruction created by the motion generation instruction unit, and returning the motion parameters to the equipment motion generation support device. Equipment motion generation system.

Images