Processing system, processing method, and recording medium

Abstract

This processing system includes: a first creation means for creating a trajectory of an action to be performed by a robot on an object transitioning from a first state to a second state; a second creation means for creating a prompt for asking a model capable of determining success or failure of a state about success or failure of the second state; and an execution means for causing the robot to perform an action to be realized in the second state according to a response to the prompt.

Description

Processing system, processing method, and recording medium

[0001] The present disclosure relates to a processing system, a processing method, and a recording medium.

[0002] Robots are used in various fields such as logistics. Patent Document 1 discloses a related technology relating to a robot system.

[0003] International Publication No. 2017 / 134735

[0004] In the technical field related to Patent Document 1, there is a demand for a technology that can improve convenience.

[0005] One of the objectives of each aspect of the present disclosure is to provide a processing system, a processing method, and a recording medium that can solve the above-mentioned problems.

[0006] According to one aspect of the present disclosure, a processing system includes a first creation means for creating a trajectory of a motion of an object to be operated by a robot as the object moves from a first state to a second state; a second creation means for creating a prompt for querying a model capable of determining the success or failure of a state as to whether the second state has been achieved; and an execution means for causing the robot to perform a motion to be realized in the second state in accordance with a response to the prompt.

[0007] According to another aspect of the present disclosure, a processing method creates a trajectory of a motion of an object to be acted upon by a robot as the object moves from a first state to a second state, creates a prompt to inquire about the success or failure of the second state from a model capable of determining the success or failure of the state, and causes the robot to act in the second state according to a response to the prompt.

[0008] According to another aspect of the present disclosure, the recording medium stores a program that causes a computer to execute the following steps: create a trajectory of a motion of an object to be acted upon by a robot as the object moves from a first state to a second state; create a prompt to inquire about the success or failure of the second state from a model capable of determining the success or failure of a state; and cause the robot to act in the second state according to a response to the prompt.

[0009] According to each aspect of the present disclosure, convenience can be improved.

[0010] FIG. 1 is a diagram illustrating an example of a configuration of a processing system according to some embodiments of the present disclosure; FIG. 2 is a diagram illustrating an example of a processing flow of a processing system according to some embodiments of the present disclosure; FIG. 3 is a diagram illustrating an example of a configuration of a processing system according to some embodiments of the present disclosure; FIG. 4 is a diagram illustrating an example of a processing flow of a processing system according to some embodiments of the present disclosure; FIG. 5 is a diagram illustrating an example of a configuration of a processing system according to some embodiments of the present disclosure; FIG. 6 is a diagram illustrating an example of a processing flow of a processing system according to some embodiments of the present disclosure; and FIG. 7 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment.

[0011] Hereinafter, an embodiment will be described in detail with reference to the drawings. <Embodiment> A processing system 1 according to an embodiment of the present disclosure is a system that estimates how to control a control object to move from a first state to a second state, which is a target state. For example, the processing system 1 calculates a target trajectory of a robot for moving the robot and an object (e.g., luggage, merchandise, etc.) from a certain location (i.e., the first state) to a target location (i.e., the second state). Below, the processing system 1 will be described in more detail.

[0012] 1 is a diagram illustrating an example of a configuration of a processing system 1 according to some embodiments of the present disclosure. As shown in FIG. 1, the processing system 1 includes a motion planning unit 10, an evaluation unit 20, a target trajectory selection unit 30, and a robot 40.

[0013] The motion planning unit 10 receives the initial state and target values ​​of the robot 40. The initial state is an initial parameter representing a state obtained from a sensor, a camera, or the like. For example, when the robot 40 moves an object from a certain location (i.e., a first state) to a target location (i.e., a second state), the target values ​​are, for example, coordinates indicating the target location and the posture of the object. The motion planning unit 10 generates a target trajectory candidate based on the received initial state and target values ​​of the robot 40. Examples of the target trajectory candidate include the position and posture of the object, the angle of the robot arm of the robot 40, and the angle of each joint of the robot 40.

[0014] As shown in FIG. 1, the evaluation unit 20 includes a state prediction unit 201, a prompt generation unit 202, an image generation unit 203, a multimodal LLM (Large Language Models) 204, and an evaluation value calculation unit 205.

[0015] The state prediction unit 201 generates time-series data of latent variables from the target trajectory candidates. The latent variables are variables that include at least feature quantities that enable the image generation unit 203 to reconstruct an image showing the state of the target object or the state of the robot 40.

[0016] The prompt generation unit 202 generates a question to the multimodal LLM 204. For example, the prompt generation unit 202 has prompt rules. The prompt generation unit 202 generates a question to the multimodal LLM 204 based on the prompt rules. The prompt rules are rules set in advance by a person, such as "Have you achieved your goal?" In this example, the actual content of the question changes depending on the goal.

[0017] Furthermore, for example, the prompt generation unit 202 includes an LLM. In this case, the LLM generates questions to be posed to a VLM (Vision-Language Model). For example, the LLM generates a question such as, "Please list the points that need to be considered in order to achieve the goal." The prompt itself at this time is determined in advance based on rules, such as, "Have you encountered any obstacles?", "Do you have A?" (when performing an action to have A), or "Do you have B?" (when B needs to be achieved).

[0018] The prompt generator 202 may ask the multimodal LLM 204 whether the final goal has been achieved, whether the prompt generator 202 has prompt rules or an LLM. The prompt generator 202 may ask the multimodal LLM 204 about multiple subgoals that must be cleared to achieve the final goal, whether the prompt generator 202 has prompt rules or an LLM.

[0019] Furthermore, the prompt generation unit 202 may cause the multimodal LLM 204 to calculate a reward whether it has prompt rules or an LLM. Furthermore, the prompt generation unit 202 may cause the multimodal LLM 204 to return an evaluation result whether it has prompt rules or an LLM.

[0020] The image generation unit 203 generates, from the time-series data of the latent variables, time-series images that indicate the transition of the state of the object or the robot 40. For example, the image generation unit 203 can realize the generation of time-series images that indicate the transition of the state of the object or the robot 40 from the time-series data of the latent variables by using neural network technology.

[0021] The multimodal LLM 204 receives a question from the prompt generator 202. The multimodal LLM 204 also receives a time-series image from the image generator 203. The multimodal LLM 204 outputs an answer to the question regarding the time-series image.

[0022] For example, the multimodal LLM 204 calculates a reward as an answer to a question about the time-series images. For example, when the multimodal LLM 204 is asked, "Was the target task accomplished?" about the time-series images, the multimodal LLM 204 answers, "Yes" or "No." For example, when the overall goal is divided into goal A and goal B and the prompt generation unit 202 asks questions about goal A and goal B, the multimodal LLM 204 answers, "Yes" or "No," about the time-series images, when the multimodal LLM 204 is asked, "Was the task for goal A accomplished?" about the time-series images. For example, when the multimodal LLM 204 is asked, "Was the task for goal B accomplished?" about the time-series images, the multimodal LLM 204 answers, "Yes" or "No."

[0023] Note that the Multimodal LLM 204 is not limited to an LLM as long as it returns an answer (or response, output) to a question (or inquiry, input).

[0024] The evaluation value calculation unit 205 calculates an evaluation value based on the answer output by the multimodal LLM 204. For example, the evaluation value calculation unit 205 sets the calculated reward as the evaluation value. Furthermore, for example, when asked, "Was the target task accomplished?", if the multimodal LLM 204 outputs "Yes," the evaluation value calculation unit 205 sets the evaluation value to, for example, 1, indicating that the target task has been accomplished. Furthermore, for example, when asked, "Was the target task accomplished?", if the multimodal LLM 204 outputs "No," the evaluation value calculation unit 205 sets the evaluation value to, for example, 0, indicating that the target task has not been accomplished. The evaluation value calculation unit 205 then outputs the evaluation value to the target trajectory selection unit 30.

[0025] The target trajectory selection unit 30 identifies the highest evaluation value among the evaluation values ​​output by the evaluation value calculation unit 205. Then, the target trajectory selection unit 30 determines whether the highest evaluation value is equal to or greater than a threshold value.

[0026] If the target trajectory selection unit 30 determines that the highest evaluation value is equal to or greater than the threshold value, it identifies the target trajectory corresponding to the highest evaluation value from among the target trajectory candidates and outputs the identified target trajectory to the robot 40.

[0027] The robot 40 moves an object (e.g., luggage, merchandise, etc.) from a certain location (i.e., a first state) to a target location (i.e., a second state) according to the target trajectory output by the target trajectory selection unit 30. The robot 40 may be physically realized in the real world. Alternatively, the robot 40 may be virtually realized in a simulation.

[0028] The above-described processing performed by the processing system 1 according to the embodiment of the present disclosure is merely an example, and the processing system 1 is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0029] (Processing Performed by Processing System) Fig. 2 is a diagram showing an example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Next, processing performed by the processing system 1 will be described with reference to Fig. 2 .

[0030] The motion planning unit 10 receives the initial state and target values ​​of the robot 40. The motion planning unit 10 generates a target trajectory candidate based on the received initial state and target values ​​of the robot 40 (step S1).

[0031] The prompt generation unit 202 generates questions for the multimodal LLM 204 (step S2). The image generation unit 203 generates time-series images showing the transition of the state of the object and the robot 40 from the time-series data of the latent variables (step S3).

[0032] The multimodal LLM 204 receives a question from the prompt generation unit 202. The multimodal LLM 204 also receives time-series images from the image generation unit 203. The multimodal LLM 204 outputs an answer to the question regarding the time-series images (step S4).

[0033] For example, the multimodal LLM 204 calculates a reward as an answer to a question about the time-series images. For example, when the multimodal LLM 204 is asked, "Was the target task accomplished?" about the time-series images, the multimodal LLM 204 answers, "Yes" or "No." For example, when the overall goal is divided into goal A and goal B and the prompt generation unit 202 asks questions about goal A and goal B, the multimodal LLM 204 answers, "Yes" or "No," about the time-series images, when the multimodal LLM 204 is asked, "Was the task for goal A accomplished?" about the time-series images. For example, when the multimodal LLM 204 is asked, "Was the task for goal B accomplished?" about the time-series images, the multimodal LLM 204 answers, "Yes" or "No."

[0034] The evaluation value calculation unit 205 calculates an evaluation value based on the response output from the multimodal LLM 204 (step S5), and outputs the evaluation value to the target trajectory selection unit 30.

[0035] The target trajectory selection unit 30 identifies the highest evaluation value among the evaluation values ​​output by the evaluation value calculation unit 205 (step S6), and then determines whether the highest evaluation value is equal to or greater than a threshold value (step S7).

[0036] If the target trajectory selection unit 30 determines that the highest evaluation value is less than the threshold value (NO in step S7), the process returns to step S1. If the target trajectory selection unit 30 determines that the highest evaluation value is equal to or greater than the threshold value (YES in step S7), the target trajectory selection unit 30 identifies a target trajectory corresponding to the highest evaluation value from among the target trajectory candidates (step S8). Then, the target trajectory selection unit 30 outputs the identified target trajectory to the robot 40.

[0037] The robot 40 moves an object (e.g., luggage, goods, etc.) from a certain location (i.e., the first state) to a target location (i.e., the second state) according to the target trajectory output by the target trajectory selection unit 30 (step S9).

[0038] (Advantages) The processing system 1 according to an embodiment of the present disclosure has been described above. In the processing system 1, the motion planning unit 10 (an example of a first creation means) creates a trajectory of a motion of an object to be operated by a robot when the object moves from a first state to a second state. The prompt generation unit 202 (an example of a second creation means) creates a prompt that queries a model capable of determining whether a state is successful or not regarding the success or failure of the second state. The target trajectory selection unit 30 (an example of an execution means) performs a motion of the robot to be realized in the second state in accordance with a response to the prompt.

[0039] The processing system 1 can easily calculate the evaluation value in response to a prompt. Then, the robot can be operated in a state determined based on the evaluation value. In other words, the processing system 1 can improve convenience.

[0040] <First Modification of the Embodiment> Similar to the processing system 1 according to the embodiment of the present disclosure, the processing system 1 according to the first modification of the embodiment of the present disclosure is a system that estimates how to control a control target to change from a first state to a second state, which is a target state. However, in the processing system 1 according to the first modification of the embodiment of the present disclosure, in addition to the multimodal LLM 204 calculating an evaluation value, the success / failure classifier 206 calculates an evaluation value and a reliability. The following will mainly describe the differences from the processing system 1 according to the embodiment of the present disclosure.

[0041] 3 is a diagram illustrating an example of a configuration of a processing system 1 according to some embodiments of the present disclosure. As shown in FIG. 3, the processing system 1 includes a motion planning unit 10, an evaluation unit 20, a target trajectory selection unit 30, and a robot 40.

[0042] 3, the evaluation unit 20 includes a state prediction unit 201, a prompt generation unit 202, an image generation unit 203, a multimodal LLM (Large Language Models) 204, and an evaluation value calculation unit 205. Furthermore, the evaluation unit 20 further includes a success / failure classifier 206, as shown in FIG.

[0043] The success / failure classifier 206 calculates an evaluation value and a reliability from the time-series data of the latent variables. The success / failure classifier 206 can be realized, for example, by using a model using a neural network (using a model that has been trained so that the evaluation value and the reliability can be calculated simultaneously). The reliability indicates the accuracy of the prediction using an estimated probability or score ranging from 0 to 1. A higher estimated probability or score indicates a higher reliability. The evaluation value calculation unit 205 calculates an evaluation value based on the answer output by the multimodal LLM 204 and the evaluation value and reliability calculated by the success / failure classifier 206. For example, the evaluation value calculation unit 205 determines weightings for the answer output by the multimodal LLM 204 (i.e., the evaluation value) and the evaluation value calculated by the success / failure classifier 206, depending on the reliability calculated by the success / failure classifier 206.

[0044] Specifically, for example, as the reliability calculated by the success / failure classifier 206 increases, the evaluation value calculation unit 205 increases the weighting coefficient of the evaluation value calculated by the success / failure classifier 206 compared to the weighting coefficient of the answer output by the multimodal LLM 204. Furthermore, specifically, for example, the evaluation value calculation unit 205 may adopt the evaluation value calculated by the success / failure classifier 206 when the reliability calculated by the success / failure classifier 206 is equal to or greater than a threshold, and may adopt the answer output by the multimodal LLM 204 when the reliability calculated by the success / failure classifier 206 is less than the threshold. Then, the evaluation value calculation unit 205 outputs the evaluation value to the target trajectory selection unit 30.

[0045] The above-described processing performed by the processing system 1 according to the first modified example of the embodiment of the present disclosure is merely an example, and the processing system 1 is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0046] (Processing Performed by Processing System) Fig. 4 is a diagram showing an example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Next, processing performed by the processing system 1 will be described with reference to Fig. 4 .

[0047] The processing system 1 performs the processes of steps S1 to S4. The success / failure classifier 206 calculates an evaluation value and a reliability from the time-series data of the latent variables (step S11). The evaluation value calculation unit 205 calculates an evaluation value based on the answer output by the multimodal LLM 204 and the evaluation value and reliability calculated by the success / failure classifier 206 (step S12). For example, the evaluation value calculation unit 205 determines weightings for the answer output by the multimodal LLM 204 (i.e., the evaluation value) and the evaluation value calculated by the success / failure classifier 206, respectively, according to the reliability calculated by the success / failure classifier 206.

[0048] Specifically, for example, as the reliability calculated by the success / failure classifier 206 increases, the evaluation value calculation unit 205 increases the weighting coefficient of the evaluation value calculated by the success / failure classifier 206 compared to the weighting coefficient of the answer output by the multimodal LLM 204. Furthermore, specifically, for example, the evaluation value calculation unit 205 may adopt the evaluation value calculated by the success / failure classifier 206 when the reliability calculated by the success / failure classifier 206 is equal to or greater than a threshold, and may adopt the answer output by the multimodal LLM 204 when the reliability calculated by the success / failure classifier 206 is less than the threshold. Then, the evaluation value calculation unit 205 outputs the evaluation value to the target trajectory selection unit 30. Then, the processing system 1 performs the processes of steps S6 to S9.

[0049] (Advantages) The processing system 1 according to the first modified example of an embodiment of the present disclosure has been described above. In the processing system 1, the success / failure classifier 206 calculates an evaluation value and a reliability from time-series data of latent variables. The evaluation value calculation unit 205 calculates an evaluation value based on the answer output by the multimodal LLM 204 and the evaluation value and reliability calculated by the success / failure classifier 206.

[0050] In the processing system 1 of the first modified example of an embodiment of the present disclosure, the evaluation value calculation unit 205 can determine weightings for the answer (i.e., the evaluation value) output by the multimodal LLM 204 and the evaluation value calculated by the success / failure classifier 206, depending on the reliability calculated by the success / failure classifier 206. As a result, the processing system 1 of the first modified example of an embodiment of the present disclosure can calculate the evaluation value more accurately than the processing system 1 of the embodiment of the present disclosure. In other words, the processing system 1 of the first modified example of an embodiment of the present disclosure can specify a target trajectory more accurately. In other words, the processing system 1 can improve convenience.

[0051] 5 is a diagram illustrating an example of a configuration of a processing system 1 according to some embodiments of the present disclosure. As illustrated in FIG. 5, the processing system 1 includes a first generating unit 301, a second generating unit 302, and an executing unit 303.

[0052] The first creating means 301 creates a trajectory of a motion of an object to be moved by the robot when the object moves from a first state to a second state. The second creating means 302 creates a prompt to inquire about the success or failure of the second state from a model capable of determining the success or failure of a state. The execution means 303 causes the robot to perform a motion to be realized in the second state according to a response to the prompt.

[0053] The first creation means 301 can be realized, for example, by using the functions of the motion planning unit 10 illustrated in Figures 1 and 3. The second creation means 302 can be realized, for example, by using the functions of the prompt generation unit 202 illustrated in Figures 1 and 3. The execution means 303 can be realized, for example, by using the functions of the target trajectory selection unit 30 illustrated in Figures 1 and 3.

[0054] 6 is a diagram illustrating an example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Next, processing of the processing system 1 according to some embodiments of the present disclosure will be described with reference to FIG.

[0055] The first creating means 301 creates a trajectory of a motion of an object to be moved by the robot when the object moves from a first state to a second state (step S101). The second creating means 302 creates a prompt to inquire about the success or failure of the second state from a model capable of determining the success or failure of a state (step S102). The executing means 303 performs a motion to be realized in the second state on the robot according to a response to the prompt (step S103).

[0056] The processing system 1 according to some embodiments of the present disclosure has been described above. The processing system 1 can improve convenience.

[0057] The order of the processes in the embodiments of the present disclosure may be changed as long as the processes are performed appropriately.

[0058] Although the embodiments of the present disclosure have been described, the processing system 1, the motion planning unit 10, the evaluation unit 20, the target trajectory selection unit 30, the robot 40, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the above processing is performed by the computer reading and executing this program. Specific examples of computers are shown below.

[0059] FIG. 7 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 7 , the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9. For example, the processing system 1, the motion planning unit 10, the evaluation unit 20, the target trajectory selection unit 30, the robot 40, and other control devices are implemented in the computer 5. The operations of the above-mentioned processing units are stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above-mentioned processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to the above-mentioned storage units in accordance with the program.

[0060] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. Furthermore, if the program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-described processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

[0061] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0062] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0063] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0064] (Supplementary Note 1) A processing system comprising: a first creation means for creating a trajectory of a motion of an object to be acted upon by a robot when the object moves from a first state to a second state; a second creation means for creating a prompt for inquiring about the success or failure of the second state to a model capable of determining the success or failure of a state; and an execution means for causing the robot to act in the second state in accordance with a response to the prompt.

[0065] (Supplementary Note 2) The processing system according to Supplementary Note 1, wherein the second creating means creates the prompt including a plurality of target values ​​and a plurality of tasks.

[0066] (Supplementary Note 3) The processing system according to Supplementary Note 1 or Supplementary Note 2, comprising: the model that responds to the prompt created by the second creation means; and an evaluation means that calculates an evaluation value and reliability from the time-series data of the state.

[0067] (Supplementary Note 4) The processing system according to Supplementary Note 3, further comprising an evaluation value calculation means for determining a weighting between the evaluation value calculated by the model and the evaluation value calculated by the evaluation means based on the reliability, and calculating the evaluation value based on the determined weighting.

[0068] (Supplementary Note 5) The processing system according to any one of Supplementary Notes 1 to 4, wherein the model is a Multimodal Large Language Model (LLM) or a Vision-Language Model (VLM).

[0069] (Supplementary Note 6) A processing system according to Supplementary Note 1, comprising: a prediction means for making predictions capable of reconstructing an image; and a generation means for generating time series images corresponding to time series data of latent variables obtained based on the prediction, from the time series data of the latent variables, wherein the model uses the time series images generated by the generation means as input.

[0070] (Supplementary Note 7) A processing method comprising: creating a trajectory of a motion of an object to be acted upon by a robot as the object moves from a first state to a second state; creating a prompt to inquire about the success or failure of the second state from a model capable of determining the success or failure of a state; and having the robot act in the second state according to a response to the prompt.

[0071] (Supplementary Note 8) The processing method according to Supplementary Note 7, further comprising creating the prompt comprising a plurality of goal values ​​and a plurality of tasks.

[0072] (Supplementary Note 9) The processing method according to Supplementary Note 7 or Supplementary Note 8, wherein the model responds to the generated prompt, and the evaluation means calculates an evaluation value and a reliability from the time-series data of the state.

[0073] (Supplementary Note 10) The processing method according to Supplementary Note 9, comprising: determining a weighting of the evaluation value calculated by the model and the evaluation value calculated by the evaluation means based on the reliability; and calculating the evaluation value based on the determined weighting.

[0074] (Supplementary Note 11) The processing method according to any one of Supplementary Note 7 to Supplementary Note 10, wherein the model is a Multimodal Large Language Model (LLM) or a Vision-Language Model (VLM).

[0075] (Supplementary Note 12) The processing method according to any one of Supplementary Note 7 to Supplementary Note 11, comprising: making a prediction capable of reconstructing an image; generating a time series of images corresponding to time series data from time series data of latent variables obtained based on the prediction; and using the generated time series of images as input to the model.

[0076] (Supplementary Note 13) A recording medium storing a program that causes a computer to execute the following: creating a trajectory of a motion of an object to be acted upon by a robot when the object moves from a first state to a second state; creating a prompt that queries a model capable of determining the success or failure of a state as to whether the second state has been achieved; and causing the robot to act in the second state in accordance with a response to the prompt.

[0077] (Supplementary Note 14) The recording medium according to Supplementary Note 13, storing a program that causes a computer to execute the steps of: creating the prompt including a plurality of target values ​​and a plurality of tasks.

[0078] (Appendix 15) A recording medium according to appendix 13 or appendix 14, storing a program that causes a computer to execute the following steps: the model responds to the created prompt; and an evaluation means calculates an evaluation value and reliability from the time-series data of the state.

[0079] (Supplementary Note 16) A recording medium according to Supplementary Note 15, storing a program for causing a computer to execute the following: determining a weighting of the evaluation value calculated by the model and the evaluation value calculated by the evaluation means based on the reliability; and calculating an evaluation value based on the determined weighting.

[0080] (Supplementary Note 17) The recording medium according to any one of Supplementary Note 13 to Supplementary Note 16, wherein the model is a Multimodal Large Language Model (LLM) or a Vision-Language Model (VLM).

[0081] (Supplementary Note 18) A recording medium according to any one of Supplementary Note 13 to Supplementary Note 17, which stores a program that causes a computer to execute the following: making a prediction capable of reconstructing an image; and generating a time series of images corresponding to time series data from time series data of latent variables obtained based on the prediction; and the model uses the generated time series of images as input.

[0082] According to each aspect of the present disclosure, convenience can be improved.

[0083] DESCRIPTION OF SYMBOLS 1... Processing system 5... Computer 6... CPU 7... Main memory 8... Storage 9... Interface 10... Motion planning unit 20... Evaluation unit 30... Target trajectory selection unit 40... Robot 201... State prediction unit 202... Prompt generation unit 203... Image generation unit 204... Multimodal LLM (Large language Models) 205... Evaluation value calculation unit 206... Success / failure classifier 301... First creation means 302... Second creation means 303... Execution means

Claims

1. A processing system comprising: a first creation means for creating a trajectory of a motion of an object to be acted upon by a robot as the object moves from a first state to a second state; a second creation means for creating a prompt to inquire about the success or failure of the second state from a model capable of determining the success or failure of a state; and an execution means for causing the robot to act in the second state in accordance with a response to the prompt.

2. The processing system according to claim 1, wherein said second creation means creates said prompt consisting of a plurality of target values ​​and a plurality of tasks.

3. The processing system according to claim 1, further comprising: the model that responds to the prompt created by the second creation means; and an evaluation means that calculates an evaluation value and reliability from the time-series data of the state.

4. The processing system according to claim 3, further comprising an evaluation value calculation means for determining a weighting between the evaluation value calculated by the model and the evaluation value calculated by the evaluation means based on the reliability, and calculating the evaluation value based on the determined weighting.

5. The processing system of claim 1, wherein the model is a Multimodal Large Language Model (LLM) or a Vision-Language Model (VLM).

6. A processing system as described in claim 1, comprising: a prediction means for making predictions capable of reconstructing an image; and a generation means for generating a time series of images corresponding to time series data of latent variables obtained based on the prediction, from the time series data of the latent variables, wherein the model uses the time series of images generated by the generation means as input.

7. A processing method comprising: creating a trajectory of a motion of an object to be acted upon by a robot as it moves from a first state to a second state; creating a prompt to inquire about the success or failure of the second state from a model capable of determining the success or failure of a state; and having the robot act in the second state in accordance with the response to the prompt.

8. A recording medium storing a program that causes a computer to execute the following steps: creating a trajectory of a movement of an object to be moved by a robot when the object moves from a first state to a second state; creating a prompt that queries a model capable of determining the success or failure of a state as to whether the second state has been reached; and having the robot perform the movement to be realized in the second state in accordance with the response to the prompt.

Images