Information processing system and information processing method

Abstract

To provide an information processing system which allows a user to intuitively easily understand a learning state of a world model.SOLUTION: An information processing system comprises: a designation unit which designates a learned world model, environmental data about environment in which an agent operates, and a learned strategy model which specifies operation of the agent; a strategy execution unit which sequentially executes strategy in the environment, on the basis of the environmental data and the strategy model; an observation image processing unit which captures the environment by a virtual camera for each execution of the strategy, and generates a plurality of observation images; a latent state processing unit which compresses each of the plurality of observation images to derive each of a plurality of latent states, on the basis of the world model; and a display unit which displays the plurality of latent states.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system and an information processing method. [Background technology]

[0002] In recent years, in the field of visual-based reinforcement learning, research on world models has progressed, which construct a model of the environment so that an agent can predict the outcome of its actions. In other words, a world model is a reinforcement learning method that learns a predictive model of the state transition of the environment. Regarding world models, it is known that an agent learns a world model based on image input and actions (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Google AI Blog, The latest from Google Research, “Introducing PlaNet: A Deep Planning Network for Reinforcement Learning”,<URL:https: / / webbigdata.jp / ai / post-2867> Summary of the Invention [Problem to be solved by the invention]

[0004] In the research and development of world models, engineers are required to confirm how an agent's world model is being learned. However, because world models are expressed as vector data, it is difficult for engineers to intuitively understand them. It is also difficult to interactively check the state of the restored image or world model in response to changes in time, changes in observation due to actions, or changes in the observation position. Furthermore, until now, there has been no means of verifying whether the learning results of the world model match the engineers' expectations, i.e., whether the learning results are satisfactory.

[0005] The present disclosure provides an information processing system and an information processing method that make it easier to intuitively understand the learning state of a world model. [Means for solving the problem]

[0006] One aspect of the present disclosure is an information processing system comprising: a designation unit that designates a trained world model, environmental data regarding an environment in which an agent operates, and a trained policy model that defines the behavior of the agent; a policy execution unit that sequentially executes policies in the environment based on the environmental data and the policy model; an observation image processing unit that captures images of the environment with a virtual camera for each execution of the policy and generates a plurality of observation images; a latent state processing unit that compresses each of the plurality of observation images based on the world model to derive each of a plurality of latent states; and a display unit that displays the plurality of latent states.

[0007] One aspect of the present disclosure is an information processing method comprising the steps of: specifying a trained world model, environmental data regarding an environment in which an agent operates, and a trained policy model that defines the behavior of the agent; sequentially executing policies in the environment based on the environmental data and the policy model; capturing images of the environment with a virtual camera for each execution of the policy to generate a plurality of observation images; compressing each of the plurality of observation images based on the world model to derive each of a plurality of latent states; and displaying the plurality of latent states on a display unit. [Effects of the Invention]

[0008] According to the present disclosure, the learning state of the world model can be easily understood intuitively. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure. [Figure 2] Block diagram showing an example of the GUI configuration [Figure 3] Block diagram showing an example of the simulator configuration [Figure 4] A diagram showing an example of a list screen [Figure 5] A diagram showing an example of a model setting screen [Figure 6] A diagram showing an example of a model setting screen with a path name displayed. [Figure 7] A diagram showing an example of a camera setting screen. [Figure 8] FIG. 10 is a diagram showing an example of a camera setting screen when multiple virtual cameras are provided; [Figure 9] A diagram showing an example of latent state mapping [Figure 10] A sequence diagram showing an example of the operation of an information processing system DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0011] For example, the term "unit" or "device" in the embodiments is not limited to a physical configuration mechanically realized by hardware, but also includes a configuration whose functions are realized by software such as a program. Furthermore, the functions of one configuration may be realized by two or more physical configurations, or the functions of two or more configurations may be realized by, for example, one physical configuration.

[0012] (Terminology explanation) This embodiment deals with reinforcement learning of a world model, etc. First, terms related to reinforcement learning and a world model will be explained.

[0013] "Reinforcement learning" is a machine learning technique in which an agent learns a task by repeatedly interacting with a dynamic environment through trial and error. The agent receives a reward that evaluates the quality of its current state. A task might be, for example, for a robotic device acting as an agent to grasp an object. When the task is accomplished, a reward is earned. Reinforcement learning learns a policy that will obtain the most reward through a series of actions.

[0014] An "agent" is a subject that interacts with the environment and learns a policy. Specific examples of agents include artificial intelligence (AI), robotic devices, and controllers.

[0015] A "state" indicates the state of the environment. Specific examples of the state include the position of a robot device, the angle of the arm of a robot device (robot arm), etc. Therefore, the state is, for example, information about the object to be controlled.

[0016] An "action" is an action that an agent takes to affect the environment and change its state. Specific examples of actions include the movement of a robot device, the torque applied to a robot arm, etc.

[0017] A "reward" is an index used to evaluate the quality of an agent's state or behavior. In reinforcement learning, a task is learned by learning to maximize the reward value. As a specific example, the reward is the distance between the goal and the robot device, that is, the distance between the target position and the current position of the robot device. Note that the magnitude of the reward value is set according to the evaluation standard for quality, and therefore does not necessarily match the magnitude of the parameter used to calculate the reward. For example, if the robot device is trained to consider approaching the goal as a "good" outcome, the reward is calculated using an index (e.g., the reciprocal of the distance) that increases the reward value as the distance decreases. On the other hand, if the robot device is trained to consider moving away from the goal as a "good" outcome, the reward is calculated using an index (e.g., the distance value itself) that increases the reward value as the distance increases.

[0018] A "Policy" is a function that returns what action should be taken in a given state (e.g., the state of the environment or the state of an agent). A concrete example of a policy is a strategy for the operation of a robotic device.

[0019] "Latent states" (latent variables) are used, for example, in world models. A latent state is information (e.g., vector information) that compresses environmental observation information (e.g., observed images) and latently represents the state of the environment.

[0020] "Latent space" is used, for example, in world models. Latent space is a vector space in which latent states are represented. Therefore, how the agent perceives observed information, that is, the learning state of the world model, can be understood by how the latent states are positioned in the latent space.

[0021] "Restored images" are used, for example, in world models. A restored image is an image in which observed information (e.g., an observed image) is restored based on a compressed latent state. Therefore, the restored image can be used to understand whether the agent correctly recognizes the observed image, that is, the learning state of the world model.

[0022] (Configuration of information processing system) Next, the configuration of the information processing system will be described.

[0023] 1 is a block diagram showing an example of the configuration of an information processing system 5 according to an embodiment of the present disclosure. The information processing system 5 includes a GUI (Graphical User Interface) unit 100 and a simulator unit 200.

[0024] The GUI unit 100 and the simulator unit 200 are communicatively connected via a communication network or by direct communication. The GUI unit 100 is operated by a user and provides display information to the user. The simulator unit 200 is a physics calculation simulator used in reinforcement learning experiments. The simulator unit 200 uses an API (Application Programming Interface) or the like to acquire information about the environment (e.g., images, robot positions and joints, time, or rewards), and inputs actions to act on the environment.

[0025] The GUI unit 100 is, for example, a PC (Personal Computer) or other information processing device. The simulator unit 200 is, for example, a server device or other information processing device. Therefore, the GUI unit 100 and the simulator unit 200 may be in a client-server relationship.

[0026] 2 is a block diagram showing an example of the configuration of the GUI unit 100. The GUI unit 100 includes a processor 110, a memory 120, a communication device 130, an operation device 140, and a display device 150.

[0027] The processor 110 may include an MPU (Micro Processing Unit), a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), or the like. The processor 110 may be configured with various integrated circuits (e.g., an LSI (Large Scale Integration), an FPGA (Field Programmable Gate Array)). The processor 110 realizes various functions by executing programs stored in the memory 120. The processor 110 comprehensively controls each section of the GUI section 100 and performs various processes.

[0028] The memory 120 includes a primary storage device (for example, a random access memory (RAM) or a read only memory (ROM)). The memory 120 may include a secondary storage device (for example, a hard disk drive (HDD) or a solid state drive (SSD)) or a tertiary storage device (for example, an optical disk or an SD card). The memory 120 may also be an external storage medium that is detachable from the GUI unit 100. The memory 120 stores various data, information, programs, etc.

[0029] The memory 120 may store, for example, an environment file, a trained world model, and a trained policy model. The environment file is a type of environment data that includes information about the environment in which the agent acts. The policy model is a learned model that learns a policy and is a model for the agent's behavior. One or more environment files, trained world models, or trained policy models may be prepared and stored in the memory 120.

[0030] The communication device 130 communicates various data or information, etc. The communication is performed according to a wired or wireless communication method by the communication device 130. The communication method may be a wide area network (WAN), a local area network (LAN), cellular communication for mobile phones (e.g., LTE, 5G), short-range communication (e.g., infrared communication or Bluetooth (registered trademark) communication), power line communication, etc.

[0031] The operation device 140 may include a mouse, a keyboard, a touchpad, a touch panel, a microphone, or other input devices, and accepts input of various data and information.

[0032] The display device 150 may include a liquid crystal display device, an organic EL device, or other display devices. The display device 150 displays various data and information. The display device 150 displays, for example, screens and images described below.

[0033] The processor 110 performs at least the processing necessary to present the learning state of the world model. As described above, the world model is a reinforcement learning method that learns a predictive model of the state transition of the environment. The state transition of the environment indicates what state will be transitioned to when what action is taken in what state. By taking the world model into account, the agent can potentially predict the outcome of a series of actions, reducing the trial and error of task learning.

[0034] The processor 110 has at least an information designation unit 111, a latent state processing unit 112, and a restored image processing unit 113 as functional components.

[0035] The information specification unit 111 specifies various pieces of information when determining the learning state of the world model. For example, the information specification unit 111 specifies information that defines the virtual environment (e.g., an environment file or a learned world model) or information that defines the behavior of an agent (e.g., a learned policy model). The information specification unit 111 also specifies information that defines the observation criteria of the virtual environment (e.g., information about a virtual camera that captures the virtual environment (camera information)). The camera information includes, for example, information about the position (camera coordinates) and orientation of the virtual camera relative to the virtual environment. The information specification unit 111 may manually specify and set various models and camera information based on, for example, information stored in the memory 120 or information about operations input to the operation device 140.

[0036] The latent state processing unit 112 maps the latent state in the latent space based on the trained world model and the observed image acquired from the simulator unit 200. The latent state processing unit 112 may calculate the latent state by compressing the observed image. The latent state may be vector data. The latent state processing unit 112 inputs the observed image into the world model and derives the latent state as the output of the world model. This latent state is, for example, multidimensional information. The latent state processing unit 112 may derive (e.g., calculate) a three-dimensional latent state by dimensionally compressing the obtained multidimensional latent state. Note that the latent state processing unit 112 may dimensionally compress into a two-dimensional latent state instead of a three-dimensional latent state. The latent state processing unit 112 maps the three-dimensional latent state on a latent state mapping screen showing the latent space and displays it on the display device 150. Note that the latent state processing unit 112 may map the two-dimensional latent state onto a two-dimensional plane instead of three-dimensionally.

[0037] Furthermore, if a user is able to handle a space of four or more dimensions after special training, the latent state processing unit 112 may perform dimensional compression on the latent states of four or more dimensions and display four-dimensional information on the latent state mapping screen. However, the number of dimensions that a typical user can intuitively handle is one to three, and in one dimension, all latent states are mapped to a single point, making it impossible to recognize multiple latent states. Therefore, it is desirable for the latent state processing unit 112 to handle two- or three-dimensional latent states. Furthermore, to accommodate the needs of a variety of users, the latent state processing unit 112 may be able to switch the number of dimensions.

[0038] Furthermore, the latent state represented by the latent state mapping changes depending on the method used for dimensionality reduction and the parameters to be retained after dimensionality reduction. Therefore, the latent state processing unit 112 may be able to change the dimensionality reduction method or the parameters to be retained. In this way, the GUI unit 100 can provide an interface that is easy to use for experienced users who know the parameters they want to prioritize in their evaluation.

[0039] The restored image processing unit 113 generates a restored image based on the trained world model and the latent state derived by the latent state processing unit 112. For example, the restored image processing unit 113 inputs the latent state into the world model and derives a restored image as an output of the world model. The restored image processing unit 113 may generate the restored image based on a multidimensional latent state or a dimensionally compressed two-dimensional or three-dimensional latent state. For example, the restored image processing unit 113 may generate the restored image by restoring the latent state. The restored image processing unit 113 displays the generated restored image on the display device 150.

[0040] Since the latent state is obtained by compressing the observed image, the restored image corresponds to the observed image. However, depending on the learning state of the world model, the observed image and the restored image may or may not match. The restored image processing unit 113 may determine the degree of correlation (e.g., the degree of match) between the observed image and the restored image. The restored image processing unit 113 can determine the degree of correlation, for example, by aggregating the differences in pixel values ​​between the observed image and the restored image. The restored image processing unit 113 may also determine the degree of correlation more precisely using known techniques used in image search, etc. If the degree of correlation meets a predetermined standard, the restored image processing unit 113 may determine that the learning state of the world model is good, and if the degree of correlation does not meet the predetermined standard, the restored image processing unit 113 may determine that the learning state of the world model is poor. Note that the restored image processing unit 113 may display information about the latent state mapping on the display device 150 without determining the learning state of the world model. The user may then determine the quality of the learning state of the world model from the state of the latent state mapping.

[0041] 3 is a block diagram showing an example of the configuration of the GUI unit 100. The simulator unit 200 includes a processor 210, a memory 220, and a communication device 230, for example.

[0042] The processor 210 may include an MPU, a CPU, a DSP, a GPU, or the like. The processor may be configured using various integrated circuits (e.g., an LSI or an FPGA). The processor 210 executes programs stored in the memory 220 to realize various functions. The processor 210 comprehensively controls each unit of the simulator unit 200 and performs various processes. The processor 210 of the simulator unit 200 may have higher performance than the processor 110 of the GUI unit 100.

[0043] The memory 220 includes a primary storage device (e.g., RAM or ROM). The memory 220 may include a secondary storage device (e.g., HDD or SSD) or a tertiary storage device (e.g., optical disk or SD card). The memory 220 may also be an external storage medium. The memory 220 stores various data, information, programs, etc.

[0044] The communication device 230 communicates various data, information, etc. The communication is performed according to a wired or wireless communication method by the communication device 230. The communication method may be a WAN, a LAN, cellular communication for mobile phones (e.g., LTE, 5G), short-range communication (e.g., infrared communication or Bluetooth (registered trademark) communication), power line communication, etc.

[0045] The processor 210 has at least an information setting unit 211, a measure execution unit 212, and an observed image processing unit 213 as functional components.

[0046] The information setting unit 211 sets various types of information. For example, it acquires and sets an environment file, camera information, and policy model from the GUI unit 100. The environment file, camera information, and policy model that are set are the same as those specified by the GUI unit 100. This setting information may be held in the memory 220.

[0047] The policy execution unit 212 executes a policy in accordance with the set, learned policy model. Specifically, the policy execution unit 212 inputs information indicating the state of the environment (e.g., a set environment file, camera information) into the policy model, and derives an action to be taken by the agent (e.g., control information for the robot device to grab a ball) as the output of the policy model. The state of the environment can change depending on the derived action of the agent.

[0048] The observation image processing unit 213 controls the virtual camera and acquires observation images from the virtual camera. The virtual camera captures an image of a dynamic environment according to the set camera information. The acquired observation images are transmitted to the GUI unit 100 by the communication device 230. The virtual camera may sequentially acquire observation images while a measure is being executed, or may acquire observation images before and after the measure is executed.

[0049] FIG. 4 is a diagram showing the list screen GL displayed on the display device 150. As shown in FIG.

[0050] The list screen GL shows the process up to obtaining a restored image from the environment defined by the environment file, and includes screens and images obtained at each timing in this process. Specifically, the list screen GL includes an environment screen G1, a model setting screen G2, a camera setting screen G3, an observation screen G4, a latent state mapping screen G5, and a restoration screen G6.

[0051] The environment screen G1 displays an environment (virtual environment) defined by an environment file. In the environment screen G1, the environment is displayed as, for example, a three-dimensional model. In the environment displayed in the environment screen G1, a robot device 10 as an agent, a ball 20, an obstacle 30, and the like are arranged. In this environment, for example, the robot device 10 is trying to grab the ball 20 while moving the robot arm 15, etc. Therefore, the task is, for example, to move the robot arm 15 closer to the ball 20.

[0052] The model setting screen G2 is a setting screen for setting information related to various models, etc. The model setting screen G2 supports the setting of, for example, an environment file related to the environment, a world model, and a policy model. The camera setting screen G3 is a screen for setting camera information related to the virtual camera.

[0053] The observation screen G4 is a screen that displays an observation image of the environment captured by the virtual camera. The observation image is obtained sequentially with each execution of a policy, and is therefore updated with each execution of the policy. Furthermore, since the agent operates with each execution of a policy, the state of the environment may change sequentially. The observation screen G4 in FIG. 4 displays an observation image in which an obstruction 30 has entered the imaging range of the virtual camera, and part of the robot device 10 is hidden behind the obstruction 30 and is difficult to see when viewed from the virtual camera.

[0054] The latent state mapping screen G5 is a screen on which latent states based on observed images are mapped onto two-dimensional or three-dimensional coordinates. On the latent state mapping screen G5, multiple latent states corresponding to multiple observed images are displayed as points and mapped. Since the latent state is a compressed version of the observed image, the latent state mapping screen G5 shows the relationship between multiple environmental states resulting from the execution of a policy by displaying multiple characteristic points of the environment shown in the observed image.

[0055] The restored screen G6 is a screen that displays a restored image based on the latent state. The restored screen G6 in Fig. 4 displays a restored image in which an obstruction 30 has entered the imaging range of the virtual camera, and a part of the robot device 10 is hidden behind the obstruction 30 and is difficult to see when viewed from the virtual camera. Note that in the example of Fig. 4, the observed image and the restored image do not match and are slightly different.

[0056] Note that the layout of each screen on the list screen GL shown in Fig. 4 is an example and is not limited to this. Furthermore, some of the screens included in the list screen GL shown in Fig. 4 may be hidden, or other screens not shown in Fig. 4 may be included and displayed on the list screen GL.

[0057] Fig. 5 is a diagram showing an example of the model setting screen G21. Fig. 6 is a diagram showing an example of the model setting screen G22 on which a path name is displayed. The model setting screen G21 is displayed on the display device 150. The model setting screens G21 and G22 are examples of the model setting screen G2.

[0058] 5, the information specification unit 111 specifies an environment file, a world model, and a policy model using a model setting screen G21 via the operation device 140. Each piece of specified information (environment file, world model, and policy model) may be input via the operation device 140, or one of multiple types stored in the memory 120 may be selected. The information specification unit 111 transmits (e.g., uploads) each piece of specified information to the simulator unit 200, and instructs the simulator unit 200 to set each piece of information.

[0059] In other words, the information specification unit 111 sets (reflects) the input execution environment in the simulator unit 200 by uploading an environment file to the simulator unit 200 via the communication device 130. The information specification unit 111 sets (reflects) in the simulator unit 200 a model (world model) used by an agent to predict environmental changes and perform image restoration by uploading a world model to the simulator unit 200 via the communication device 130. The information specification unit 111 sets (reflects) in the simulator unit 200 a model (policy model) used by an agent to act by uploading a policy model to the simulator unit 200 via the communication device 130. Note that, when the upload of the setting information is completed, the path names of the respective items may be displayed (see FIG. 6 ). Note that it is not essential for the information specification unit 111 to upload the world model and the policy model to the simulator unit 200 and set them, and processing based on the world model and the policy model may be completed on the GUI unit 100 side.

[0060] The information designation unit 111 may designate the environment file, the world model, and the policy model in accordance with predetermined designation criteria without operation by the operation device 140.

[0061] Fig. 7 is a diagram showing an example of a camera setting screen G31. Fig. 8 is a diagram showing an example of a camera setting screen G32 in the case where a plurality of virtual cameras are provided. The camera setting screens G31 and G32 are displayed on the display device 150. The camera setting screens G31 and G32 are examples of the camera setting screen G3.

[0062] The information designation unit 111 sets details of the camera information using the camera setting screen G3 via the operation device 140. The camera information is information about a virtual camera that captures an image of the environment. The camera information includes, for example, information about the position (camera coordinates) and orientation of the virtual camera relative to the environment. In this case, the information designation unit 111 may designate, via the operation device 140, the X coordinate, Y coordinate, and Z coordinate in the environment as the position where the virtual camera is to be placed. Furthermore, the information designation unit 111 may designate, via the operation device 140, values ​​of roll, pitch, and yaw as the orientation of the virtual camera. Furthermore, the information designation unit 111 may store default information of the camera information in the memory 120, for example, and designate this default information as the camera information, without performing an operation via the operation device 140.

[0063] Furthermore, the information designation unit 111 may set camera information based on an operation of the operation device 140 on the environment screen G1. For example, the information designation unit 111 may adjust the position and orientation of the virtual camera by dragging the virtual camera by operating a mouse on the environment screen G1.

[0064] Furthermore, when setting the camera information, a list screen GL may be displayed on the display device 150. In this case, when setting the camera information, the user can set the camera information to be installed in the environment (to be acquired by the agent) while viewing the environment screen G1 included in the list screen GL.

[0065] Furthermore, when the information designation unit 111 sets the camera information, the information designation unit 111 may interactively reflect the set camera information on the environment screen G1 and the observation screen G4. That is, the information designation unit 111 may update the displayed environment screen G1 and the observation screen G4 based on an image captured by the virtual camera according to the set position (camera coordinates) and orientation (camera angle). In this case, the user can intuitively grasp the position or orientation, etc., of the virtual camera to observe the environment while adjusting the designation of the camera information (for example, while moving the viewpoint). Note that the observation image of the observation screen G4 can be acquired via the communication device 130 by operating in cooperation with the simulator unit 200.

[0066] Furthermore, the number of virtual cameras capturing images of the environment may not be one, but multiple cameras may be provided. For example, the information designation unit 111 may set camera information for the second and subsequent virtual cameras (see FIG. 8) by accepting a press of the add camera button B1 (see FIG. 7) via the operation device 140. By having the multiple virtual cameras have different camera information, the environment can be observed from various viewpoints.

[0067] Next, the execution of the policy model will be described.

[0068] The policy model is a model that outputs the next action depending on the observed image. When the policy model is changed, the behavior of the agent changes according to the execution of the policy.

[0069] In the GUI unit 100, when the information specifying unit 111 receives an instruction to execute a measure via the operation device 140, it transmits the instruction to execute the measure to the simulator unit 200 via the communication device 130. For example, the instruction to execute a measure via the operation device 140 may be a press of the play button B2 (see FIG. 6) on the model setting screen G2 (G22). In the simulator unit 200, when the instruction to execute a measure is received, the measure executing unit 212 executes the measure according to the execution instruction and in accordance with the set measure model. In accordance with the execution of the measure, the agent operates in the environment.

[0070] Furthermore, in the GUI unit 100, when the information specifying unit 111 receives an instruction to stop the execution of a policy via the operation device 140, it transmits the instruction to stop the execution of a policy to the simulator unit 200 via the communication device 130. For example, the instruction to stop the execution of a policy via the operation device 140 may be, for example, pressing the stop button B3 (see FIG. 6) on the model setting screen G2 (G22). In the simulator unit 200, when the processor 210 receives the instruction to stop the execution of a policy, it stops the execution of the set policy in accordance with the stop instruction. In accordance with the stop of the execution of the policy model, the agent stops operating in the environment.

[0071] The GUI unit 100 and the simulator unit 200 cooperate with each other to interactively display an environment screen G1 and an observation screen G4 that reflect changes in the environment due to the execution of a measure. Specifically, in the simulator unit 200, the communication device 230 transmits the changed environment image and observation image to the GUI unit 100 when the measure model is executed. In the GUI unit 100, the measure execution unit 212 receives the environment image and the observation image from the simulator unit 200 via the communication device 130 and displays the environment screen G1 including the environment image and the observation screen G4 including the observation image via the display device 150. Note that changes in the environment image and the observation image may occur sequentially during the period in which the execution of the measure continues. Therefore, the GUI unit 100 may acquire and display the sequentially changing environment image and observation image from the simulator unit 200. Thus, the information processing system 5 can visualize and provide to the user the environment and observation image that change successively due to the execution of the measure. This allows the user to intuitively grasp changes in the environment due to the execution of the measure in real time.

[0072] Furthermore, the policy execution unit 212 may acquire operation information input through the operation device 140 of the GUI unit 100 via the communication device 230, and may move the virtual camera during execution of the policy model in accordance with this movement instruction. For example, the policy execution unit 212 may acquire operation information for dragging the camera icon CI of the virtual camera by operating a mouse on the environment screen G1 during execution of the policy model, and adjust the position and orientation of the virtual camera (for example, move the viewpoint) based on this operation information. In other words, the policy execution unit 212 may change the camera information of the virtual camera set before execution of the policy model during execution of the policy model. For example, the policy execution unit 212 may change the position (viewpoint) of the virtual camera by moving the position of the camera icon CI by a drag operation. For example, the policy execution unit 212 may change the orientation (line of sight) of the virtual camera by moving the position of the lens (not shown) of the camera icon CI. Note that the display example of the camera icon CI shown in FIG. 4 is just an example, and the camera icon CI may be displayed in other positions or in other display modes. The display device 150 can display information (e.g., an environmental image, an observed image, a latent state, and a restored image) based on an image captured by a virtual camera whose camera information has been changed. Thus, the user can interactively search for suitable camera information (e.g., the camera position and orientation) and adjust the image based on the image captured by the virtual camera while checking the behavior of the agent resulting from the execution of the policy model, the latent state, the restored image, and the like.

[0073] Fig. 9 is a diagram showing an example of the latent state mapping screen G5. In Fig. 9, the appearance of the latent state depicted in the latent state mapping screen G5 changes over time. Fig. 9 also illustrates an example in which a restored image is generated based on one latent state mapped on the latent state mapping screen G5.

[0074] In the simulator unit 200, the observed image processing unit 213 controls the virtual camera. The virtual camera captures an image of the environment and generates an observed image according to the set camera information. The agent's actions according to the execution of the policy affect the environment, and the environment may change. The virtual camera may sequentially capture images of the changing environment in time series and sequentially generate observed images. The communication device 230 transmits the sequentially generated observed images to the GUI unit 100.

[0075] In the GUI unit 100, the communication device 130 receives an observed image from the simulator unit 200. The latent state processing unit 112 compresses the received observed image to generate a latent state corresponding to the observed image. In this case, the latent state processing unit 112 may generate the latent state based on the observed image using a specified world model. For example, the latent state processing unit 112 uses the observed image as input to the world model and acquires a multidimensional latent state as output of the world model. The latent state processing unit 112 dimensionally compresses the acquired multidimensional latent state to generate a two-dimensional or three-dimensional latent state. In this case, the latent state processing unit 112 may generate the dimensionally compressed latent state according to a dimensionality compression method (e.g., PCA (principal component analysis)). The two-dimensional or three-dimensional latent state can be visually mapped onto a two-dimensional plane or three-dimensional space. The latent state processing unit 112 maps and draws, for example, a three-dimensional latent state on a latent state mapping screen G5 that shows the latent space.

[0076] The latent state processing unit 112 also sequentially receives observed images from the simulator unit 200. The latent state processing unit 112 generates a plurality of latent states based on the sequentially received observed images. The latent state processing unit 112 then renders the generated latent states on the latent state mapping screen G5. Since the observed images are obtained sequentially in time series, the number of rendered latent states also increases in time series. The latent state processing unit 112 maps each generated latent state on the latent state mapping screen G5 and displays it on the display device 150.

[0077] 9, first, a current latent state p1 corresponding to a current observed image is mapped and displayed on a latent state mapping screen G51. The current observed image here is, for example, an observed image of the environment before the execution of the policy model.

[0078] When the simulator unit 200 starts executing a policy, the latent state processing unit 112 draws, on the latent state mapping screen G52, each latent state corresponding to each observed image obtained at each step of policy execution. Thus, the number of mapped latent states increases as time passes during policy execution. On the latent state mapping screen G52, latent state p11 is the current latent state (the most recent latent state obtained during the execution of this policy). Latent state p12 is a latent state (a series of latent states) obtained sequentially by the execution of the policy, and indicates a latent state other than latent state p11.

[0079] The proximity of latent states on the latent state mapping screen G5 indicates the semantic similarity of the latent states. Therefore, by checking the display of the latent state mapping screen G5, the user can understand which observations the agent recognizes as being close to each other corresponding to which latent states. Changes in the environment due to the execution of policies are expected to occur gradually over time, and are unlikely to change significantly over time. Therefore, multiple latent states corresponding to multiple observation images that are close in time are expected to be rendered close to each other on the latent state mapping screen G5. Therefore, if multiple latent states obtained at close points in time are positioned far apart on the latent state mapping screen G5, the user can recognize that the learning of the world model that derives latent states from observation images may be insufficient or may have been learned in the wrong direction.

[0080] Note that, when the latent state is dimensionally compressed into three dimensions, the screen displayed on the latent state mapping screen G5 is information obtained by projecting the three-dimensional information onto a two-dimensional projection surface of the display device 150. In this case, if the user wishes to observe the projection results from another viewpoint, the latent state processing unit 112 may rotate the latent state mapping screen G5, which shows the three-dimensional space onto which the latent state is mapped, in any direction via the operation device 140 and display it on the display device 150. For example, the latent state processing unit 112 may accept a drag operation using a mouse and rotate the orientations of three mutually orthogonal axes that define the three-dimensional space in accordance with the drag operation. This allows the user to check the positional relationships of multiple latent states from various viewpoints and line-of-sight directions.

[0081] Furthermore, the latent state mapping screen G53 is a state that is later in the timeline than the latent state mapping screen G52, and the number of latent states is increased. On the latent state mapping screen G53, latent state p21 is the current latent state (the most recent latent state obtained during the execution of this policy). Latent state p22 is a latent state (a series of latent states) obtained sequentially by the execution of the policy, and indicates a latent state other than latent state p21.

[0082] Here, the restored image processing unit 113 may specify one latent state p23 from among the multiple latent states depicted on the latent state mapping screen G53 via the operation device 140. The restored image processing unit 113 may display the specified latent state p23 on the display device 150 in a display mode different from that of the other latent states p21 and p22. The restored image processing unit 113 generates a restored image corresponding to the specified latent state p23 based on the specified latent state p23. For example, the restored image processing unit 113 performs restoration processing on the specified latent state p23 to generate a restored image. In this case, the restored image processing unit 113 may use the specified latent state p23 as an input to the world model and obtain the restored image as an output of the world model. The restored image processing unit 113 displays the generated restored image on the display device 150. By checking the display of the restored screen G61 including this restored image, the user can confirm how the agent understands the environment, the agent's learning level of the world model, and the like.

[0083] The latent state is a compressed observed image. Also, the restored image is a restored latent state, and therefore is a compressed and restored observed image. Therefore, if the learning accuracy of the world model that derives the latent state from the observed image is high, the observed image G4 showing the observed image and the restored image G6 (G61) showing the restored image should match. In the list screen GL of Figure 4, the observed image on the observed screen G and the restored image on the restored screen G6 are slightly different. Therefore, in this case, the user can understand that the learning state of the world model is not complete.

[0084] Next, an example of the operation of the information processing system 5 will be described. FIG. 10 is a sequence diagram showing an example of the operation of the information processing system 5.

[0085] First, in the GUI unit 100, the information designation unit 111 inputs and designates a trained world model and a trained policy model via the operation device 140 (S11). The trained models here include a world model and a policy model. For example, one or more trained world models and one or more trained policy models may be stored in the memory 120, and it may be specified which world model and policy model to use.

[0086] The information designation unit 111 inputs the environmental conditions via the operation device 140 and designates the environmental conditions (S12). The environmental conditions here include an environment file related to the environment, camera information of the virtual camera (e.g., the position and angle (direction) of the virtual camera), information on the behavior of the agent that affects the environment (e.g., information on the operation of the robot device), etc. For example, one or more environment files may be stored in the memory 120, and it may be designated which environment file to use.

[0087] The communication device 130 transmits the determined policy model and environmental conditions to the simulator unit 200, and instructs the simulator unit 200 to set (reflect) the policy model and environmental conditions (S13).

[0088] In the simulator unit 200, the information setting unit 211 receives the policy model and the environmental conditions from the GUI unit 100 via the communication device 230, sets the policy model and the environmental conditions, and stores the setting information in the memory 220 (S21).

[0089] The policy execution unit 212 executes a policy in the environment where the agent is placed, based on the set environmental conditions and policy model (S22). The execution of the policy causes the agent to operate in the environment, which may change the environment. The observed image processing unit 213 controls the virtual camera to capture an image of the environment in accordance with the set camera information, and generates an observed image (S23). That is, the observed image processing unit 213 acquires an observed image via the virtual camera as environmental information after reflecting the environmental conditions. The communication device 230 transmits the acquired observed image to the GUI unit 100 (S24).

[0090] The measures are executed continuously in time series according to the policy model. That is, the policy execution unit 212 executes the policy model sequentially, and the observed image processing unit 213 acquires observed images sequentially via the virtual camera. The communication device 230 may transmit the sequentially acquired observed images one by one, or may repeatedly transmit several observed images at once.

[0091] In the GUI unit 100, the communication device 130 receives the observed images from the GUI unit 100 (S14). In this case, the communication device 130 may sequentially receive the observed images one by one, or may sequentially receive several observed images at a time.

[0092] The latent state processing unit 112 compresses the observed image based on the observed image and the learned world model (specified world model) to derive a latent state corresponding to the observed image (S15). Since the latent state is generated for each observed image, multiple latent states are derived. The multiple latent states are mapped to a latent space, which is a vector space. In this case, the original multidimensional latent state may be dimensionally compressed into a two-dimensional or three-dimensional latent state.

[0093] The restored image processing unit 113 restores the latent state based on the derived latent state and the learned world model (specified world model) to generate a restored image (S16). The restored image processing unit 113 may generate a restored image corresponding to a latent state specified via, for example, the operation device 140. Alternatively, the restored image processing unit 113 may generate one or more restored images corresponding to any one or more restored images without specifying a particular latent state.

[0094] The display device 150 displays the derived observed image, latent state, and restored image (S17). For example, the observed image is displayed on the observation screen G4, the latent state is displayed on the latent state mapping screen G5, and the restored image is displayed on the restored screen G6. The display device 150 may also display a list screen GL including these images. In this way, the GUI unit 100 calculates and visualizes the restored image and latent space.

[0095] In this way, the information processing system 5 can present to the user the distribution of multiple latent states on the latent state mapping screen G5 that shows the latent space, and a restored image in which the latent states are restored. Therefore, the information processing system 5 can visualize the quality of the learning state of the world model.

[0096] In the world model, the observed image and the restored image are not completely reversible, and the amount of information may gradually decrease as compression and restoration are repeated. In the world model, latent states that characteristically capture the observed information of the task are acquired, and the restored image is trained to approach the original observed image. In addition, since the world model predicts time series by nature, feature points (latent states) with similar properties are designed to be close in vector space.

[0097] By checking the distribution of multiple latent states, the user can recognize which observations the agent has judged to be close (semantically similar). Furthermore, by checking the restored image, the user can confirm how the agent understands and how well the world model has learned. If the restored image is correctly restored, the user can confirm that both the latent states and the restoration have been properly learned. Therefore, the information processing system 5 can improve the interpretability of the world model by the user (e.g., an engineer) and facilitate experiments.

[0098] In this embodiment, the example mainly shows a single virtual camera and a single viewpoint, but the example may also show multiple virtual cameras and multiple viewpoints.

[0099] Furthermore, in this embodiment, the task is exemplified as the robot device 10 of the agent grabbing the ball 20, but is not limited to this. In other words, the task may be an action other than grabbing an object by the robot device 10. Furthermore, the agent may be something other than a robot device, such as an autonomous driving simulator or a game simulator. This embodiment is applicable to general technologies that use reinforcement learning of a world model.

[0100] Furthermore, in this embodiment, the processes shared between the GUI unit 100 and the simulator unit 200 may be shared in other ways. For example, the GUI unit 100 may be configured to have the simulator unit 200 perform as much of the processes as possible other than input via the operation device 140 and display by the display device 150. For example, the simulator unit 200 may include the latent state processing unit 112 and the restored image processing unit 113.

[0101] In this embodiment, the GUI unit 100 and the simulator unit 200 may be configured as an integrated information processing device. That is, the simulator unit 200 may be omitted, and processing may be completed only on the GUI unit 100 side. That is, the GUI unit 100 may have the measure execution unit 212 and the observed image processing unit 213.

[0102] Furthermore, in this embodiment, an example has been described in which a virtual environment is observed as an image, but the concept of this embodiment can also be applied to the observation of other information such as sound. For example, when observing sound, the virtual camera can be used as a virtual microphone, and the sound measured by the virtual microphone or the restored sound can be output instead of the observed image or the restored image.

[0103] Furthermore, in this embodiment, the display device 150 displays multiple latent states as points arranged at two-dimensional or three-dimensional coordinates in the dimensionally compressed latent space. However, other displays corresponding to two-dimensional or three-dimensional coordinates may also be used. For example, the display device 150 may display a table listing each coordinate as a numerical value. Furthermore, when displaying two-dimensional or three-dimensional points, the display device 150 may display lines or the like between adjacent points. In this case, the display device 150 may change the style (color or thickness) of the lines to reflect the similarity in higher dimensions. As a result of the loss of higher-level information during the dimensional compression process, latent states may appear close in the latent state mapping but may be far apart in higher dimensions. For example, when dimensional compression is performed to two dimensions, even if they appear close in the planar coordinates remaining as two-dimensional information, the depth coordinates lost during the compression process may be significantly far apart. A similar problem exists when four- or more-dimensional information is compressed to three dimensions. Therefore, the display device 150 can intuitively express the similarity taking into account the lost information by displaying the information lost in the process of dimensionality reduction as other information such as the shape of a line.

[0104] [Outline of the embodiment] As described above, the information processing system 5 of the above embodiment includes an information specifying unit 111 (an example of a specifying unit) that specifies a trained world model, environmental data (e.g., an environment file) related to the environment in which an agent operates, and a trained policy model that defines the behavior of the agent. The information processing system 5 also includes a policy execution unit 212 that sequentially executes policies in the environment based on the specified environmental data and policy model, and an observed image processing unit 213 that captures an image of the environment with a virtual camera for each execution of a policy and generates a plurality of observed images. The information processing system 5 also includes a latent state processing unit 112 that compresses each of the plurality of observed images based on the world model to derive each of a plurality of latent states, and a display device 150 (an example of a display unit) that displays the plurality of latent states.

[0105] That is, the information processing system 5 can sample the results of execution according to the specified environment and policy and visualize the latent state mapping. The latent states are obtained from the observed image based on the world model. The latent states correspond to the observed image and indicate characteristic parts of the environment. Therefore, if the features indicated by the multiple latent states are similar, the distance between the multiple mapped latent states will be close, and if the features are dissimilar, the distance between the multiple mapped latent states will be far. Therefore, by checking the display of the latent state mapping, the user can easily intuitively understand the learning state of the world model based on, for example, the difference between the distance between the multiple latent states and the expected distance.

[0106] Furthermore, the latent state processing unit 112 may compress each of the multiple observed images based on the world model to derive each of multiple multidimensional latent states, and may dimensionally compress each of the multiple multidimensional latent states to derive each of multiple two-dimensional or three-dimensional latent states. The display device 150 may display the multiple latent states in association with two-dimensional or three-dimensional coordinates in the dimensionally compressed latent space.

[0107] This allows the information processing system 5 to adjust the latent state so that it can be displayed even if the latent state is multidimensional when it is first derived based on the world model.

[0108] The information processing system 5 may further include a restored image processing unit 113. The restored image processing unit 113 may specify one of the multiple latent states, and restore the specified one latent state based on the world model to generate a restored image. The display device 150 may display the restored image.

[0109] This allows the user to check the display of the restored image to see how the agent understands the environment, the degree of learning of the agent's world model, and so on.

[0110] The information designation unit 111 may designate camera information including the position and orientation of a virtual camera relative to the environment. The observed image processing unit 213 may capture an image of the environment with the virtual camera based on the camera information, and generate a plurality of observed images.

[0111] This allows the information processing system 5 to acquire observation images captured from various positions and in various directions, and to grasp the learning state of the world model in detail.

[0112] The information processing system 5 may further include an operation device 140 (an example of a first operation unit). The observational image processing unit 213 may change the camera information specified by the information specification unit 111 based on an operation on the operation device 140, and may capture an image of the environment with a virtual camera based on the changed camera information to generate a changed observational image. The display device 150 may display the changed observational image.

[0113] This allows the information processing system 5 to specify the camera information of the virtual camera while checking the observed image generated in accordance with the change in the camera information. Therefore, the camera information can be adjusted so that the observed image desired by the user can be obtained.

[0114] The information processing system 5 may further include an operation device 140 (an example of a second operation unit). The display device 150 may rotate and display the latent space depicting the latent state based on an operation on the operation device 140.

[0115] As a result, the information processing system 5 can view the latent space from various viewpoints and in various directions, compared to when the viewpoint (camera position) and line of sight (camera orientation) for checking the latent space are fixed, and the details of the positional relationships between multiple latent states arranged in the latent space can be easily confirmed.

[0116] The information processing system 5 may also include a GUI unit 100 (an example of an operation display device) and a simulator unit 200 (an example of a simulator device). The GUI unit 100 and the simulator unit 200 may be communicatively connected. The GUI unit 100 may include an information designation unit 111, a latent state processing unit 112, and a display device 150. The simulator unit 200 may include a measure execution unit 212 and an observed image processing unit 213.

[0117] As a result, the information processing system 5 can execute, for example, a measure with a high processing load on the simulator unit 200 side, which has a high processing capacity, and can perform distributed processing according to the processing capacity of each device.

[0118] Furthermore, the information processing system 5 may be configured by a single information processing device (for example, the GUI unit 100). This allows the information processing system 5 to complete the process for checking the learning state of the world model using a single information processing device.

[0119] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components of the above-described embodiments may be combined in any manner without departing from the spirit of the disclosure.

[0120] The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings, is not specifically stated as "before," "prior to," etc., and can be realized in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, this does not mean that it is essential to perform the process in that order. [Industrial Applicability]

[0121] The present disclosure is useful for an information processing device and an information processing method that can make it easier to intuitively understand the learning state of a world model. [Explanation of symbols]

[0122] 5. Information Processing Systems 100 GUI section 110 processors 111 Information specification section 112 Latent State Processing Unit 113 Image restoration processing section 120 memory 130 Communication Devices 140 Operating Devices 150 display devices 200 Simulator Section 210 processor 211 Information Setting Section 212 Policy Implementation Department 213 Observation image processing section 220 memory 230 Communication Devices GL list screen G1 environment screen G2 model setting screen G3 camera settings screen G4 observation screen G5 Latent State Mapping Screen G6 recovery screen

Claims

1. a designation unit that designates a trained world model, environmental data relating to an environment in which an agent operates, and a trained policy model that defines the behavior of the agent; a policy execution unit that sequentially executes policies in the environment based on the environmental data and the policy model; an observed image processing unit that captures an image of the environment with a virtual camera for each execution of the measure and generates a plurality of observed images; a latent state processing unit that compresses each of the plurality of observed images based on the world model to derive each of a plurality of latent states; a display unit that displays the plurality of potential states; An information processing system comprising:

2. The latent state processing unit compressing each of the plurality of observed images based on the world model to derive each of a plurality of multidimensional latent states; dimensionally compressing each of the plurality of multidimensional latent states to derive each of a plurality of two-dimensional or three-dimensional latent states; the display unit displays the plurality of latent states in association with two-dimensional or three-dimensional coordinates in the dimensionally reduced latent space. The information processing system according to claim 1 .

3. a restored image processing unit, The restored image processing unit designating one latent state from the plurality of latent states; Restoring the specified latent state based on the world model to generate a restored image; The display unit displays the restored image.

3. The information processing system according to claim 1 or 2.

4. the designation unit designates camera information including a position and an orientation of the virtual camera with respect to the environment; the observation image processing unit captures an image of the environment with the virtual camera based on the camera information, and generates the plurality of observation images. The information processing system according to any one of claims 1 to 3.

5. a first operating unit, The observed image processing unit changing the camera information designated by the designation unit based on an operation on the first operation unit; capturing an image of the environment with the virtual camera based on the modified camera information to generate the modified observed image; the display unit displays the modified observation image. The information processing system according to claim 4 .

6. a second operating unit, the display unit rotates and displays the latent space depicting the latent state based on an operation on the second operation unit. The information processing system according to any one of claims 1 to 5.

7. the information processing system includes an operation display device and a simulator device, the operation display device and the simulator device are communicably connected; the operation display device includes the designation unit, the observed image processing unit, the latent state processing unit, and the display unit; the simulator device includes the policy execution unit, The information processing system according to any one of claims 1 to 6.

8. The information processing system is configured by a single information processing device. The information processing system according to any one of claims 1 to 6.

9. specifying a learned world model, environmental data relating to the environment in which the agent operates, and a learned policy model that governs the behavior of said agent; sequentially executing strategies in the environment based on the environmental data and the strategy model; capturing images of the environment with a virtual camera for each execution of the strategy to generate a plurality of observed images; compressing each of the plurality of observed images based on the world model to derive each of a plurality of latent states; displaying the plurality of latent states on a display unit; An information processing method comprising:

Images