Information processing system, information processing method, and program

Abstract

Provided is an information processing system capable of obtaining information for controlling the operation of an autonomous moving body. The information processing system is provided with: an image acquisition unit that acquires an image of an object including the surroundings of a moving body; a statement output unit that inputs an image and a statement generation instruction into a model that creates a statement including information pertaining to navigation of the moving body among objects included in the acquired image, and outputs the statement; and an operation method specifying unit that specifies an operation method of the moving body on the basis of the output utterance.

Description

Information processing systems, information processing methods and procedures Technical Field

[0001] This invention relates to an information processing system, information processing method, and program. Background Technology

[0002] Patent document 1 discloses a mobile body control device that uses a learned model that learns the correspondence between language and images to determine the parking position of the mobile body.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2024-31978 The mobile body control device described in Patent Document 1 predetermines how to control the mobile body's movements, but it cannot autonomously determine and control its movement based on constantly changing environments. Therefore, the object of the present invention is to provide an information processing system or the like that capable of obtaining information for controlling the movements of an autonomous mobile body.

[0004] The information processing system of the present invention comprises: an image acquisition unit that acquires an image of an object including the surrounding environment of a moving body; a description output unit that inputs the image and a description output instruction into a model, the model creating a description including information related to navigation of the moving body in the acquired image and the object included therein, and outputting the description; and an action method determination unit that determines an action method of the moving body based on the output description.

[0005] The above structure enables the provision of an information processing system capable of acquiring information for controlling the actions of autonomous moving bodies.

[0006] The information processing system of the present invention is characterized in that the model for creating the explanatory text of the information related to the navigation of the moving body in the object included in the image uses a visual language model (VLM).

[0007] The above structure is an example of a model for creating a descriptive text that includes information related to the navigation of a moving object.

[0008] The information processing system of the present invention is characterized in that the action method determination unit uses a model learned through reinforcement learning or imitation learning.

[0009] The above structure is an example of a learning method in machine learning.

[0010] The information processing method of the present invention includes the following steps: acquiring an image of an object including the surrounding environment of a moving body; inputting the image and a description text generation instruction into a model, the model creating the description text including information related to the navigation of the moving body in the object included in the acquired image, and outputting the description text; and determining the action method of the moving body based on the output description text.

[0011] The above structure provides an information processing method that can obtain information for controlling the actions of an autonomous moving body.

[0012] The program of the present invention causes an information processing device to perform the following steps: acquiring an image of an object including the surrounding environment of a moving body; inputting the image and a description text generation instruction into a model, the model creating the description text including information related to the navigation of the moving body in the objects included in the acquired image, and outputting the description text; and determining a method of action for the moving body based on the output description text.

[0013] The above structure enables the provision of a program that can obtain information for controlling the actions of an autonomous moving body.

[0014] Effects of the Invention: This invention provides an information processing system and the like that can obtain information for controlling the actions of an autonomous moving body. Attached Figure Description

[0015] Figure 1 is a schematic block diagram illustrating the information processing system and information processing method involved in the implementation.

[0016] Figure 2 is a block diagram illustrating an example of an information processing system according to an embodiment.

[0017] Figure 3 is a flowchart of the information processing method involved in the implementation method. Detailed Implementation

[0018] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the invention to the following embodiments. Furthermore, not all configurations described in the embodiments are necessarily necessary as means of solving the problem. For clarity, appropriate omissions and simplifications have been made in the following description and drawings. In the drawings, the same elements are labeled with the same symbols, and repeated descriptions are omitted as necessary.

[0019] (Brief Description of the Information Processing System According to the Embodiment) FIG1 is a block diagram showing a general description of the information processing system and information processing method according to the embodiment. Referring to FIG1, a general description of the information processing system according to the embodiment will be given. The information processing system is used to output the movement path of the moving body.

[0020] As shown in Figure 1, the information processing system 100 involved in the embodiment includes a visual language model (VLM) 101 and a learning database 102.

[0021] VLM101 is an artificial intelligence model that combines the capabilities of visual and language models. VLM101 reads and learns from descriptions of images and text, associating them. The visual component of the VLM101 model extracts information from images, while the language component extracts information from text. VLM101 then analyzes and outputs information from both images and language. VLM101 is also described as a model that creates descriptive text that includes information related to navigation of moving objects within an image.

[0022] Learning database 102 is a collection of text and information about corresponding action methods associated with the text. Learning database 102 is a database used to input text and output action methods.

[0023] As shown in the upper part of Figure 1, the image acquisition unit in the information processing system acquires an image (step S101). The image is an image of objects including the surrounding environment of the moving object. As shown in the lower part of Figure 1, the image includes a person indicating an action, directional signs, etc.

[0024] The acquired image is input into VLM101. Then, the interpretation of the direction as a human gesture (step S102) and the interpretation of the sign as a signboard are performed (step S103). VLM101, which is the explanatory text output unit, outputs explanatory text for objects such as signs and people making gestures included in the image based on the explanatory text generation instruction, which includes information related to the navigation of the moving body. The explanatory text generation instruction is generated by recognizing events such as signs, gestures, and obstacles in the image. As shown in the lower part of Figure 1, VLM101 is used to convert the images of gestures and signs into language.

[0025] For example, inputting an image and the question "Which direction should we go?" into the VLM101 transforms it into language that reflects the robot's actions, such as right and left directions. The VLM101 is a foundational model for combining image and text processing and has been trained on a large dataset. Because it learns from diverse datasets, rather than specific instructions and labels, the VLM101 has the ability to interpret navigation-related information such as instructions, labels, and signs. Furthermore, the VLM101 can associate its interpretations with linguistic information, thus outputting the processing results as language.

[0026] As shown in the upper part of Figure 1, the motion method determination unit in the information processing system inputs the description of the object into the motion method model that has learned the motion method for generating the moving body (step S104), and outputs the motion method based on the description (step S105). As shown in the lower part of Figure 1, the model that has learned the motion method generates the robot's motion conditionally based on language tags. Then, the processing ends.

[0027] In step S105, the model learned through reinforcement learning or imitation learning is used to generate actions in step S104. The model that has learned the action method uses reinforcement learning or imitation learning to learn a policy for language z and observed image s. The image and language are compressed using an encoder such as a transformer to reduce the dimensionality of the feature quantities. By conditionalizing with language, actions that take that language into account can be generated. For example, given the language information "right," the optimal path in the right direction within the current image can be selected from the current image. In step S105, any method can be used as long as the action can be inferred based on the language and the current observation. For example, methods such as imitation learning, where a person teaches actions based on language labels and learns from that data, or reinforcement learning, where the learning occurs autonomously based on a reward function, can be considered.

[0028] (Explanation of the functions of the information processing system according to the embodiment) Figure 2 is a functional block diagram showing an example of the information processing system according to the embodiment. Referring to Figure 2, the functions of the information processing system according to the embodiment will be explained.

[0029] As shown in FIG2, the information processing system 100 according to the embodiment includes a camera 201, a meaning understanding unit 202, a movement planning unit 203, and a moving body 204.

[0030] Camera 201 acquires images of objects including the surrounding environment of the moving object. As shown in Figure 2, it can be mounted on the moving object 204 or be a surveillance camera that captures images of the area around the moving object. Camera 201 can be a camera equipped with an imaging element such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS). Camera 201 can be an artificial intelligence (AI) stereo camera, an RGBD camera, or a single-lens reflex camera.

[0031] The meaning understanding unit 202 uses VLM 101 to acquire images from the camera and understands the meaning of the images. For example, the meaning understanding unit 202 understands the meaning of human gestures or symbols. The meaning understanding unit 202 understands the content shown in the image and outputs it as text.

[0032] The movement planning unit 203 calculates movement parameters of the moving body, such as movement direction and speed. The movement planning unit 203 acquires images from the camera and determines the presence of obstacles. The movement planning unit 203 determines the movement direction to avoid obstacles. Furthermore, the movement planning unit 203 obtains the meaning of the images from the meaning understanding unit 202 using text. The movement planning unit 203 uses the learning database 102 to output actions corresponding to the text and issues instructions to the moving body.

[0033] The mobile body 204 can be a trolley robot, a walking robot, a humanoid robot, an autonomous vehicle, or a drone, etc. The mobile body 204 may include, for example, a control unit, a trolley drive unit, an upper body drive unit, a display unit, a stereo camera, a laser scanner, a memory, and a hand camera. Alternatively, the upper body drive unit, display unit, stereo camera, laser scanner, and hand camera may be omitted.

[0034] The control unit is a processor such as a CPU, and its contents are stored in a control unit located in the torso. The control unit performs overall control and various calculations of the mobile body 204 by executing control programs read from memory.

[0035] For example, the control unit executes rotation control of the drive wheels by sending drive signals to the trolley drive unit, based on the latest motion plan stored in the memory. Furthermore, the control unit receives feedback signals from the trolley drive unit, such as encoders, to determine the trolley's direction of movement or speed.

[0036] The trolley drive unit includes a drive wheel and a drive circuit or motor for driving the drive wheel.

[0037] The upper body drive unit includes the arms and hands, the torso and head, and drive circuits or motors for driving these. The control unit realizes extension movements, gripping movements, or gestures by sending drive signals to the torso drive unit. Furthermore, the control unit receives feedback signals from the torso drive unit, such as encoders, to monitor the position or movement speed of the arms and hands, and the orientation or rotation speed of the torso and head.

[0038] The display unit receives and displays the image signals generated by the control unit.

[0039] The stereo camera captures images of the surrounding environment of the moving object 204 upon request from the control unit and transfers the capture signal to the control unit. The control unit performs image processing using the capture signal or converts the capture signal into an image according to a predetermined format. The laser scanner detects the presence of obstacles in the direction of movement upon request from the control unit and transfers the detection signal, which is the detection result, to the control unit.

[0040] A hand camera, such as a distance image sensor, is used to identify the distance, shape, and orientation of an object being held. The hand camera includes an imaging element that photoelectrically converts an optical image incident from the object space into light, arranged in a two-dimensional pattern, and outputs the distance to the subject per pixel to a control unit. Specifically, the hand camera includes an illumination unit that illuminates patterned light into the object space, receives the reflected light using an imaging element, and outputs the distance to the subject captured by each pixel based on the distortion or size of the pattern in the image. Furthermore, the control unit uses a stereo camera to grasp a wider range of surrounding conditions and a hand camera to monitor the area near the object being held.

[0041] The memory is a non-volatile storage medium, such as a solid-state drive. In addition to the control program used to control the moving body 204, the memory also stores various parameter values, functions, lookup tables, etc., used for control or calculation. In particular, the memory stores an environment map and action plans.

[0042] The above structure enables the provision of an information processing system capable of acquiring information for controlling the actions of autonomous moving bodies.

[0043] (Explanation of the information processing method according to the embodiment) FIG3 is a flowchart of the information processing method according to the embodiment. Referring to FIG3, the information processing method according to the embodiment will be described.

[0044] First, an image is acquired (step S301). An image is acquired by camera 201. The image is preferably represented as an RGB image. Next, information usable for navigation is extracted from the image using VLM 101, and language tags are assigned (step S302). The information usable for navigation includes symbols and gestures, etc. Assigning language tags means converting the navigation information in the image into text. The meaning understanding unit 202 receives a command to generate explanatory text including information related to the navigation of the moving body, and converts the navigation information, such as symbols, into text.

[0045] Next, actions are output from the language tags and the current observation screen (step S303). The movement planning unit 203 uses the learning database 102 to output the actions that the moving body should take based on the text and the observation screen, and issues instructions to the moving body. Then, the process ends.

[0046] The above structure provides an information processing method that can obtain information for controlling the actions of an autonomous moving body.

[0047] The above-described information processing method is implemented by an information processing device executing a program. The information processing device includes a memory for executing the program and a memory for storing the program. The information processing device can be a single device or multiple devices. For the purpose of constructing an autonomous mobile body, it is preferable to include the information processing device within the mobile body. However, the mobile body can also be controlled by communicating with an external information processing device.

[0048] Part or all of the processing in the information processing apparatus and information processing system 100 can be implemented as a computer program. Such a program can be stored and supplied to the computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of physical recording media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), read-only memory (CD-ROM), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash memory ROM, random access memory (RAM)). Furthermore, the program can also be supplied to the computer via various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can supply the program to the computer via wired or wireless communication paths such as wires and optical fibers.

[0049] Furthermore, the present invention is not limited to the above-described embodiments and can be appropriately modified without departing from the main idea. For example, a VLM can be used to convert semantic information from an image that can be used for navigation into language, but in step S104, as long as language information is input, the robot's actions conditionalized in that language can be generated. For example, when the robot does not know which direction it should move in, the robot asks a person "Which direction is ○○?" Then, human-robot interaction (HRI) can be considered, that is, asking a person for directions such as right and left and inputting it into step S105. At this time, it can be easily implemented by using an audio language model such as Whisper to convert speech into language or using a large language model (LLM) such as ChatGPT (registered trademark) to convert language information into expressions that are easy to use in robot actions.

[0050] Symbol Explanation: 100 - Information Processing System, 101 - VLM, 102 - Learning Database, 201 - Camera, 202 - Meaning Understanding Department, 203 - Mobility Planning Department, 204 - Mobility Body.

Claims

1. An information processing system, characterized in that, It comprises: an image acquisition unit that acquires images of objects including the surrounding environment of a moving body; a description output unit that inputs the image and a description generation instruction into a model, which creates and outputs the description including information related to the navigation of the moving body in the objects included in the acquired image; and an action method determination unit that determines the action method of the moving body based on the output description.

2. The information processing system according to claim 1, characterized in that, A visual language model was used to create a model of the descriptive text relating to navigation of the moving body in the objects included in the image.

3. The information processing system according to claim 1, characterized in that, The action method determination unit uses a model learned through reinforcement learning or imitation learning.

4. An information processing method, characterized in that, The method includes the following steps: acquiring an image of objects including the surrounding environment of the moving body; inputting the image and a description text generation instruction into a model, which creates a description text including information related to the navigation of the moving body in the objects included in the acquired image, and outputting the description text; and determining the action method of the moving body based on the output description text.

5. A program, characterized in that, The information processing device performs the following steps: acquiring an image of an object including the surrounding environment of the moving body; inputting the image and a description text generation instruction into a model that creates a description text including information related to the navigation of the moving body in the objects included in the acquired image, and outputting the description text; and determining the action method of the moving body based on the output description text.

Images