Information processing systems, information processing methods, and programs

Abstract

This system provides an information processing system that can obtain information that can be used for controlling the movements of autonomous mobile objects. [Solution] The present invention provides an information processing system comprising: an image acquisition unit that acquires an image including objects in the surrounding environment of a moving object; an explanatory text output unit that inputs the image and instructions for generating the explanatory text to a model that creates an explanatory text containing information about the navigation of the moving object for the objects included in the acquired image, and outputs the explanatory text; and an operation method determination unit that determines the operation method of the moving object based on the output explanatory text.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Patent Document 1 discloses a movement control device that identifies a parking position of a moving body using a learned model that has learned the correspondence between language and images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The movement control device described in Patent Document 1 determines in advance how to control the operation of the moving body, and does not control the movement while autonomously determining according to the environment that can change at any time. Therefore, an object of the present disclosure is to provide an information processing system or the like that can obtain information that can be used for autonomous movement control of a moving body.

Means for Solving the Problems

[0005] The information processing system of the present disclosure includes an image acquisition unit that acquires an image including an object in the surrounding environment of the moving body, a description text output unit that inputs the image and a generation instruction of the description text to a model that creates a description text including information related to navigation of the moving body in the object included in the acquired image, and outputs the description text, and an operation method determination unit that determines an operation method of the moving body based on the output description text.

[0006] The above configuration provides an information processing system that can obtain information that can be used for controlling the movement of autonomous mobile objects.

[0007] The information processing system disclosed herein is The model for generating a descriptive text containing information about the navigation of the moving object in the aforementioned image is characterized by using a Visual Language Model (VLM).

[0008] The above configuration is an example of a model for creating a descriptive text that includes information about the navigation of a moving object.

[0009] The information processing system disclosed herein is The operation method determination unit is characterized by using a model that has undergone reinforcement learning or imitation learning.

[0010] The above configuration is one example of a machine learning training method.

[0011] The information processing method disclosed herein is: Acquire an image of the surrounding environment of the moving object, including objects. A model that generates a descriptive text containing information about the navigation of the moving object in the acquired image is given the image and an instruction to generate the descriptive text, and the model outputs the descriptive text. This is an information processing method that determines the operation method of the mobile body based on the outputted explanatory text.

[0012] The above configuration provides an information processing method that can obtain information that can be used for controlling the movement of an autonomous mobile object.

[0013] The program disclosed herein is Acquire an image of the surrounding environment of the moving object, including objects. A model that generates a descriptive text containing information about the navigation of the moving object in the acquired image is given the image and an instruction to generate the descriptive text, and the model outputs the descriptive text. A program that causes an information processing apparatus to determine an operation method of the mobile body based on the output description text.

[0014] The above configuration can provide a program that can obtain information available for autonomous control of the operation of a mobile body.

Effect of the Invention

[0015] According to the present disclosure, an information processing system or the like that can obtain information available for autonomous control of the operation of a mobile body can be provided.

Brief Description of the Drawings

[0016] [Figure 1] It is a block diagram showing an outline of an information processing system and an information processing method according to an embodiment. [Figure 2] It is a block diagram showing an example of an information processing system according to an embodiment. [Figure 3] It is a flowchart of an information processing method according to an embodiment.

Modes for Carrying Out the Invention

[0017] Embodiment Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiments. Also, not all of the configurations described in the embodiments are essential as means for solving the problems. For clarity of explanation, the following description and drawings have been appropriately omitted and simplified. In each drawing, the same elements are denoted by the same reference numerals, and duplicate explanations are omitted as necessary.

[0018] (Explanation of the Outline of the Information Processing System According to the Embodiment) FIG. 1 is a block diagram showing an outline of an information processing system and an information processing method according to an embodiment. With reference to FIG. 1, the outline of the information processing system according to the embodiment will be described. The information processing system is used to output the movement path of the mobile body.

[0019] As shown in FIG. 1, the information processing system 100 according to the embodiment includes a VLM (Vision Language Model) 101 and a learning database 102.

[0020] The VLM 101 is an artificial intelligence model that combines the capabilities of vision and language models. The VLM 101 takes in an image and a text description, associates them, and learns. In the VLM 101, the visual part of the model extracts information from the image, and the language part extracts information from the text. The VLM 101 obtains information from both the image and the language, analyzes it, and outputs the result. The VLM 101 is also called a model that creates a description text containing information related to the navigation of a moving object in an object included in an image.

[0021] The learning database 102 is a set of information on text and the operation method associated with the text. The learning database 102 is a database for inputting text and outputting an operation method.

[0022] As shown in the upper diagram of FIG. 1, the image acquisition unit of the information processing system acquires an image (step S101). The image is an image including objects in the surrounding environment of the target moving object. The image includes, as shown in the lower diagram of FIG. 1, a person pointing, a direction sign, and the like.

[0023] The acquired image is input into the VLM 101. Then, finger-pointing interpretation (step S102) as a human gesture and sign interpretation (step S103) as a signboard are executed. The VLM 101, which is a description text output unit, outputs a description text of objects such as signs and gesturing people included in the image according to a generation instruction of a description text including information related to the navigation of the moving object. The generation instruction of the description text occurs when a sign, gesture, obstacle, etc. are recognized in the image and an event occurs. As shown in the lower diagram of FIG. 1, the VLM 101 is used to convert images of gestures and signboards into language.

[0024] For example, an image and the question "Which direction should I go?" are input to VLM101, which then converts them into language that can be reflected as robot actions, such as right and left. VLM101 is a type of foundational model that handles images and text in combination and is trained on a large dataset. Because VLM101 is trained on a wide variety of datasets, rather than just specific instruction and sign datasets, it has the ability to interpret semantic information related to navigation, such as instructions, signs, and billboards. Furthermore, VLM101 can link its interpretations to linguistic information, allowing it to output processing results as language.

[0025] As shown in the upper part of Figure 1, the operation method determination unit of the information processing system inputs a description of the object to a model that has learned the operation method to generate the movement of a moving object (step S104), and outputs an operation method based on the description (step S105). As shown in the lower part of Figure 1, the model that has learned the operation method generates the movement of the robot conditioned by the language label. Then the process ends.

[0026] The model trained in step S105 using reinforcement learning or imitation learning is used to generate actions in step S104. The model, having learned how to perform actions, is trained using reinforcement learning or imitation learning to develop policies for language z and observed image s. However, the image and language are dimensionally reduced to features using an encoder such as a transformer. By conditioning with language, it is possible to generate actions that take that language into account. For example, if the language information "right" is input, it is possible to select the optimal path in the right direction within the passable area from the current image. Step S105 can be any method as long as it is possible to estimate an action from language and the current observation. For example, a method in which a human teaches an action for a language label, as in imitation learning, and the model learns based on that data, or a method in which the model learns autonomously based on a reward function, as in reinforcement learning, can be considered.

[0027] (Description of the functions of the information processing system according to the embodiment) Figure 2 is a functional block diagram showing an example of an information processing system according to the embodiment. The functions of the information processing system according to the embodiment will be explained with reference to Figure 2.

[0028] As shown in Figure 2, the information processing system 100 according to this embodiment includes a camera 201, a semantic understanding unit 202, a movement planning unit 203, and a moving body 204.

[0029] Camera 201 acquires an image including objects in the surrounding environment of the target moving object. As shown in Figure 2, it may be mounted on the moving object 204, or it may be a surveillance camera that images the surroundings including the moving object. Camera 201 may be a camera equipped with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 201 may be an AI (Artificial Intelligence) stereo camera, an RGBD camera, or a monocular camera.

[0030] The semantic understanding unit 202 acquires images from the camera using the VLM 101 and understands the meaning of the images. For example, the semantic understanding unit 202 understands the meaning of a person's gesture or a sign. The semantic understanding unit 202 understands the content shown in the image and outputs it as text.

[0031] The movement planning unit 203 calculates movement parameters of the moving object, such as the direction of movement and the speed of movement. The movement planning unit 203 acquires images from the camera and determines the presence or absence of obstacles. The movement planning unit 203 determines the direction of movement to avoid obstacles. The movement planning unit 203 also acquires the meaning of the images as text from the semantic understanding unit 202. The movement planning unit 203 uses the learning database 102 to output actions corresponding to the text and issues commands to the moving object.

[0032] The mobile unit 204 may be a trolley robot, a legged robot, a humanoid, an autonomous vehicle, or a drone. The mobile unit 204 may include, for example, a control unit, a trolley drive unit, an upper body drive unit, a display unit, a stereo camera, a laser scanner, memory, a hand camera, etc. However, the upper body drive unit, display unit, stereo camera, laser scanner, and hand camera may be omitted.

[0033] The control unit is a processor such as a CPU, and is stored, for example, in a control unit located in the torso. The control unit executes control programs read from memory, thereby controlling the entire mobile body 204 and performing various calculations.

[0034] For example, the control unit controls the rotation of the drive wheels by sending drive signals to the trolley drive unit according to the latest operation plan stored in memory. The control unit also receives feedback signals from the trolley drive unit, such as encoders, to determine the direction and speed of movement of the trolley.

[0035] The bogie drive unit includes drive wheels and a drive circuit and motor for driving the drive wheels.

[0036] The upper body drive unit 146 includes an arm and a hand, a torso and a head, and drive circuits and motors for driving them. The control unit sends drive signals to the upper body drive unit to realize extension, gripping, and gestures. The control unit also receives feedback signals from the upper body drive unit, such as encoders, to determine the position and movement speed of the arm and hand, and the orientation and rotation speed of the torso and head.

[0037] The display unit receives and displays the image signal generated by the control unit.

[0038] The stereo camera, in accordance with a request from the control unit, captures images of the surrounding environment where the moving object 204 is located and transmits the captured signal to the control unit. The control unit performs image processing using the captured signal or converts the captured signal into an image according to a predetermined format. The laser scanner, in accordance with a request from the control unit, detects whether or not an obstacle exists in the direction of movement and transmits the detection result, the detection signal, to the control unit.

[0039] A hand camera is, for example, a distance image sensor used to recognize the distance, shape, and direction of an object being grasped. The hand camera includes an image sensor in which pixels that convert an optical image incident from the target space into photoelectric data are arranged in a two-dimensional manner, and each pixel outputs the distance to the subject to the control unit. Specifically, the hand camera includes an illumination unit that irradiates the target space with patterned light, receives the reflected light with the image sensor, and outputs the distance to the subject captured by each pixel based on the distortion and size of the pattern in the image. The control unit grasps the surrounding environment from a wider area using a stereo camera and grasps the area near the grasped object using the hand camera.

[0040] Memory is a non-volatile storage medium, such as a solid-state drive. In addition to the control program for controlling the mobile unit 204, memory stores various parameter values, functions, lookup tables, and other data used for control and calculations. In particular, memory stores the environment map and the operation plan.

[0041] The above configuration provides an information processing system that can obtain information that can be used for controlling the movement of autonomous mobile objects.

[0042] (Description of the information processing method according to the embodiment) Figure 3 is a flowchart of the information processing method according to the embodiment. The information processing method according to the embodiment will be explained with reference to Figure 3.

[0043] First, an image is acquired (step S301). An image is acquired using camera 201. Preferably, the image is shown as an RGB image. Next, information usable for navigation is extracted from the image using VLM 101 and a language label is assigned (step S302). Information usable for navigation includes signs and gestures. Assigning a language label means converting the navigation information in the image into text. The semantic understanding unit 202 receives an instruction to generate an explanatory text containing information about the navigation of a moving object and converts the navigation information from signs and the like into text.

[0044] Next, the system outputs an action based on the language label and the current observation screen (step S303). The movement planning unit 203 uses the learning database 102 to output the action that the mobile body should take based on the text and the observation screen, and instructs the mobile body accordingly. Then the process ends.

[0045] The above configuration provides an information processing method that can obtain information that can be used for controlling the movement of an autonomous mobile object.

[0046] The above information processing method is implemented by executing a program on an information processing device. The information processing device includes a memory for executing a program to perform processing and a memory for storing the program. The information processing device may consist of one device or multiple devices. To constitute an autonomous mobile device, it is preferable that the information processing device is included in the mobile device. However, the mobile device may be controlled by communicating with an external information processing device.

[0047] Some or all of the processing in the information processing device and information processing system 100 can be implemented as a computer program. Such a program can be stored using various types of non-temporary computer-readable media and supplied to a computer. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Alternatively, the program may be supplied to the computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can be supplied to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0048] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, although VLM was used to convert semantic information usable for navigation into language from an image, step S104 can generate robot movements conditioned on that language if language information is input. For example, if the robot is unsure which direction to go, the robot will ask the human, "Which direction is XX?" The human will then instruct the robot, "Right," "Left," etc., and this information will be input into step S105. This can be applied to navigation that takes Human-Robot Interaction (HRI) into consideration. This can be easily achieved by using an Audio Language Model that converts speech into language, such as Whisper, or an LLM (Large Language Model) such as ChatGPT (registered trademark) to convert language information into an expression that is easy to use for robot movements. [Explanation of Symbols]

[0049] 100 Information processing system, 101 VLM, 102 Learning database, 201 Camera, 202 Semantic understanding unit, 203 Movement planning unit, 204 Mobile body

Claims

1. An image acquisition unit that acquires an image including objects in the surrounding environment of a moving object, A model that generates a descriptive text containing information about the navigation of the moving object in the acquired image receives the image and an instruction to generate the descriptive text as input, and outputs the descriptive text. An information processing system comprising: an operation method determination unit that determines the operation method of the moving body based on the outputted explanatory text.

2. The information processing system according to claim 1, wherein the model for generating a descriptive text containing information about the navigation of the moving object in the image uses a Visual Language Model (VLM).

3. The information processing system according to claim 1, wherein the operation method determination unit uses a model that has undergone reinforcement learning or imitation learning.

4. Acquire an image of the surrounding environment of the moving object, including objects. A model that generates a descriptive text containing information about the navigation of the moving object in the acquired image is given the image and an instruction to generate the descriptive text, and the model outputs the descriptive text. An information processing method for determining the operation method of the mobile body based on the outputted explanatory text.

5. Acquire an image of the surrounding environment of the moving object, including objects. A model that generates a descriptive text containing information about the navigation of the moving object in the acquired image is given the image and an instruction to generate the descriptive text, and the model outputs the descriptive text. A program that causes an information processing device to determine the operation method of the mobile body based on the outputted explanatory text.

Images