Information processing method, mobile body, and robot

Abstract

An information system according to an embodiment comprises a determination unit and an adding unit. The determination unit determines whether or not processing relating to an object arranged in an environment is appropriate on the basis of information on an image containing the object. The adding unit adds label information specified by a user to data on the object when the processing is determined not to be appropriate.

Description

[Technical field]

[0001] An embodiment of the present disclosure comprises: love Information processing method, Mobile , and robots To Regarding. [Background technology]

[0002] Smartphones can obtain the real-world position and time when an image was taken through an IMU (Inertial Measurement Unit) or GPS (Global Positioning System). Robots and self-driving cars can obtain more detailed information through SLAM (Simultaneous Localization And Mapping). Development of robots and self-driving cars is progressing with the technical goal of autonomous operation. For example, users are required to understand the world recognized by devices related to autonomous movement in robots and self-driving cars. In addition, in devices related to AR (Augmented Reality), real-world images obtained from a camera are linked to CG (Computer Graphics) or simulation space through calibration. These devices can link objects and environments with known shapes with 3D data, but there are many restrictions, such as the environment needing to be generated as a model in advance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO 11 / 083732 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the invention is to add information according to the needs of a user to data relating to an object. [Means for solving the problem]

[0005] Information processing according to the embodiment method teeth ,ring Inside the temple Recognizing objects in the environment by a movable body capable of moving the object Recognition results Based on the above, Regarding R A plurality of label candidates are displayed on a terminal different from the mobile object, and the label candidates are displayed on the terminal. User Based on an instruction from label Among the candidates The object Attached Add Determine the label . [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a perspective view showing an example of the appearance of a robot as a moving body according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional block in a processor according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a region-object correspondence table according to the embodiment. [Diagram 5] FIG. 5 is a flowchart illustrating an example of a processing procedure of the label addition processing according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a user interface displayed on a terminal according to the embodiment. [Figure 7] FIG. 7 is a diagram showing a display example of a label list, a label request, and data of a recognition-inappropriate object in a user interface displayed on a terminal according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a processing procedure of an alternative task addition process according to an application example of the embodiment. [Figure 9]FIG. 9 is a diagram showing an example of display of a user interface displayed on a terminal, which is related to an application example of an embodiment, and shows a list of notifications during task execution, a list of task target objects related to completed tasks, and images of task-impossible objects. [Figure 10] FIG. 10 is a diagram showing an example of a user interface displayed on a terminal in which a task list, requests for alternative tasks, and data on objects that cannot perform the task are displayed, according to an application example of the embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a user interface displayed on a terminal in which a task list TL, a request for an alternative task, and data on an object that cannot perform the task are displayed, according to an application example of the embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a user interface display for resetting a destination when "reset destination" is selected in the task list shown in FIG. 11 according to an application example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0008] 1 is a diagram showing an example of a hardware configuration of an information processing system 1 according to the present embodiment. As shown in FIG. 1, the information processing system 1 may include a moving object 2, an external device 7, and a terminal 9.

[0009] The moving body 2 of this embodiment may include an information processing device 21, a moving device 23, a gripping device 25, and a photographing device 27. A part or all of the components of the information processing device 21 may be arranged as a server via the communication network 5. A part or all of the processing executed in the information processing device 21 may be executed in a server (e.g., a cloud) via the communication network 5. In the following, for the sake of concrete explanation, the moving body 2 is assumed to be, for example, a tidying robot introduced into a home environment. The moving body 2 is not limited to a tidying robot, and may be, for example, a robot that is arranged in a logistics warehouse and manages items, a robot that performs housework, or a robot that is arranged in an environment related to moving, or any other robot that handles items. The moving body 2 is capable of autonomous travel and autonomous operation, and is charged at a charging station during standby.

[0010] FIG. 2 is a perspective view showing an example of the appearance of a robot as the moving body 2. A camera 27 may be mounted on the upper surface of a main body (hereinafter referred to as the robot main body) 29 of the moving body 2 of this embodiment. An information processing device 21 may further be mounted on the robot main body 29. The display device 211 and the input device 213 in the information processing device 21 are provided, for example, on the upper surface of the robot main body 29, on the rear side of the camera 27. A gripping device 25 may be provided on the side of the robot main body 29. A moving device 23 may be mounted on the lower surface of the robot main body 29. A blade 22 capable of pushing out an object may be provided on the front side of the robot main body 29.

[0011] The information processing device 21 includes, for example, a computer 3, and a display device 211 and an input device 213 connected to the computer 3 via a device interface 39. The computer 3 includes, for example, a processor 31, a main storage device 33, an auxiliary storage device 35, a network interface 37, and a device interface 39. The processor 31, the main storage device 33, the auxiliary storage device 35, the network interface 37, and the device interface 39 are connected to each other via, for example, a bus 41.

[0012] The computer 3 shown in FIG. 1 includes one of each component, but may include multiple of the same components. Although one computer 3 is shown in FIG. 1, the software may be installed in multiple computers, and each of the multiple computers may execute the same or different parts of the software. In this case, the computer may be in the form of distributed computing in which each computer communicates via a network interface 37 or the like to execute the processing. That is, each device in the embodiment may be configured as a system that realizes various functions described below by one or more computers executing instructions stored in one or more storage devices. Also, the information transmitted from the terminal 9 may be processed by one or more computers provided on the cloud, and the processing results may be transmitted to the mobile object 2 or the like.

[0013] Various calculations in the embodiments may be executed in parallel using one or more processors, or using multiple computers via a network. Also, various calculations may be distributed to multiple arithmetic cores in a processor and executed in parallel. Also, a part or all of the processes, means, etc. of the present disclosure may be executed by at least one of a processor and a storage device provided on a cloud that can communicate with the computer 3 via a network. In this way, various things described later in the embodiments may be in the form of parallel computing by one or more computers.

[0014] The processor 31 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, or the like) including a control device and an arithmetic device of the computer 3. The processor 31 may also be a semiconductor device or the like including a dedicated processing circuit. The processor 31 is not limited to an electronic circuit using electronic logic elements, and may be realized by an optical circuit using optical logic elements. The processor 31 may also include an arithmetic function based on quantum computing.

[0015] The processor 31 performs arithmetic processing based on data and software (programs) input from each device, etc., configured internally of the computer 3, and can output arithmetic results and control signals to each device, etc. The processor 31 may control each component constituting the computer 3 by executing the OS (Operating System) of the computer 3, applications, etc.

[0016] Various functions in the embodiment may be realized by one or more processors 31. Here, the processor 31 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other by wire or wirelessly.

[0017] The main memory 33 is a memory device that stores instructions executed by the processor 31 and various data, and information stored in the main memory 33 may be read by the processor 31. The auxiliary memory 35 is a memory device other than the main memory 33. These memory devices refer to any electronic components capable of storing electronic information, and may be semiconductor memories. The semiconductor memories may be either volatile memories or non-volatile memories. A memory device for saving various data used in various functions described later in the embodiments may be realized by the main memory 33 or the auxiliary memory 35, or may be realized by an internal memory built into the processor 31. For example, the memory unit in the embodiments is realized as the main memory 33 or the auxiliary memory 35.

[0018] A plurality of processors may be connected (coupled) to one storage device (memory), or a single processor 31 may be connected. A plurality of storage devices (memories) may be connected (coupled) to one processor. This configuration may also be realized by storage devices (memories) and processors included in a plurality of computers. Furthermore, a configuration in which a storage device (memory) is integrated with the processor 31 (for example, a cache memory including an L1 cache and an L2 cache) may also be included.

[0019] The network interface 37 is an interface for connecting to the communication network 5 wirelessly or by wire. The network interface 37 may be one that conforms to an existing communication standard. The network interface 37 may exchange information with the external device 7 and the terminal 9 connected via the communication network 5.

[0020] The external device 7 includes, for example, an output destination device, an external sensor, or an input source device. The external device 7 may include an external storage device (memory), for example, a network storage. The external device 7 may also be a device having some of the functions of the components of the various devices in the embodiment. The computer 3 may receive some or all of the processing results via the communication network 5 like a cloud service, or may transmit them to the outside of the computer 3.

[0021] The device interface 39 may be an interface related to a terminal that directly or indirectly connects an output device such as the display device 211 to the input device 213. The device interface 39 may have a connection terminal such as a USB. An external storage medium or a storage device (memory) may be connected to the device interface 39 via the connection terminal. The output device may have a speaker that outputs sound and the like.

[0022] The display device 211 displays, for example, the position information of the moving object 2, the name of a task to be executed by the moving object 2 for an object placed in the environment, the processing status of the task, etc. If the moving object 2 is a cleaning robot, the task corresponds to, for example, a task of cleaning up objects placed or scattered in a room in a home environment. Examples of the display device 211 include, but are not limited to, an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), an organic EL (Electro Luminescence) panel, etc.

[0023] The input device 213 includes devices such as a keyboard, a mouse, a touch panel, or a microphone, and provides information input by these devices to the computer 3. The input device 213 inputs settings related to a task, such as a task to be executed by the moving object 2 and a start time of the task. The input device 213 may also input the start of an operation (hereinafter referred to as an environment recognition operation) before the execution of a task, in which the moving object 2 recognizes an environment in which the task is not required (hereinafter referred to as an initial environment). If the moving object 2 is a tidying robot, the initial environment corresponds to a home environment in which tidying is not required. In this case, the environment recognition operation may correspond to an operation in which the moving object 2 recognizes the home environment in which tidying is not required as the initial environment.

[0024] The moving device 23 may be coupled to the lower part of the robot body 29 so as to support the robot body 29, for example. The moving device 23 may have a plurality of wheels 231 and a motor for driving the wheels 231. The moving device 23 drives the motor so as to execute a task, for example, under the control of the processor 31. The wheels 231 are rotated by driving the motor. As a result, the moving body 2 moves to a position where the task can be executed. Furthermore, if no instruction has been input by the user, the moving device 23 drives the motor, for example, so that the moving body 2 returns to the charging station. The moving device 23 may drive the motor to capture an image of a home environment, under the control of the processor 31 in response to an input of an environment recognition operation. As a result, the moving body 2 moves freely in, for example, a home environment.

[0025] Note that the blade 22 capable of pushing out an object may be connected to the front side of the moving device 23. In this case, the moving device 23 may support the blade 22 so that it can move in the vertical direction or the like. In this case, the moving device 23 further has a motor that moves the blade 22 in the vertical direction. The movement of the blade 22 in the vertical direction is realized by driving the motor under the control of the processor 31 based on the task.

[0026] The gripping device 25 has, for example, a gripping unit (also called an end effector) 251 that grips an object, a plurality of links (also called arms) 253 that are connected via a plurality of joints, and a motor that drives each of the plurality of joints. One end of one of the plurality of links 253 is rotatably connected, for example, to the front side of the robot body 29 so as to protrude forward of the robot body 29. In addition, one end of one of the plurality of links 253 is rotatably connected, for example, to the gripping unit 251. The gripping unit 251 has, for example, a hand with a bifurcated tip and a motor that drives the hand. The gripping unit 251 grips the object by, for example, pinching the object. The gripping unit 251 may have a mechanism that adsorbs the object by suction. The gripping unit 251 may be used in the operation of pushing out an object instead of the blade 22.

[0027] The gripping device 25 executes, for example, an operation of gripping an object (hereinafter referred to as a gripping operation), an operation of releasing the gripped object (hereinafter referred to as a releasing operation), and an operation of moving the gripped object (hereinafter referred to as a moving operation) under the control of the processor 31 based on the task. For example, the processor 31 may read out the motion trajectories of the link 253 and the gripping unit 251 related to these operations from the main storage device 33 prior to execution of the gripping operation, the releasing operation, and the moving operation. The gripping device 25 may drive the motors in the multiple joints and the motor in the gripping unit 251 under the control of the processor 31 based on the motion trajectory. In this way, the gripping device 25 may execute the gripping operation, the releasing operation, and the moving operation.

[0028] The imaging device 27 is mounted, for example, on the upper surface side of the robot body 29 so as to be rotatable about at least one rotation axis. The imaging device 27 has a predetermined imaging range. The imaging device 27 of this embodiment may capture an image over a wider area than the imaging range by appropriately rotating about the rotation axis under the control of the processor 31. The imaging device 27 may generate an image by capturing an image.

[0029] The photographing device 27 is realized by, for example, an RGB-D camera having an RGB (Red Blue Green) camera and a three-dimensional measuring camera (hereinafter referred to as a D (Depth) camera). The photographing device 27 of this embodiment may generate an image having distance information and color information by photographing using the RGB-D camera. In FIG. 2, the photographing device 27 is mounted on the upper surface side of the robot body 29, but may be installed on the front side of the robot body 29, for example. In addition, the photographing device 27 is not limited to an RGB-D camera as long as it can photograph an environment in which an object is placed and generate information about the environment such as an image or a point cloud. The photographing device 27 may photograph the environment in which a task is performed under the control of the processor 31 based on the input of the environment recognition operation. The photographing device 27 may perform photographing at any location in the environment under the control of the processor 31 based on the task. The photographing device 27 may output the generated image to the processor 31.

[0030] 3 is a diagram showing an example of a functional block in the processor 31. The processor 31 of this embodiment may have an image processing unit 311, a determination unit 313, a shooting position determination unit 315, a generation unit 317, a transmission / reception unit 319, an addition unit 321, and a control unit 323 as functions realized by the processor 31. The functions realized by the image processing unit 311, the determination unit 313, the shooting position determination unit 315, the generation unit 317, the transmission / reception unit 319, the addition unit 321, and the control unit 323 are stored as programs in, for example, the main storage device 33 or the auxiliary storage device 35. The processor 31 realizes functions related to the image processing unit 311, the determination unit 313, the shooting position determination unit 315, the generation unit 317, the transmission / reception unit 319, the addition unit 321, and the control unit 323 by reading and executing the programs stored in, for example, the main storage device 33 or the auxiliary storage device 35.

[0031] The image processing unit 311 executes image recognition processing using, for example, a machine learning model for object recognition in an image, for example, Deep Neural Networks (hereinafter referred to as object recognition DNN). The object recognition DNN corresponds to, for example, a classifier that classifies objects in an environment. Specifically, the image processing unit 311 may recognize objects in an image generated by the photographing device 27 using the object recognition DNN. The object recognition DNN may be trained in advance using learning data for recognizing objects in an image. The image generated by the photographing device 27 may be input to the object recognition DNN. The object recognition DNN outputs, for example, a degree of match (probability) between an object in an image used when training the object recognition DNN and an object in an input image, a label of the object related to the degree of match, and a position of the object in the input image (i.e., in the environment). Hereinafter, for the sake of concrete explanation, the degree of match is expressed as a percentage (%). The label of the object is a label that identifies an object, and corresponds to, for example, the name of the object. The position of the object is indicated, for example, by a bounding box indicating an area including the object in the input image. Note that the position of the object is not limited to the bounding box, and may be coordinates in the input image.

[0032] Among the machine learning models related to object recognition, the object recognition DNN is, for example, a DNN model that performs instance segmentation (hereinafter referred to as an instance segmentation model), and is realized by R-CNN (Region-based CNN), Faster R-CNN, Mask R-CNN, etc. Note that the machine learning model related to object recognition, the object recognition DNN, is not limited to the instance segmentation model, R-CNN, Faster R-CNN, Mask R-CNN, and may be any machine learning model or DNN model as long as it can output the recognition result of the object for the input image. Note that the training method of the object recognition DNN is well known to those skilled in the art, so a detailed description will be omitted. The object recognition DNN may be learned in advance and may be stored in the main storage device 33 or the auxiliary storage device 35, etc.

[0033] The image processing unit 311 may obtain a recognition result of an object in the initial environment using an image generated in response to the environment recognition operation. For example, the image processing unit 311 obtains a recognition result of an object in the initial environment by inputting an image generated in response to the environment recognition operation to an object recognition DNN. Note that the acquisition of the recognition result of an object by the image processing unit 311 is not limited to a machine learning model such as a DNN. The object in the initial environment corresponds to an object that does not require a task. The object in the initial environment corresponds to, for example, a static object in a home environment, that is, a structure in a room, furniture, etc. The image processing unit 311 may generate an environment map representing the initial environment based on the recognition result of the object in the initial environment. The environment map is, for example, a map showing a home environment that does not require tidying up, which corresponds to a structure in a room, furniture, etc. The image processing unit 311 may store the environment map in the main storage device 33 or the auxiliary storage device 35.

[0034] The image processing unit 311 may input the image generated during task execution to the object recognition DNN to obtain the recognition result of the object during task execution. The image processing unit 311 may recognize the object that is the target of the task (hereinafter referred to as the task target object) by comparing the recognition result of the object during task execution with the environment map. Specifically, the image processing unit 311 may optimize the relative positional relationship between the recognition result of the object during task execution and the environment map, for example, an existing alignment process (also called a registration process), in comparing the recognition result of the object during task execution with the environment map, and then may identify the task target object by subtracting the environment map from the recognition result of the object during task execution. That is, the task target object may correspond to an object in the image generated during task execution, excluding the object indicating the environment map. The recognition result regarding the task target object may be, for example, a label, a degree of agreement, and a position regarding the object in the image generated during task execution.

[0035] The determination unit 313 may determine whether or not a process related to an object is appropriate based on information about an object placed in the environment. The determination by the determination unit 313 may be performed based on a threshold. The process is identification of an object by image recognition. Specifically, the determination unit 313 of this embodiment may read a threshold stored in advance in the main storage device 33 or the auxiliary storage device 35. The threshold is a value related to the degree of match (e.g., 90%) and is set in advance, for example. The determination unit 313, for example, compares the degree of match output from the object recognition DNN with the read threshold when a task is executed. When a plurality of task target objects are included in an image generated when a task is executed, the determination unit 313 may perform the comparison for each of the plurality of task target objects. If, for example, the degree of match exceeds a threshold by the comparison, the determination unit 313 determines that the task target object has been recognized, that is, that the process is appropriate. If the degree of match is equal to or less than the threshold, the determination unit 313 determines that the recognition of the task target object is not appropriate. Hereinafter, a task target object determined by the determination unit 313 to be inappropriate for recognition will be referred to as an inappropriate recognition object. At this time, data on the inappropriate recognition object may be stored, for example, in the main storage device 33 or the auxiliary storage device 35. In this embodiment, data on the inappropriate recognition object is, for example, data output from the object recognition DNN regarding the inappropriate recognition object, that is, data indicating the degree of match, label, and position.

[0036] The photographing position determination unit 315 determines the photographing position where the object is photographed in the environment. Specifically, the photographing position determination unit 315 of this embodiment acquires the position information of the moving object 2 when the object is photographed in the environment, for example, based on a GPS (Global Positioning System) signal or the like. The photographing position determination unit 315 determines the photographing position in the environment map, for example, based on the position information of the moving object 2, the environment map, and the result of the alignment. Note that the position information of the moving object 2 may be obtained by calculating the relative position of the moving object 2 in the environment map based on the result of the alignment by the image processing unit 311 and the image generated by the photographing device 27 and used for the alignment, in addition to the GPS signal.

[0037] If the process is determined to be inappropriate, the generating unit 317 may generate a label list including multiple label candidates for a request for a label to identify an object. If the determining unit 313 determines that the recognition of the task target object is inappropriate, the generating unit 317 reads out correspondence data stored in advance in, for example, the main storage device 33 or the auxiliary storage device 35. In this specification, the correspondence data is data in which multiple areas in the environment are associated with multiple labels. If the environment is a home environment, the multiple areas in the environment may correspond to, for example, the names of multiple rooms, or if the environment is a warehouse, may correspond to, for example, multiple sections according to object categories. In this embodiment, the correspondence data may be held as a correspondence table, and the correspondence table may be a table in which each of multiple areas is associated with a task target object that is likely to exist in the area (hereinafter referred to as an area-object correspondence table).

[0038] Fig. 4 is a diagram showing an example of the region-object correspondence table ROT. As shown in Fig. 4, for example, a living room, which is a name indicating a region, is associated with labels of task target objects indicating clothes, tableware, toys, etc. Also, a room name of a study room, which is an area, is associated with labels of task target objects indicating writing implements, clothes, toys, etc.

[0039] The generating unit 317 generates a label list in which one or more labels related to requests for labels to identify the unrecognizable object are arranged in a predetermined order, for example, in an order in which the user is recommended to respond with priority, based on, for example, the photographing position of the object and the matching degree of the unrecognizable object with the region-object correspondence table ROT. Specifically, the generating unit 317 of this embodiment may compare the region indicating the photographing position of the unrecognizable object with the region-object correspondence table ROT. Next, the generating unit 317 may specify multiple labels corresponding to the region indicating the photographing position of the unrecognizable object by the matching. Next, the generating unit 317 may compare multiple labels (hereinafter referred to as output labels) output from the object recognition DNN for the unrecognizable object with the specified multiple labels (hereinafter referred to as specified labels). By the comparison, the generating unit 317 may select a label (hereinafter referred to as a duplicate label) that overlaps between the specified label and the output label from the output label. Furthermore, the generating unit 317 of this embodiment may generate a label list by arranging multiple duplicate labels in order of the degree of matching using the degree of matching corresponding to the output label. The generating unit 317 may generate a label list according to, for example, the image capturing time of the recognition-inappropriate object. At this time, the correspondence between the image capturing time and the plurality of labels may be stored in advance as a correspondence table in the main storage device 33 or the auxiliary storage device 35.

[0040] When the determination unit 313 determines that the recognition of the task target object is inappropriate, the transmission / reception unit 319 may transmit a request for a label that identifies the inappropriate object and data related to the inappropriate object to the terminal 9. The data related to the inappropriate object is, for example, an image, a degree of match, a position, etc. related to the inappropriate object. The transmission / reception unit 319 may receive information such as a label specified by a user via the terminal 9 from the terminal 9. Specifically, the transmission / reception unit 319 may transmit to the terminal 9 a label list, data of the inappropriate object, and a request for a label that identifies the inappropriate object. In this case, the transmission / reception unit 319 may receive from the terminal 9 one piece of label information specified by the terminal 9 from the label list.

[0041] For example, when it is determined that processing regarding an object placed in the environment is inappropriate, the adding unit 321 adds information on a label designated by a user (label information) to data regarding the object. For example, the adding unit 321 adds label information designated by a user via the terminal 9 to data of the inappropriate object. The adding unit 321 may store the added label information in the main storage device 33 or the auxiliary storage device 35 as a label of the inappropriate object.

[0042] For example, in response to an input of an environment recognition operation, the control unit 323 controls the moving device 23, the photographing device 27, the image processing unit 311, etc. to photograph the environment. In this embodiment, the control unit 323 may control the moving device 23, the gripping device 25, the photographing device 27, the image processing unit 311, the determination unit 313, etc. to execute the task in response to an instruction to execute the task or a start time of the task. When the determination unit 313 determines that the recognition of the task target object is not appropriate, the control unit 323 may control the generation unit 317, the transmission / reception unit 319, the addition unit 321, etc.

[0043] The terminal 9 may be connected to the information processing device 21 in the mobile object 2 via the communication network 5. The terminal 9 is realized by, for example, a personal computer, a tablet terminal, or a smartphone. In the following, for the sake of specificity, the terminal 9 is described as being a smartphone. The terminal 9 receives a request for a label for identifying an inappropriate object and data of the inappropriate object from the transmission / reception unit 319, for example, by wireless communication via the network interface 37 and the communication network 5. The terminal 9 displays, for example, the request for the label and the data of the inappropriate object on its own display. The terminal 9 specifies a label corresponding to the inappropriate object from among a plurality of labels, for example, in response to a user's instruction. The terminal 9 may input a character string indicating a label in response to a user's instruction. The terminal 9 transmits, for example, label information corresponding to the inappropriate object specified by the user to the transmission / reception unit 319.

[0044] The terminal 9 may further receive a label list from the transceiver 319. At this time, the terminal 9 may display the label list on its display together with the label request and data of the misrecognition object. In this case, the terminal 9 may specify (select) a label corresponding to the misrecognition object from among the multiple labels shown in the label list, in response to a user instruction.

[0045] The configuration of the information processing system 1 has been described above. Hereinafter, a process of adding label information of a recognition inappropriate object during task execution in the information processing system 1 (hereinafter, referred to as a label addition process) will be described. Fig. 5 is a flowchart showing an example of a processing procedure in the label addition process. An environment map may be generated before the task is executed.

[0046] (Label addition processing) (Step S501) The control unit 323 may respond to an instruction to execute a task or a start time of a task to control the moving device 23 in the moving object 2. The moving object 2 may move due to this control.

[0047] (Step S502) The photographing device 27 may perform photographing in the environment, for example, under the control of the control unit 323. At this time, the photographing position determination unit 315 may determine the photographing position. The photographing device 27 may generate an image after performing photographing. The photographing device 27 may output the generated image to the processor 31. The image processing unit 311 may acquire a recognition result of the task target object by image recognition processing of the generated image.

[0048] (Step S503) The determination unit 313 of this embodiment may determine whether or not the task target object can be recognized by comparing the degree of agreement in the recognition result of the task target object with a threshold value. If the degree of agreement in the recognition result of the task target object is greater than the threshold value (Yes in step S503), the process of step S504 may be executed. If the degree of agreement in the recognition result of the task target object is equal to or less than the threshold value (No in step S503), the process of step S505 may be executed.

[0049] (Step S504) The control unit 323 may control the moving device 23 and the gripping device 25 to execute the task. For example, if the task is to clear an object in the environment, the control unit 323 controls the gripping device 25 to execute a gripping operation, a releasing operation, and a moving operation. These operations complete the task for the recognized task target object. The operation contents in steps S501 to S504 may be transmitted to the terminal 9 by the transmitting / receiving unit 319. The terminal 9 may sequentially display the operation contents on its own display.

[0050] 6 is a diagram showing an example of a user interface displayed on the terminal 9 with respect to the operation contents in steps S501 to S504. As shown in FIG. 6, the user can check the execution process of the task on his / her own terminal 9.

[0051] (Step S505) The main storage device 33 or the auxiliary storage device 35 may store data of the misrecognition object. The data of the misrecognition object may be, for example, an image of the misrecognition object in the image acquired in step S502, a label for the misrecognition object, a degree of agreement for the label, and a photographing position determined in step S502.

[0052] (Step S506) The generating unit 317 may generate the label list based on, for example, the photographing position of the misrecognition object, the correspondence table, and the degree of coincidence of the misrecognition object. The generated label list may be stored in the main storage device 33 or the auxiliary storage device 35.

[0053] (Step S507) The determination unit 313 determines whether or not the tidying up of the task target objects, excluding the recognition inappropriate objects, is completed over the entire area of ​​the environment map. If the task is not completed (No in step S507), the processes of steps S502 to S507 may be repeated. If the task is completed (Yes in step S507), the process of step S508 may be executed.

[0054] (Step S508) The transceiver unit 319 may transmit the generated label list together with data on the unrecognizable objects to the terminal 9 via the network interface 37 and the communication network 5.

[0055] (Step S509) The terminal 9 of this embodiment may receive a request for a label for identifying an unrecognizable object, a label list, and data of the unrecognizable object. Upon receiving these, the terminal 9 may display the label list, the request for the label, and data of the unrecognizable object on its own display.

[0056] Fig. 7 is a diagram showing an example of a user interface displayed on the terminal 9, which includes a label list LL, a label request (decision button), and data on the misrecognized object (degree of match). The label list LL is displayed, for example, in a pull-down format. The degree of match may not be displayed in the label list LL. In this case, it may be possible to omit transmission of data on the degree of match to the terminal 9. As shown in Fig. 7, an input box may be displayed at the end of the label list LL, in which the label, i.e., the name of the task target object, can be arbitrarily input, for example, as a proper noun.

[0057] (Step S510) When the user selects one label from the label list or inputs required information by inputting the name of a label, etc., in the terminal 9, the terminal 9 may transmit the designated or input label information to the transmission / reception unit 319. The transmission / reception unit 319 may receive the label information transmitted from the terminal 9. The addition unit 321 may add the received label information to the data of the misrecognition object. The data of the misrecognition object to which the label has been added may be stored in the main storage device 33 or the auxiliary storage device 35 as a recognized task target object. With the above, the label addition process may end. Note that at least one of the processes in steps S508 to S510 may be executed between the processes in steps S506 and S507.

[0058] According to the information processing system 1 of the present embodiment, it may be determined whether or not a process related to an object (for example, a process for identifying an object by image recognition) is appropriate based on information about an object placed in an environment, and if it is determined that the process is inappropriate, label information designated by a user may be added to data related to the object. The determination may be made, for example, based on a threshold value. Specifically, the information processing system 1 according to the present embodiment may determine whether or not an object can be recognized in an image based on a result of an image recognition process for an image including an object placed in an environment, and if it is determined that the recognition of the object is inappropriate, label information designated by a user may be added to data related to the object. As a result, even if the recognition of the task target object is poor, label information for identifying the inappropriately recognized object can be added, for example, by free description, according to the needs of the user, without searching for the inappropriately recognized object. In other words, it is possible to provide the user with a user interface that allows personalization of the label of the inappropriately recognized object, and to improve operability regarding the execution of the task.

[0059] Furthermore, according to the information processing system 1 of this embodiment, a label list may be generated in which a plurality of labels related to a label request are arranged in order of recommendation to the user, based on a correspondence table that associates a plurality of regions in the environment with a plurality of labels, the photographing position related to the object, and the result of the image recognition process, and the generated label list may be further transmitted to the terminal 9, the label list may be displayed on the terminal 9 together with data related to the object, or one piece of label information specified in the label list may be received from the terminal 9. This makes it possible to improve the operability and efficiency of label selection by the user in setting a label for identifying an object that is not suitable for recognition.

[0060] For these reasons, the information processing system 1 according to this embodiment can reduce the burden on the user in setting labels for identifying recognition-inappropriate objects, and can improve the processing efficiency of label setting.

[0061] (Application example) This application example involves, for example, determining whether a task can be performed on an object by a mobile body 2 capable of moving within an environment, and if it is determined that the task is impossible, transmitting to a terminal 9 a request for a task that can replace the task (hereinafter referred to as an alternative task) and data on the object for which the task is determined to be impossible, receiving from the terminal 9 an alternative task specified by a user, and adding the specified alternative task to the data.

[0062] Hereinafter, a process (hereinafter referred to as alternative task addition process) of adding an alternative task during task execution to data of a task target object (hereinafter referred to as a task-impossible object) for which a task is determined to be impossible in the information processing system 1 will be described. Fig. 8 is a flowchart showing an example of a processing procedure in the alternative task addition process. The alternative task addition process may be executed, for example, following the process of step S504 in Fig. 5.

[0063] (Step S801) The determination unit 313 may determine whether the task for the task target object by the moving body 2 is possible or not. That is, the determination unit 313 may determine whether the task for the task target object is successful or not. For example, the determination unit 313 determines that the task has failed (hereinafter, the task is impossible) when the gripping device 25 cannot grip the task target object (hereinafter, the task is impossible), when the gripped task target object cannot find a destination position indicating a preset moving destination (hereinafter, the task is unknown), when the task target object cannot be gripped and the destination position is not reached (hereinafter, the task is impossible). When the task for the task target object is successful (Yes in step S801), the process of step S507 may be executed. When it is determined that the task is impossible (No in step S801), the process of step S802 may be executed.

[0064] (Step S802) The main storage device 33 or the auxiliary storage device 35 may store data of the task-impossible object. At this time, the main storage device 33 or the auxiliary storage device 35 may store the cause of the task-impossible object, i.e., the inability to grasp, the unknown destination position, the inability to reach, etc., in association with the data regarding the task-impossible object.

[0065] (Step S803) The generating unit 317 of this embodiment may generate a task list indicating a plurality of alternative tasks related to the request for an alternative task based on the label of the task-impossible object and the cause of the task-impossible object. Specifically, when the task-impossible object is lighter and shorter than the moving body 2 and the cause of the task-impossible object is that it cannot be grasped, the generating unit 317 generates, as an alternative task, moving the task-impossible object to a destination position using the blade 22 (hereinafter referred to as object sliding). In addition, when the task-impossible object can be grasped by the moving body 2 and the cause of the task-impossible object is that the destination position is unknown or cannot be reached, the generating unit 317 generates, as an alternative task, resetting the destination position to which the task-impossible object is to be moved and performing the task again (hereinafter referred to as destination resetting). The location related to the destination resetting may be set according to the additional information related to the task-impossible object (for example, the user's name, the position, the photographing time, etc.). In this embodiment, the generating unit 317 may generate a task list by compiling the generated alternative task and a plurality of alternative tasks set in advance into a list. The multiple pre-set alternative tasks include, for example, leaving the task-ineligible object alone, notifying the user of the task-ineligible object when the user arrives in the environment, marking the task-ineligible object, and the like.

[0066] (Step S804) If the task has been completed (Yes in step S507), the transmitting / receiving unit 319 may transmit a request for an alternative task and data on the task-unavailable object to the terminal 9. More specifically, the transmitting / receiving unit 319 may transmit the generated task list to the terminal 9 together with the request for an alternative task and data on the task-unavailable object.

[0067] (Step S805) The terminal 9 may receive the request for the alternative task and data on the task-impossible object from the transceiver 319 by wireless communication via the network interface 37 and the communication network 5. The terminal 9 may display the request for the alternative task and the data on the task-impossible object on its own display. When the terminal 9 receives a task list from the transceiver 319, the terminal 9 may display the task list on its own display together with the request for the alternative task and the data on the task-impossible object.

[0068] (Step S806) When the user designates (selects) one alternative task from the task list on the terminal 9, the terminal 9 may transmit the designated alternative task to the transmission / reception unit 319. The transmission / reception unit 319 may receive the alternative task transmitted from the terminal 9. For example, in designating the alternative task, the terminal 9 may reset a destination position to which the task-unavailable object is slid by the object slide. The addition unit 321 may add the received alternative task to the data of the task-unavailable object. The data of the task-unavailable object to which the alternative task has been added may be stored in the main storage device 33 or the auxiliary storage device 35. With the above, the alternative task addition process may end. Note that at least one process in steps S804 to S806 may be executed between the processes in steps S804 and S507.

[0069] After this step, the control unit 323 may execute an alternative task for the task-impossible object. For example, when object sliding is specified as the alternative task, the control unit 323 controls the moving body 2 to move the task-impossible object associated with the object sliding to a target position using the blade 22.

[0070] 9 is a diagram showing an example of displaying a list of notifications during task execution, a list of task target objects related to completed tasks, and an image TNG of a task-impossible object in a user interface displayed on the terminal 9. As shown in FIG. 9, in the user interface of the terminal 9 of this embodiment, the image TNG of the task-impossible object may be accompanied by a mark that calls the user's attention. For example, when the image TNG of the task-impossible object is clicked, a request for an alternative task and data of the task-impossible object TNG are displayed on the screen of the terminal 9. Note that the label, i.e., the name, of the automatically registered task target object may be editable as appropriate by a user's instruction via the terminal 9.

[0071] 10 is a diagram showing an example of a user interface displaying a task list TL, a request for an alternative task, and data on a task-impossible object on a terminal 9. The task list TL is displayed, for example, in a pull-down format. As shown in FIG. 10, a sock, which is a task-impossible object, is small compared to the moving object 2, so that an alternative task that is considered appropriate, in this example, the alternative task "pull together" corresponding to object sliding, may be preferentially displayed in the task list TL.

[0072] 11 is a diagram showing an example of a user interface display on the terminal 9, in which a task list TL, a request for an alternative task, and data on an object that is impossible to perform a task are displayed. The task list TL is displayed, for example, in a pull-down format. As shown in FIG. 11, since the cause of the task being impossible in this example is an unknown destination position, "reset destination" may be displayed in the task list TL.

[0073] Fig. 12 is a diagram showing an example of a user interface display for resetting a destination when "Reset destination" is selected in the task list shown in Fig. 11. As shown in Fig. 12, the environment map EM of the terminal 9 of this embodiment may display the location TP of the task-impossible object and a plurality of candidate CPs for the cleanup destination together with the current destination location PP. The user can easily reset the destination of the task-impossible object by touching the environment map EM in the user interface.

[0074] According to the information processing system 1 according to the application example of the present embodiment, it is further possible to determine whether or not a task can be performed on an object by a mobile object 2 that can move in the environment, and if it is determined that the task is impossible, an alternative task designated by a user may be added to data related to the object. In this way, even if the task on the task target object is not successful, an alternative task for the task-impossible object can be added without searching for the task-impossible object. In other words, it is possible to provide the user with a user interface that allows personalization of the alternative task for the task-impossible object, thereby improving operability regarding the execution of the task.

[0075] Furthermore, according to the information processing system 1 according to the application example of this embodiment, when it is determined that a task is impossible, a task list indicating a plurality of alternative tasks related to an alternative task request may be generated based on the label of the object and the cause of the task being impossible, the request for an alternative task to substitute for the task, data related to the object, and the task list may be transmitted to the terminal 9, the task list transmitted from the transmission / reception unit 319 to the terminal 9 may be displayed on the terminal 9 together with the data of the object, and one alternative task specified in the task list may be received from the terminal 9. This makes it possible to improve operability and efficiency in the selection of an alternative task by a user when setting an alternative task for a task-impossible object.

[0076] For these reasons, according to the information processing system 1 relating to an application example of this embodiment, it is possible to reduce the burden on the user regarding setting alternative tasks for task-impossible objects, and to improve the processing efficiency of setting alternative tasks.

[0077] From the above, according to the present disclosure, information according to the needs of the user can be added to data related to the object. That is, according to the present disclosure, information on the environment recognized by the user can be added to the environmental map and the task target object held by the information processing system 1, and information on the real world recognized by the user can be linked (personalized) to data held by the information processing system 1. More specifically, when a task related to an object is determined to be impossible, the user can transmit information that makes the task possible, make the task executable, and add information to the object. As a result, according to the present information processing system 1, for example, the operability and processing efficiency of the moving object 2 can be improved.

[0078] (Modification) In this modification, even if the degree of match exceeds a threshold or the threshold is set low, the process may be determined to be inappropriate in response to receipt of label information specified by the user, and the received label information may be added to data related to the object. The determination by the determination unit 313 in this modification may be made based on information transmitted by the user. The process in this modification may be executed arbitrarily, for example, after Yes in step S503 in FIG. 5. In this modification, the determination by the determination unit 313 may be executed based on receipt of a user instruction from the terminal 9, i.e., receipt of label information.

[0079] The transmitting / receiving unit 319 may transmit information (hereinafter, referred to as notification information) to the terminal 9 to notify the user of the label associated with the task target object based on the recognition result. The notification information is information that prompts the user to confirm or judge the recognition result, for example, the label associated with the task target object. Specifically, the notification information may be the label associated with the task target object, an image of the task target object, or data that prompts the user to confirm. The transmitting / receiving unit 319 may receive label information designated by the user at the terminal 9 as the user's response to the notification information transmitted to the terminal 9. Note that the transmitting / receiving unit 319 may transmit information on the label associated with the task target object to the user in response to a request from the user. That is, the user may obtain the label information by determining that a recognition inappropriateness may occur from, for example, the behavior of the moving object 2 without relying on the notification information from the system. In addition, such notification information may be transmitted to the moving object 2 other than the terminal 9. For example, the user may be notified as voice data (for example, a tweet such as "That's a cat") indicating that the moving object 2 recognizes the task target object.

[0080] The terminal 9 may provide the user with notification information. Specifically, the terminal 9 may output a voice based on the voice data, display a list of images and labels of task target objects, and the like. For example, the terminal 9 may display the list in a user interface (hereinafter referred to as a confirmation UI) for allowing the user to confirm whether the labels of the automatically registered task target objects and the classification of the labels differ from the user's recognition. In this way, the terminal 9 may provide the user with confirmation of the labels of the automatically registered task target objects, and the like. When a user's instruction such as tagging or editing is input to the confirmation UI, the terminal 9 may transmit information regarding the input (hereinafter referred to as confirmation result information) to the transmission / reception unit 319. In addition, the terminal 9 may transmit label information designated by the user to the transmission / reception unit 319 as a response to the notification information.

[0081] The determination unit 313 may determine that the recognition process is inappropriate in response to receiving the label information via the transmission / reception unit 319. That is, the determination unit 313 may determine whether or not the process related to the task target object, i.e., the recognition result of the task target object, is appropriate based on information about objects arranged in the environment, in response to an instruction from the user via the terminal 9.

[0082] If the process is determined to be inappropriate, the adding unit 321 may add the label information designated by the user to the data related to the task target object. The adding unit 321 may add the confirmation result information to the data of the corresponding task target object.

[0083] According to the information processing system 1 according to the modified example of the present embodiment, information for notifying the user of the label corresponding to the object may be transmitted to the terminal 9, label information designated by the user may be received from the terminal 9, and in response to the reception of the label information, it may be determined that the processing is inappropriate. That is, the determination by the determination unit 313 may be made based on the information transmitted by the user. As a result, even if the degree of match exceeds the threshold or the threshold is set low, the label information designated by the user can be added to the data related to the object. That is, even if the degree of match exceeds the threshold in the recognition result of the object, the recognition result is provided to the user, so that the user can determine whether the recognition result is a false positive. From the above, according to the information processing system 1 according to the modified example, the label to be added to the task target object can be personalized according to the user's desire, regardless of the recognition result.

[0084] When the technical idea of ​​the embodiment is realized by a method, the information processing method may determine whether or not processing related to an object is appropriate based on information about the object placed in the environment, and may add label information designated by a user to data related to the object if the processing is determined to be inappropriate, as shown in Fig. 5. The processing procedure, processing contents, effects, etc. of this information processing method are similar to those of the embodiment, and therefore description thereof will be omitted.

[0085] When the technical idea according to the embodiment is realized by a program, the information processing program may cause the computer 3 to determine whether or not a process related to an object placed in an environment is appropriate based on information about the object, and may add label information designated by a user to data related to the object if the process is determined to be inappropriate. Note that in this information processing program, a part of the process or the entire process may be processed by one or more computers provided on a cloud, and the process results may be transmitted to the mobile object 2 and the terminal 9. The process procedure, process contents, effects, etc. of the information processing program are the same as those of the embodiment, and therefore description thereof will be omitted.

[0086] When the technical idea according to the embodiment is realized in a robot system, the robot system includes a robot 2 having a determination unit 313 that determines whether or not a process related to an object is appropriate based on information about the object placed in the environment, and a terminal 9 that transmits label information designated by a user to the robot 2 when the process is determined to be inappropriate, and the robot 2 further includes an addition unit 321 that adds the label information to data related to the object. The processing procedures, processing contents, effects, etc. in the robot system are similar to those of the embodiment, and therefore description thereof will be omitted.

[0087] A part or all of each device in the above-mentioned embodiment may be configured with hardware, or may be configured with information processing of software (programs) executed by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). In the case of being configured with information processing of software, software for realizing at least a part of the functions of each device in the above-mentioned embodiment may be stored in a non-transient storage medium (non-transient computer-readable medium) such as a flexible disk, a CD-ROM (Compact Disc-Read Only Memory), or a USB (Universal Serial Bus) memory, and the software may be read into the computer 3 to execute the information processing. The software may also be downloaded via the communication network 5. Furthermore, the software may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), so that the information processing is executed by the hardware.

[0088] There is no limitation on the type of storage medium that stores the software. The storage medium is not limited to removable media such as magnetic disks or optical disks, but may be fixed storage media such as hard disks or memory. The storage medium may be provided inside the computer or outside the computer.

[0089] In this specification (including the claims), the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) includes any of a, b, c, ab, ac, bc, or abc. Also, it may include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of other elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0090] In this specification (including claims), expressions such as "using data as input / based on / according to / in response to data" (including similar expressions) include cases where various data itself is used as input, and cases where various data that have been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) are used as input, unless otherwise specified. In addition, when it is stated that a result is obtained "based on / according to / in response to data," it includes cases where the result is obtained based only on the data, and may also include cases where the result is obtained by being influenced by other data, factors, conditions, and / or states other than the data. In addition, when it is stated that "data is output," it includes cases where various data itself is used as output, and cases where various data that have been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) are output, unless otherwise specified.

[0091] In this specification (including the claims), the terms "connected" and "coupled" are intended as open-ended terms including any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, physically connection / coupling, etc. The terms should be interpreted appropriately depending on the context in which the terms are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in the terms without any restriction.

[0092] In this specification (including the claims), the expression "A configured to B" may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured / set to actually perform operation B. For example, when element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, when element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0093] In this specification (including the claims), terms implying containing or possessing (e.g., "comprising / including" and "having") are intended to be open-ended terms that include cases in which the term contains or possesses things other than the object indicated by the object of the term. When the object of such terms implies no quantity or a singular number (such as an article "a" or "an"), the expression should be construed as not being limited to a specific number.

[0094] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or suggest a singular number (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles a or an) should be interpreted as not necessarily being limited to a specific number.

[0095] In this specification, when a particular advantage / result is described as being obtained from a particular configuration of an embodiment, it should be understood that the same effect can also be obtained from one or more other embodiments having the same configuration, unless there is a reason to the contrary. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, and that the effect is not necessarily obtained by the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states are satisfied, and the effect is not necessarily obtained in the invention according to the claim that specifies the configuration or a similar configuration.

[0096] In this specification (including the claims), the term "maximize" and the like include finding a global maximum, finding an approximation of a global maximum, finding a local maximum, and finding an approximation of a local maximum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations of these maxima probabilistically or heuristically. Similarly, the term "minimize" and the like include finding a global minimum, finding an approximation of a global minimum, finding a local minimum, and finding an approximation of a local minimum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations of these minima probabilistically or heuristically. Similarly, the term "optimize" and the like include finding a global optimum, finding an approximation of a global optimum, finding a local optimum, and finding an approximation of a local optimum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations of these optimums probabilistically or heuristically.

[0097] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the above-mentioned individual embodiments. Various additions, modifications, replacements, partial deletions, etc. are possible within the scope of the conceptual idea and intent of the present invention derived from the contents defined in the claims and their equivalents. For example, in all the above-mentioned embodiments, when numerical values ​​or formulas are used in the explanation, they are shown as examples and are not limited to these. In addition, the order of each operation in the embodiments is shown as an example and is not limited to these.

Claims

1. Recognizing objects in an environment by a mobile object capable of moving in the environment; displaying a plurality of label candidates relating to the object on a terminal different from the mobile unit based on the object recognition result; determining a label to be added to the object from among the plurality of label candidates based on an instruction from a user via the terminal; An information processing method comprising:

2. generating an environment map of the environment prior to recognition of the object; The information processing method according to claim 1 .

3. The object recognition is performed by an imaging device that can rotate about a rotation axis.

3. The information processing method according to claim 1.

4. The mobile object can return to a charging station. The information processing method according to any one of claims 1 to 3.

5. The moving body is driven by a plurality of wheels. The information processing method according to any one of claims 1 to 4.

6. The moving body is driven by a motor. The information processing method according to any one of claims 1 to 5.

7. determining whether an action associated with the object in the environment is appropriate based on information about the object; If the process is determined to be inappropriate, a plurality of label candidates are transmitted to the terminal; determining a label to be added to the object from among the plurality of label candidates based on an instruction from a user via the terminal; An information processing method comprising:

8. the processing is recognition of objects in the environment; The information processing method according to claim 7.

9. the terminal is in a different location from the environment; The information processing method according to any one of claims 1 to 8.

10. The terminal is a smartphone or a tablet terminal. The information processing method according to any one of claims 1 to 9.

11. The object is displayed on the terminal by a bounding box. The information processing method according to any one of claims 1 to 10.

12. At least a part of information related to the information displayed on the terminal is transmitted to the cloud. The information processing method according to any one of claims 1 to 11.

13. A moving body capable of moving in an environment, the moving body comprising: Recognizing objects in the environment; transmitting a plurality of label candidates relating to the object to a terminal different from the mobile unit based on the object recognition result; receiving a label selected from the plurality of label candidates from the terminal; Mobile body.

14. determining whether an action associated with the object is appropriate based on information about the object in the environment; If the process is determined to be inappropriate, a plurality of label candidates are transmitted to the terminal; receiving a label selected from the plurality of candidate labels; A robot comprising:

Images