Information processing device, information processing method, and program

Abstract

[Purpose] To achieve more effective learning related to pattern recognition. [Solution] An information processing device is provided, which is provided with an operation control unit, which controls the operation of an autonomous mobile body that acts based on recognition processing. The operation control unit enables the autonomous mobile body to obtain information about a learning object learned in association with a taught label based on the detection of the start of teaching related to pattern recognition learning. An information processing method is also provided, the method includes a processor controlling the operation of an autonomous mobile body that acts based on recognition processing, wherein the control operation also includes enabling the autonomous mobile body to obtain information about a learning object learned according to the taught label based on the detection of the start of teaching related to pattern recognition learning.

Description

Technical Field

[0001] The present application relates to an information processing device, an information processing method and a program. Background Art

[0002] In recent years, various devices equipped with learning functions have been developed. Such devices include autonomous mobile objects, for example, robots that perform autonomous operations based on estimated situations. In addition, many technologies have been proposed to further diversify the available actions of such autonomous mobile objects. For example, in Patent Document 1, a technology is disclosed in which a new action mode is created based on the characteristics of other home information devices, and the characteristics of the autonomous mobile object are changed accordingly.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Laid-Open Patent Publication No. 2003-140710 Summary of the invention

[0006] Technical issues

[0007] Meanwhile, it can be considered that the actions of an autonomous mobile body generally become more diverse in proportion to the number of objects that the autonomous mobile body can recognize. However, in order to significantly increase the number of objects, a large amount of learning data related to the objects needs to be prepared.

[0008] In this regard, in the present application, a new and improved form of an information processing apparatus, an information processing method, and a program capable of performing learning related to pattern recognition with higher efficiency are proposed.

[0009] Solution to the problem

[0010] According to the present application, an information processing device is provided, including an operation control unit, which controls the operation of an autonomous mobile body that performs an action according to a recognition operation. Based on detecting the start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile body to obtain information about a learning object learned in a manner corresponding to the taught label.

[0011] In addition, according to the present application, an information processing method implemented in a processor is provided, including: controlling the operation of an autonomous mobile body that performs an action according to a recognition operation. Based on detecting the start of teaching related to pattern recognition learning, the control operation includes instructing the autonomous mobile body to obtain information about a learning object learned in a manner corresponding to the taught label.

[0012] In addition, according to the present application, there is provided a program for causing a computer to function as an information processing device, the information processing device including an operation control unit that controls the operation of an autonomous mobile body that performs an action according to a recognition operation. Based on detection of the start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile body to obtain information about a learning object learned in a manner corresponding to a taught label.

[0013] Advantageous Effects of the Invention

[0014] As described above, according to the present application, learning related to pattern recognition can be performed more efficiently.

[0015] Meanwhile, the above-described effects are not necessarily limited in scope, and any other effects indicated in this written description or any other effects that may appear in this written description may be achieved instead of or in addition to the above-described effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a diagram showing an exemplary hardware configuration of an autonomous mobile body according to a first embodiment of the present application;

[0017] Figure 2 is an exemplary configuration of an actuator installed in an autonomous mobile body according to the first embodiment;

[0018] Figure 3 is a diagram for explaining the operation of the actuator installed in the autonomous mobile body according to the first embodiment;

[0019] Figure 4 is a diagram for explaining the operation of the actuator installed in the autonomous mobile body according to the first embodiment;

[0020] Figure 5 is a diagram for explaining the function of a display installed in an autonomous mobile body according to the first embodiment;

[0021] Figure 6 is a diagram showing an exemplary action of the autonomous mobile body according to the first embodiment;

[0022] Figure 7 is a diagram showing an exemplary system configuration according to the first embodiment;

[0023] Figure 8 is a diagram showing an exemplary functional configuration of an autonomous mobile body according to the first embodiment;

[0024] Fig. 9 is a diagram showing an exemplary functional configuration of an information processing server according to the first embodiment;

[0025] Fig.10is a diagram showing an overview of pattern recognition learning performed based on teaching according to the first embodiment;

[0026] Fig.11 is a diagram showing an overview of action recognition learning performed based on teaching according to the first embodiment;

[0027] Fig.12 is a diagram showing an overview of spatial region recognition learning performed based on teaching according to the first embodiment;

[0028] Fig.13 is a diagram for explaining photographing control performed for photographing a learning object according to the first embodiment;

[0029] Fig.14 is a diagram for explaining photographing control performed for photographing a learning object according to the first embodiment;

[0030] Fig.15 is a diagram for explaining teaching performed using a marker according to the first embodiment;

[0031] Fig.16 is a diagram for explaining teaching performed using wireless communication according to the first embodiment;

[0032] Fig.17 is a diagram for explaining teaching performed using inaudible sound according to the first embodiment;

[0033] Fig.18 is a diagram showing an example of a user interface according to a first embodiment;

[0034] Fig.19 is a diagram showing an example of a user interface according to a first embodiment;

[0035] Fig. 20 is a diagram for explaining a booting operation according to the first embodiment;

[0036] Fig.21 is a flowchart for explaining the flow of operations performed in the autonomous mobile body 10 according to the first embodiment;

[0037] Fig. 22 is a diagram showing an example of a user interface for implementing teaching using an environment map according to a second embodiment of the present application;

[0038] Fig.23 is a diagram showing an example of a generation process for generating an environment map according to the second embodiment;

[0039] Fig.24is a diagram showing an example of a generation process for generating an environment map according to the second embodiment;

[0040] Fig.25 is a diagram showing an example of a generation process for generating an environment map according to the second embodiment;

[0041] Fig.26 is a diagram showing an example of a generation process for generating an environment map according to the second embodiment;

[0042] Fig. 27 is a diagram showing an example of a generation process for generating an environment map according to the second embodiment;

[0043] Fig.28 is a diagram for explaining movement control and information acquisition performed for moving to a spatial area corresponding to a designated area according to a second embodiment;

[0044] Fig.29 is a flowchart for explaining an example of a flow of teaching-based learning using an environment map according to the second embodiment;

[0045] Fig.30 is a block diagram showing an exemplary hardware configuration of an information processing server according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present application are described in detail below with reference to the accompanying drawings. In this written description and the accompanying drawings, constituent elements having substantially the same functional configuration are referred to by the same reference numerals, and explanations are not repeated.

[0047] The explanation is given in the following order.

[0048] 1. First Implementation

[0049] 1.1. Overview of the Autonomous Moving Object 10

[0050] 1.2. Exemplary Hardware Configuration of Autonomous Moving Object 10

[0051] 1.3. Example System Configuration

[0052] 1.4. Exemplary Functional Configuration of Autonomous Moving Object 10

[0053] 1.5. Exemplary Functional Configuration of Information Processing Server 20

[0054] 1.6. Functional Details

[0055] 1.7. Operation process

[0056] 2. Second Implementation

[0057] 2.1. Overview

[0058] 2.2. Instructional details using environment maps

[0059] 2.3. Process of instruction-based learning using environment map EM

[0060] 3. Example Hardware Configuration

[0061] 4. Summary

[0062] <1. First Embodiment>

[0063] <<1.1. Overview of Autonomous Moving Object 10>>

[0064] First, an explanation is given about the overview of the autonomous mobile body 10 according to the first embodiment of the present application. The autonomous mobile body 10 according to the first embodiment of the present application is an information processing device that performs situation estimation based on collected sensor information, and autonomously selects and performs various operations according to the situation. Unlike a robot that performs operations only by following user commands, one feature of the autonomous mobile body 10 is that it autonomously performs operations that are considered to be most appropriate in each situation.

[0065] The autonomous mobile body 10 according to the first embodiment of the present application can perform, for example, user recognition or object recognition based on the captured image, and take various autonomous actions according to the recognized user or recognized object. In addition, for example, the autonomous mobile body 10 according to the first embodiment of the present application can perform voice recognition based on the user's voice, and can take actions according to instructions from the user.

[0066] At this time, in order to perform user recognition and object recognition, it is necessary to learn the characteristics of the user and the object in advance in a manner corresponding to the label. For example, as a result of performing teaching learning based on the provided correct-answer label, the autonomous mobile body 10 becomes able to recognize the object extracted from the image by associating the characteristics of the object with the label.

[0067] On the other hand, in order to realize such teacher learning, such learning data in which the correct-answer label of the object is associated with the image in advance is required. However, it is actually difficult for the developer to prepare in advance the learning data of all possible objects that the autonomous mobile body 10 may encounter on a daily basis and to instruct the autonomous mobile body 10 to learn all of the learning data. In addition, for example, even when pattern recognition learning of a large number of objects is performed in advance, if an object that has never existed before becomes popular or an object with a design different from that of an existing object becomes popular, it is difficult to cope with such an object using only the knowledge that has been learned.

[0068] The technical idea according to the present application is conceived by focusing on the problems explained above, and pattern recognition learning can be performed with higher efficiency. In this regard, a feature of the autonomous mobile body 10 according to the first embodiment of the present application is to automatically collect learning data based on teaching and perform pattern recognition learning about objects; therefore, even if learning data is not prepared in advance, the ability to recognize various events can be obtained.

[0069] For example, the autonomous mobile body 10 according to the first embodiment is capable of automatically collecting learning data based on the user's teaching. For example, the autonomous mobile body 10 can recognize an object indicated by the user as a learning object by performing a gesture such as pointing with a finger, and can extract a label from the sound subsequently emitted by the user. In addition, the autonomous mobile body 10 according to the first embodiment can automatically photograph the learning object learned in a manner corresponding to the taught label.

[0070] As a result of being equipped with the above-mentioned functions, the autonomous mobile body 10 according to the first embodiment can automatically generate learning data related to pattern recognition learning of various events based on teaching; and can acquire recognition capabilities for events that may be newly observed on a daily basis.

[0071] A detailed explanation of the configuration of the autonomous mobile body 10 having the above-mentioned features is given below. The autonomous mobile body 10 according to the first embodiment of the present application may be an autonomous mobile robot that moves autonomously in space and performs various operations. For example, the autonomous mobile body 10 may be an autonomous mobile robot having a shape and movement capability that imitates a human or an animal (e.g., a dog). Alternatively, for example, the autonomous mobile body 10 may be a vehicle or some other device having the capability to communicate with a user. Therefore, with respect to the autonomous mobile body 10 according to the first embodiment, the shape, capabilities, and requirements may be appropriately designed according to the objectives and roles of the autonomous mobile body 10.

[0072] <<1.2. Exemplary Hardware Configuration of Autonomous Moving Object 10>>

[0073] An explanation is given below of an exemplary hardware configuration of the autonomous mobile body 10 according to the first embodiment of the present application. An explanation is given below of an example in which the autonomous mobile body 10 is a quadruped robot imitating a dog.

[0074] Figure 1 1 is a diagram showing an exemplary hardware configuration of the autonomous mobile object 10 according to the first embodiment of the present application. Figure 1 As shown, the autonomous mobile body 10 is a dog-shaped quadruped robot having a head region, a trunk region, four leg regions, and a tail region. In addition, the autonomous mobile body 10 includes two displays 510 in the head region.

[0075] In addition, the autonomous mobile body 10 includes various sensors. For example, the autonomous mobile body 10 includes a microphone 515, a camera 520, a ToF (Time of Flight) sensor 525, a human presence sensor 530, a distance sensor 535, a touch sensor 540, an illumination sensor 545, a foot button 550, and an inertial sensor 555.

[0076] (Microphone 515)

[0077] The microphone 515 has a function of collecting surrounding sounds. Examples of sounds include the user's voice and surrounding environmental sounds. The autonomous mobile body 10 may include, for example, 4 microphones in the head area. As a result of having a plurality of microphones 515, sounds generated in the surrounding area can be collected with high sensitivity, and localization of the sound source can be performed.

[0078] (Camera 520)

[0079] The camera 520 has a function of photographing the user and the surrounding environment. For example, the autonomous mobile body 10 may include two wide-angle cameras, one at the tip of the nose, and one at the waist area. In this case, the wide-angle camera installed at the tip of the nose captures an image corresponding to the front field of view of the autonomous mobile body (i.e., corresponding to the dog's line of sight); and the wide-angle camera installed at the waist area captures an image of the surrounding area centered on the upward direction. For example, based on the image captured by the wide-angle camera installed at the waist area, the autonomous mobile body 10 can extract feature points of the ceiling and implement SLAM (Simultaneous Localization and Mapping).

[0080] (ToF sensor 525)

[0081] The ToF sensor 525 has a function of detecting the distance of an object located in front of the head area. The ToF sensor 525 is installed at the tip of the nose of the head area. As a result of using the ToF sensor 525, the distances of various objects can be detected with high accuracy, and operations can be performed according to the relative positions relative to the target object including the user and obstacles.

[0082] (Human presence sensor 530)

[0083] The human presence sensor 530 has a function of detecting the whereabouts of a user or a pet of the user. The human presence sensor 530 is installed, for example, in the chest area. As a result of using the human presence sensor 530, any animal subject existing in front can be detected, and various operations with respect to the animal subject can be performed, for example, operations according to feelings including interest, fear, and surprise can be performed.

[0084] (Distance sensor 535)

[0085] The distance measuring sensor 535 has a function of acquiring the floor condition in front of the autonomous mobile body 10. For example, the distance measuring sensor 535 is installed in the chest area. As a result of using the distance measuring sensor 535, the distance of any object existing on the floor in front of the autonomous mobile body 10 can be detected with high accuracy, and an operation can be performed according to the relative position with respect to the object.

[0086] (Touch sensor 540)

[0087] The touch sensor 540 has a function of detecting a user's touch. For example, the touch sensor 540 is installed at a body part that is likely to be touched by the user, such as the top of the head, the lower jaw, and the back of the autonomous mobile body 10. The touch sensor 540 may be, for example, a capacitive or pressure-sensitive touch sensor. As a result of using the touch sensor 540, a touch action performed by the user, such as touching, stroking, tapping, and pressing, may be detected, and an operation may be performed according to the touch action.

[0088] (Lighting sensor 545)

[0089] The illumination sensor 545 detects illumination of the space where the autonomous mobile body 10 is located. For example, the illumination sensor 545 may be installed at the base of the tail region on the back of the head region. As a result of using the illumination sensor 545, the surrounding brightness may be detected, and operations may be performed according to the brightness.

[0090] (Foot button 550)

[0091] The sole button 550 has a function of detecting whether the lower surface of the leg of the autonomous mobile body 10 contacts the floor. For this purpose, the sole button 550 is installed in the portion corresponding to the sole pad of the four legs. As a result of using the sole button 550, contact or non-contact between the autonomous mobile body 10 and the floor surface can be detected; and for example, it can be understood that the autonomous mobile body 10 has been lifted by the user.

[0092] (Inertial sensor 555)

[0093] The inertial sensor 555 is a 6-axis sensor for detecting physical quantities such as speed, acceleration, and rotation of the head region or the trunk region. That is, the inertial sensor 555 detects acceleration and angular velocity of the X-axis, Y-axis, and Z-axis. The inertial sensor 555 is installed in the head region and the trunk region. As a result of using the inertial sensor 555, the movement of the head region and the trunk region of the autonomous mobile body 10 can be detected with high accuracy, and operation control can be performed according to the situation.

[0094] So far, explanation has been given about an example of the sensor installed in the autonomous mobile body 10 according to the first embodiment of the present application. Figure 1 The explained configuration is only exemplary, and the configuration of the sensor that can be included in the autonomous mobile body 10 is not limited to this example. In addition to the configuration explained above, the autonomous mobile body 10 may also include a temperature sensor, a geomagnetic sensor, and various communication devices including a GNSS (Global Navigation Satellite System) signal receiver. Therefore, the configuration of the sensor installed in the autonomous mobile body 10 can be flexibly modified according to the specification and operation.

[0095] An explanation is given below of an exemplary configuration of a joint region of the autonomous mobile body 10 according to the embodiment. Figure 2 is an exemplary configuration of the actuator 570 installed in the autonomous mobile body 10 according to the first embodiment of the present application. The autonomous mobile body 10 according to the first embodiment of the present application has a total of 22 rotational degrees of freedom, including Figure 2 In addition to the rotating components shown, there are two rotational degrees of freedom in the ear area, two rotational degrees of freedom in the tail area and one rotational degree of freedom in the mouth.

[0096] For example, as a result of having three rotational degrees of freedom in the head area, the autonomous mobile body 10 can perform nodding and tilting the head to the side. In addition, as a result of installing the actuator 570 in the waist area, the autonomous mobile body 10 can reproduce the swinging motion of the waist. This enables natural and flexible movements that are more similar to those of a dog in real life.

[0097] Meanwhile, in the autonomous mobile body 10 according to the embodiment of the present application, for example, a combination of a uniaxial actuator and a biaxial actuator can be used to achieve 22 degrees of rotational freedom. For example, a uniaxial actuator can be used for the elbow and the knee of the leg area, and a biaxial actuator can be used for the shoulder and the base of the thigh.

[0098] Figure 3 and Figure 4 1 is a diagram for explaining the operation of the actuator 570 installed in the autonomous mobile body 10 according to the first embodiment of the present application. Figure 3 , in each actuator 570, as a result of rotating the output gear using the motor 575, the movable arm 590 can be driven at an arbitrary rotational position and at an arbitrary rotational speed.

[0099] refer to Figure 4 The actuator 570 according to the first embodiment of the present application includes a rear cover 571, a gear box cover 572, a control board 573, a gear box base 574, a motor 575, a first gear 576, a second gear 577, an output gear 578, a detection magnet 579 and two bearings 580.

[0100] The actuator 570 according to the first embodiment of the present application may be, for example, a magnetic svGMR (spin valve gain magnetoresistive). The control board 573 rotates the motor 575 under the control of the main processor, so that power can be transmitted to the output gear 578 via the first gear 576 and the second gear 577, and the movable arm 590 can be driven.

[0101] In addition, the position sensor installed in the control board 573 detects the rotation angle of the detection magnet 579 that rotates synchronously with the output gear 578; thereby, the rotation angle of the movable arm 590, that is, the rotation position of the movable arm 590 can be detected with high precision.

[0102] Meanwhile, since the magnetic svGMR is essentially contactless, it has excellent durability; and since it is used in the GMR saturation region, the magnetic svGMR is less affected by signal fluctuations due to changes in the distance to the detection magnet 579 or the position sensor.

[0103] So far, an explanation has been given about an exemplary configuration of the actuator 570 installed in the autonomous mobile body 10 according to the first embodiment of the present application. With this configuration, the bending motion of the joint area of ​​the autonomous mobile body 10 can be controlled with high precision, and the rotational position of the joint area can be accurately detected.

[0104] Reference below Figure 5 The function of the display 510 installed in the autonomous mobile object 10 according to the first embodiment of the present application is explained. Figure 5 1 is a diagram for explaining the function of the display 510 installed in the autonomous mobile object 10 according to the first embodiment of the present application.

[0105] (Display 510)

[0106] The display 510 has the function of visually expressing the eye movement and feeling of the autonomous moving object 10. Figure 5 As shown, display 510 can express the movement of eyeball, pupil and eyelid according to feeling and action.In addition, ensure that the image of character and symbol irrelevant to visual movement is not displayed in display 510, so as to create the natural movement similar to real animal (for example, dog).

[0107] like Figure 5As shown, the autonomous mobile body 10 includes two displays, that is, a display 510r corresponding to the right eye and a display 510l corresponding to the left eye. The displays 510r and 510l are implemented using, for example, two independent OLEDs (organic light emitting diodes). As a result of using OLEDs, the curved surface of the eyeball can be reproduced, and thus a more natural appearance can be obtained compared to the case where a pair of eyeballs is expressed using a single flat panel display or the case where two eyeballs are expressed using two independent flat panel displays.

[0108] As described above, as a result of using displays 510r and 510l, it is possible to express information such as Figure 5 The glance and feeling of the autonomous moving object 10 shown. In addition, based on the movement of the eyeball displayed on the display 510, the user becomes able to intuitively understand the state of the autonomous moving object 10.

[0109] So far, an explanation has been given about an exemplary hardware configuration of the autonomous mobile body 10 according to the first embodiment of the present application. With this hardware configuration, as Figure 6 As shown, the motions of the joint areas and eyeballs of the autonomous mobile body 10 can be controlled with high precision and flexibility, and thus motions and emotional expressions close to those of a real living object can be achieved. Figure 6 1 is a diagram showing an exemplary operation of the autonomous mobile object 10 according to the first embodiment of the present application. Figure 6 , focusing on explaining the joint areas and eye movements of the autonomous mobile body 10. For this reason, the external structure of the autonomous mobile body 10 is shown in a simplified form. In the same way, in the following explanation, the external structure of the autonomous mobile body 10 is sometimes shown in a simplified form. However, the hardware configuration and appearance of the autonomous mobile body 10 according to the first embodiment of the present application are not limited to the examples shown in the drawings, and can be designed as needed.

[0110] <<1.2. Exemplary Hardware Configuration of Autonomous Moving Object 10>>

[0111] An explanation is given below of an exemplary system configuration according to the first embodiment of the present application. Figure 7 is a diagram showing an exemplary system configuration according to a first embodiment of the present application. Figure 7 The information processing system according to the first embodiment of the present application includes an information processing server 20 and a plurality of autonomous mobile bodies 10 (including 10a and 10b). The autonomous mobile bodies 10 are connected to the information processing server 20 via a network 30 and are connected to each other in a mutually communicable manner.

[0112] (Autonomous moving object 10)

[0113] The autonomous mobile body 10 according to the first embodiment of the present application is an information processing device that performs situation estimation based on the collected sensor information, and autonomously selects and performs various operations according to the situation. As described above, the autonomous mobile body 10 according to the first embodiment of the present application can be an autonomous mobile robot that has the shape and movement ability to imitate a person or an animal (e.g., a dog). In addition, the autonomous mobile body 10 according to the first embodiment of the present application can automatically collect learning data based on teaching, and can perform pattern recognition learning based on the learning data.

[0114] (Information Processing Server 20)

[0115] The information processing server 20 according to the first embodiment of the present application is an information processing device that is connected to a plurality of autonomous mobile bodies 10 and has a function of collecting various information from the autonomous mobile bodies 10. For example, the information processing server 20 stores learning data collected by the plurality of autonomous mobile bodies 10. In addition, the information processing server 20 according to the first embodiment of the present application controls a user interface that enables a user to confirm a learning state of the corresponding autonomous mobile body 10.

[0116] (Network 30)

[0117] The network 30 has a function of connecting the autonomous mobile body 10 to the information processing server 20 and to each other. The network 30 may include a public line network, such as the Internet, a telephone line network, or a satellite communication network; various types of LANs (local area networks), such as Ethernet (registered trademark); and WANs (wide area networks). Alternatively, the network 30 may include a dedicated line network, such as an IP-VPN (Internet Protocol Virtual Private Network). Still alternatively, the network 30 may include a wireless communication network, such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0118] So far, explanations have been given about the exemplary system configuration according to the first embodiment of the present application. Figure 7 The explained configuration is only exemplary, and the configuration of the information processing system according to the first embodiment of the present application is not limited to this example. Alternatively, for example, in addition to transmitting information with the information processing server 20, the autonomous mobile body 10 may also transmit information with various external devices. Examples of external devices include servers that provide weather information, news, and other service information; various information processing terminals held by users; and home appliances. Therefore, the system configuration according to the first embodiment of the present application can be flexibly modified according to specifications and operations.

[0119] <<1.4. Exemplary Functional Configuration of Autonomous Moving Object 10>>

[0120] An explanation is given below of an exemplary functional configuration of the autonomous mobile object 10 according to the first embodiment of the present application. Figure 8 1 is a diagram showing an exemplary functional configuration of the autonomous mobile object 10 according to the first embodiment of the present application. Figure 8 According to the first embodiment of the present application, the autonomous mobile body 10 includes an input unit 110, a recognition unit 120, a learning unit 130, an action planning unit 140, an operation control unit 150, a driving unit 160, an output unit 170 and a server communication unit 180.

[0121] (Input unit 110)

[0122] The input unit 110 has the function of collecting various information related to the user and the surrounding environment. For example, the input unit 110 collects the user's voice and the surrounding environment sound, collects image information related to the user and the surrounding environment, and collects various sensor information. To this end, the input unit 110 includes Figure 1 Various sensors are shown.

[0123] (Identification Unit 120)

[0124] The recognition unit 120 has a function of performing various recognitions related to the user, the surrounding environment, and the state of the autonomous mobile body 10 based on various information collected by the input unit 110. As an example, the recognition unit 120 can perform human recognition, expression and glance recognition, object recognition, motion recognition, spatial region recognition, color recognition, shape recognition, mark recognition, obstacle recognition, level difference recognition, and brightness recognition.

[0125] For example, the recognition unit 120 according to the first embodiment of the present application may have a function of recognizing a learning object indicated by a user by performing a gesture such as pointing with a finger. In addition, for example, the recognition unit 120 according to the first embodiment of the present application may have a function of performing lexical analysis on a sound emitted after the user performs the above-mentioned gesture and extracting a noun phrase. The noun phrase can be used as a label in the pattern recognition learning performed by the learning unit 130.

[0126] In addition, the recognition unit 120 performs emotion recognition, word understanding, and sound source localization for the user's voice. In addition, the recognition unit 120 can recognize the user's touch, the surrounding temperature, the presence of an animal subject, and the posture of the autonomous mobile object 10.

[0127] In addition, the recognition unit 120 has a function of estimating and understanding the surrounding environment and the situation where the autonomous moving body 10 exists based on the above-mentioned recognized information. At this time, the recognition unit 120 can perform comprehensive situation estimation using pre-stored environmental knowledge.

[0128] (Study Unit 130)

[0129] The learning unit 130 has a function of learning an environment (situation), learning actions, and the impact of learning actions on the environment. The learning unit 130 uses, for example, a machine learning algorithm such as deep learning to perform the above learning. However, the learning algorithm implemented by the learning unit 130 is not limited to this example and can be designed as needed.

[0130] The learning unit 130 according to the first embodiment performs pattern recognition learning based on the labels collected by the recognition unit 120 according to the teaching and based on the image of the learning object photographed by the input unit 110 .

[0131] (Action planning unit 140)

[0132] The action planning unit 140 has a function of planning the action of the autonomous mobile object 10 based on the situation estimated by the recognition unit 120 and based on the knowledge learned by the learning unit 130 .

[0133] (Operation Control Unit 150)

[0134] The operation control unit 150 has a function of controlling the operation of the driving unit 160 and the output unit 170 based on the motion planning performed by the motion planning unit 140. For example, based on the motion planning, the operation control unit 150 performs rotation control of the actuator 570, display control of the display 510, and sound output control of the speaker.

[0135] For example, based on the fact that the recognition unit 120 has detected the start of teaching related to pattern recognition learning, the operation control unit 150 according to the first embodiment can instruct the input unit 110 to obtain information related to the learning object to be learned in a manner corresponding to the taught label.

[0136] More specifically, the operation control unit 150 according to the first embodiment controls the driving unit 160 in such a manner that the learning object recognized by the recognition unit 120 is in a photographable position and posture, and instructs the input unit 110 to photograph the learning object.

[0137] At this time, the operation control unit 150 according to the first embodiment can control the drive unit 160 to change the position and posture of the autonomous mobile body 10, and instruct the input unit 110 to photograph the learning object from multiple angles. Due to the function of the operation control unit 150 according to the first embodiment, for example, images of multiple side surfaces and top surfaces of the learning object can be collected and used as learning data, and pattern recognition learning can be performed more efficiently. Regarding the function of the operation control unit 150 according to the first embodiment, a detailed explanation will be given separately later.

[0138] (Drive unit 160)

[0139] The driving unit 160 has a function of bending a plurality of joint regions of the autonomous mobile body 10 under the control of the operation control unit 150. More specifically, under the control of the operation control unit 150, the driving unit 160 drives the actuators 570 installed in the joint regions.

[0140] (Output unit 170)

[0141] The output unit 170 has a function of outputting visual information and audio information under the control of the operation control unit 150. To this end, the output unit 170 includes a display 510 and a speaker.

[0142] (Server Communication Unit 180)

[0143] The server communication unit 180 has a function of transmitting information with the information processing server 20 and other autonomous mobile bodies 10. For example, the server communication unit 180 transmits the collected learning data to the information processing server 20. In addition, for example, the server communication unit 180 receives the learning data stored in the information processing server 20, and receives information related to the learning knowledge acquired based on the learning data.

[0144] So far, explanation has been given about the exemplary functional configuration of the autonomous mobile body 10 according to the first embodiment of the present application. Figure 8 The explained configuration is only exemplary, and the functional configuration of the autonomous mobile body 10 according to the first embodiment of the present application is not limited to this example. Therefore, the functional configuration of the autonomous mobile body 10 according to the first embodiment of the present application can be flexibly modified according to specifications and operations.

[0145] <<1.5. Exemplary Functional Configuration of Information Processing Server 20>>

[0146] An explanation is given below of an exemplary functional configuration of the information processing server 20 according to the first embodiment of the present application. Fig. 9 2 is a diagram showing an exemplary functional configuration of the information processing server 20 according to the first embodiment of the present application. Fig. 9 The information processing server 20 according to the first embodiment of the present application includes a recognition unit 210 , a learning unit 220 , a storage unit 230 , an application control unit 240 , and a terminal communication unit 250 .

[0147] (Identification Unit 210)

[0148] The recognition unit 210 according to the first embodiment may have a function equivalent to that of the recognition unit 120 of the autonomous mobile body 10. That is, the recognition unit 210 according to the first embodiment has a function of recognizing a learning object based on teaching and extracting a label. At the same time, the recognition unit 210 according to the first embodiment may recognize a learning object and extract a label based on information collected by a plurality of autonomous mobile bodies 10.

[0149] (Study Unit 220)

[0150] The learning unit 220 according to the first embodiment may have a function equivalent to that of the learning unit 130 of the autonomous mobile body 10. That is, the learning unit 220 according to the first embodiment performs pattern recognition learning based on the label obtained by the recognition unit 210 and the image related to the learning object. At the same time, the learning unit 220 according to the first embodiment may perform pattern recognition learning based on information collected by a plurality of autonomous mobile bodies. That is, the learning unit 220 according to the first embodiment serves as a collective knowledge body shared by a plurality of autonomous mobile bodies 10.

[0151] (Storage unit 230)

[0152] The storage unit 230 according to the first embodiment is used to store the collected learning data, the learning knowledge acquired by the learning unit 220 and the autonomous mobile body 10 , and the result of pattern recognition performed based on the learning knowledge.

[0153] (Application Control Unit 240)

[0154] The application control unit 240 according to the first embodiment controls a user interface related to state confirmation and function control of the autonomous mobile body 10. The application control unit 240 can display the user interface in an information processing terminal owned by the user. In the user interface, for example, the application control unit 240 provides a means for confirming the learning state of the corresponding autonomous mobile body 10 and a means for associating a label related to pattern recognition learning with a learning object.

[0155] Specifically, the application control unit 240 according to the first embodiment can provide a way for the user to correct the association between the label and the learning object learned by the corresponding autonomous mobile body 10. Regarding the function of the application control unit 240 according to the present embodiment, a detailed explanation will be given separately later. At the same time, the function of the application control unit 240 according to the present embodiment can be implemented as the function of the autonomous mobile body 10 instead. That is, the autonomous mobile body 10 according to the first embodiment has Figure 8 In addition to the configuration shown, an application control unit may also be included separately.

[0156] (Terminal Communication Unit 250)

[0157] The terminal communication unit 250 according to the first embodiment transmits information with the plurality of autonomous mobile bodies 10 via the network 30. For example, the terminal communication unit 250 according to the first embodiment receives learning data and pattern recognition results from the autonomous mobile bodies 10. In addition, the terminal communication unit 250 transmits the learning data collected by the plurality of autonomous mobile bodies 10 and the learning knowledge acquired based on the learning data to each autonomous mobile body 10.

[0158] So far, explanation has been given about the exemplary functional configuration of the information processing server 20 according to the first embodiment of the present application. Fig. 9 The explained configuration is only exemplary, and the functional configuration of the information processing server 20 according to the first embodiment is not limited to this example. For example, as described above, the information processing server 20 according to the first embodiment may have a function equivalent to that of the action planning unit 140 or the operation control unit 150 of the autonomous mobile body 10. In this case, the information processing server 20 according to the first embodiment may perform situation estimation based on sensor information collected by the input unit 110 of the autonomous mobile body 10, and may remotely control the driving unit 160 and the output unit 170. Therefore, the functional configuration of the information processing server 20 according to the first embodiment may be flexibly modified according to specifications and operations.

[0159] <<1.6. Functional Details>>

[0160] A detailed explanation of the functions of the autonomous mobile body 10 and the information processing server 20 according to the first embodiment of the present application is given below. In the following explanation, the explanation is mainly given for an example in which pattern recognition learning is performed by the learning unit 130 of the autonomous mobile body 10. However, alternatively, as described above, pattern recognition learning may be performed by the learning unit 220 of the information processing server 20, or may be performed by the learning unit 130 as well as the learning unit 220.

[0161] First, an explanation is given about the overview of pattern recognition learning performed based on the teaching according to the first embodiment. Fig.10 FIG. 1 is a diagram showing an overview of pattern recognition learning performed based on teaching according to the first embodiment. Fig.10 , an example in which the teaching according to the first embodiment is performed by the user is shown.

[0162] For example, Fig.10 As shown on the left hand side of , the teaching according to the first embodiment can be performed by the user using gestures (e.g., pointing with a finger) and using voice. Fig.10In the illustrated example, the user points to the object O1 representing “vacuum cleaner” with the finger UH, and teaches the autonomous moving object 10 that the object O1 is a “vacuum cleaner” using the voice UO.

[0163] At this time, first, based on a user voice such as "remember it", or based on a user gesture indicating the start of teaching-based learning, or based on the sound of fingers snapping, the recognition unit 120 according to the first embodiment detects the start of teaching. In addition, at this time, the operation control unit 150 according to the first embodiment can instruct the autonomous mobile body 10 to perform an action indicating the start of teaching-based learning. For example, the operation control unit 150 can instruct the autonomous mobile body 10 to bark or prick up ears or tail, or change the color of irises expressed using the display 510. In the case where the autonomous mobile body 10 communicates with the user by using language, the operation control unit 150 can instruct the output unit 170 to output a sound indicating the start of learning.

[0164] Then, the operation control unit 150 controls the drive unit 160 and changes the position and posture of the autonomous mobile object 10 in such a manner that the user's finger UH and the object O1 pointed by the finger UH fit within the field of view FV.

[0165] Then, the recognition unit 120 recognizes the object O1 as a learning object based on the direction indicated by the user's finger UH. In addition, the operation control unit 150 according to the first embodiment instructs the input unit 110 to photograph the object O1 recognized as a learning object by the recognition unit 120 .

[0166] Furthermore, the recognition unit 120 performs lexical analysis on the user's voice UO and extracts a noun phrase "vacuum cleaner" for use as a label.

[0167] Then, the learning unit 130 associates the extracted label with the captured image of the learning object, regards the label and the image as learning data, and performs object recognition learning about the object O1.

[0168] In this way, in the autonomous mobile body 10 according to the first embodiment, even if the learning data is not prepared in advance, the learning data on various objects encountered daily can be automatically collected, and the object recognition learning can be performed based on the collected learning data.

[0169] Furthermore, as a result of the function of the autonomous mobile body 10 according to the first embodiment, learning based on teaching is performed in a repetitive manner, and the collective knowledge acquired by the information processing server 20 is used, so that Fig.10As shown on the right-hand side of , object O2 having a shape (feature) different from object O1 can also be recognized as a “vacuum cleaner.” As a result, for example, even when a user purchases a new “vacuum cleaner,” this situation can be handled flexibly without relearning from scratch.

[0170] So far, an explanation has been given about the overview of pattern recognition learning performed based on teaching according to the first embodiment. Fig.10 Although an explanation about object recognition learning is given as an example of pattern recognition learning according to the first embodiment, pattern recognition learning according to the first embodiment is not limited to this example. For example, in pattern recognition learning according to the first embodiment, action recognition learning is included. That is, according to the first embodiment, the action of an object can represent a learning object.

[0171] Fig.11 : is a diagram showing an overview of action recognition learning performed based on teaching according to the first embodiment. Fig.11 In the example shown, the user teaches the autonomous moving object 10 about the "jump" action performed by a person.

[0172] More specifically, if Fig.11 As shown on the left hand side of , the user is using finger UH to indicate person P1 who is "jumping", and is using voice UO to teach that the action that person P1 is performing is "jumping".

[0173] At this time, the recognition unit 120 according to the first embodiment can recognize the phrase "action" included in the voice UO, and can detect that the user is not teaching object recognition (or user recognition) about the person P1, but is teaching about the action that the person P1 is performing. In addition, based on the voice such as "remember action" spoken by the user, the recognition unit 120 can detect that the teaching is about action recognition.

[0174] Then, the recognition unit 120 recognizes the motion performed by the person P1 as a learning object based on the direction indicated by the user's finger UH. In addition, the operation control unit 150 according to the first embodiment instructs the input unit 110 to photograph the motion of the person P1 recognized by the recognition unit 120 as a learning object.

[0175] Furthermore, the recognition unit 120 performs lexical analysis on the user's voice UO and extracts a phrase "jump" for use as a label.

[0176] Then, the learning unit 130 associates the extracted labels with the captured images of the learning objects, regards the labels and the images as learning data, and performs object recognition learning regarding the action performed by the person P1.

[0177] In this way, in the autonomous mobile body 10 according to the first embodiment, not only learning data related to the object but also learning data related to various actions performed by the object can be automatically collected, and object recognition learning based on the learning data can be performed.

[0178] In the above example, an image of an action is used as learning data. Alternatively, for example, the learning unit 130 according to the first embodiment may use motion data collected by an information processing terminal worn by a person who is performing an action as learning data.

[0179] As a result of the above-described functions of the autonomous mobile body 10 according to the first embodiment, learning based on teaching is performed in a repetitive manner, and the collective knowledge acquired by the information processing server 20 is used, so that Fig.11 As shown on the right-hand side of , the “jump” action performed by person P2 different from person P1 can also be recognized with high accuracy.

[0180] At the same time, pattern recognition learning according to the first embodiment may also include, for example, spatial area recognition learning. That is, according to the first embodiment, any spatial area may represent a learning object. According to the first embodiment, the spatial area may be any predetermined area (area) in the space. In addition, according to the first embodiment, the spatial area does not have to be a closed area physically separated from other spatial areas. For example, according to the first embodiment, a "house" may represent a spatial area, or the "first floor" of a "house" may represent a spatial area. In addition, according to the first embodiment, a "living room" on the "first floor" may represent a spatial area, or the "area near the sofa" in the "living room" may represent a spatial area.

[0181] Fig.12 FIG. 1 is a diagram showing an overview of spatial region recognition learning performed based on teaching according to the first embodiment. Fig.12 In the illustrated example, the user teaches the autonomous mobile object 10 the “front door” indicating the spatial area D1 .

[0182] More specifically, if Fig.12 As shown on the left hand side of , the user is using the voice UO to teach the autonomous mobile object 10 that the spatial area D1 where the autonomous mobile object 10 exists represents the "front door".

[0183] At this time, the recognition unit 120 according to the first embodiment can recognize the phrase "here" included in the voice UO, and can detect that the user is teaching about the spatial area D1. In addition, based on the voice such as "remember the area" spoken by the user, the recognition unit 120 can detect that the teaching is about spatial area recognition.

[0184] Then, based on the user's voice UO, the recognition unit 120 recognizes the space area D1 where the autonomous mobile body 10 is currently located as a learning object. In addition, the operation control unit 150 according to the first embodiment instructs the input unit 110 to photograph the space area recognized by the recognition unit 120 as a learning object.

[0185] Furthermore, the recognition unit 120 performs lexical analysis on the user's voice UO and extracts the noun phrase "front door" for use as a label.

[0186] Then, the learning unit 130 associates the extracted labels with the captured images of the learning object, regards the labels and the images as learning data, and performs spatial region recognition learning about the “front door”.

[0187] In this way, in the autonomous mobile body 10 according to the first embodiment, in addition to collecting learning data about objects and actions, learning data about various spatial areas can be automatically collected, and spatial area recognition learning can be performed based on the learning data.

[0188] In the above example, the image of the space area is used as learning data. In this regard, for example, the learning unit 130 according to the first embodiment can use the fact that the user tracked in the space area D1 often disappears (i.e., goes out), and the fact that conversations such as "I'm home" and "I'm leaving" are often detected as features of the space area to be learned.

[0189] As a result of the above-described functions of the autonomous mobile body 10 according to the first embodiment, learning based on teaching is performed in a repetitive manner, and the collective knowledge acquired by the information processing server 20 is used, so that Fig.12 As shown on the right-hand side of , a spatial region D2 having a different environment from the spatial region D1 can also be identified as a “front door”.

[0190] So far, an explanation has been given about an example of pattern recognition learning according to the first embodiment. A detailed explanation about photographing a learning object according to the first embodiment is given below. As described above, the operation control unit 150 according to the first embodiment has a function of instructing the input unit 110 to photograph a learning object recognized by the recognition unit 120.

[0191] At this time, the operation control unit 150 according to the first embodiment may control the input unit 110 and the driving unit 160 in such a manner that pattern recognition is performed with high accuracy and high efficiency.

[0192] For example, the operation control unit 150 according to the first embodiment may control the drive unit 160 to change the position and posture of the autonomous mobile body 10 in such a manner that the recognized object is correctly photographed as a whole.

[0193] Fig.13 FIG. 1 is a diagram for explaining photographing control performed for photographing a learning object according to the first embodiment. Fig.13 , an example is shown in which the user teaches the autonomous mobile object 10 an object O3 representing a “refrigerator” using a voice UO.

[0194] At this time, if Fig.13 As shown in the upper part of , when teaching starts, the position and posture of the autonomous mobile body 10 are such that the distance to the object O3 is too short, and therefore there is a case where the entire object O3 does not fit within the field of view FV. If the image I1 captured in this state is used to perform recognition learning, the features of the object O3 cannot be correctly extracted, and incorrect learning is likely to be performed.

[0195] For this reason, Fig.13 As shown in the lower part of , the operation control unit 150 according to the first embodiment can move the autonomous mobile body 10 to a position and posture that can capture the recognized object O3 as a whole, and then instruct the input unit 110 to capture the object O3. As a result of the function of the operation control unit 150 according to the first embodiment, based on the image I2 that correctly captures the entire object O3, pattern recognition can be performed with high accuracy.

[0196] Meanwhile, if the recognition unit 120 determines that the boundary of the object is not recognizable, the operation control unit 150 according to the first embodiment may perform the control as described above. Fig.13 In the image I1 shown, there is an area at the upper end where the boundary between the background and the object O3 is not recognizable. In this way, when the recognition unit 120 cannot correctly recognize the boundary between the background and the learning object, the operation control unit 150 can control the drive unit 160 until the distance between the learning object and the autonomous mobile body 10 becomes long enough so that the recognition unit 120 can correctly recognize the above boundary.

[0197] On the other hand, if the distance between the learning object and the autonomous mobile body 10 is too long, the operation control unit 150 can perform control to move the autonomous mobile body 10 closer to the learning object. At this time, based on the fact that the recognition unit 120 has determined that the ratio of the background in the image is greater than the predetermined ratio, the operation control unit 150 can perform the above control.

[0198] As a result of the above-described function of the operation control unit 150 according to the first embodiment, pattern recognition learning can be efficiently performed based on an image captured when the distance to the learning object is appropriate.

[0199] Furthermore, the operation control unit 150 according to the first embodiment may control the driving unit 160 and the input unit 110 in such a manner that the learning object recognized by the recognition unit 120 is photographed from a plurality of angles.

[0200] For example, in Fig.14 In the illustrated example, the operation control unit 150 controls the drive unit 160 and the input unit 110 in such a manner that the object O4 representing “plastic bottle” recognized as a learning target by the recognition unit 120 is photographed from the top side and five lateral sides.

[0201] That is, based on images capturing the learning object from various angles, the operation control unit 150 according to the first embodiment can move the autonomous mobile object 10 and instruct capturing a plurality of images in such a manner that pattern recognition learning is efficiently performed.

[0202] As a result of the above-mentioned functions of the operation control unit 150 according to the first embodiment, compared with the case where learning is performed based on capturing an image of only one lateral side of the learning object, features of the learning object can be extracted from various angles, and a learning effect with high generalization can be achieved.

[0203] So far, a detailed explanation has been given about the shooting control according to the first embodiment. So far, an explanation has been mainly given about the case where the teaching according to the first embodiment is performed by the user. However, the teaching according to the first embodiment is not limited to this example. Alternatively, for example, the teaching according to the first embodiment may be performed based on information obtained from a learning object.

[0204] For example, the teaching according to the first embodiment can be realized using a mark such as QR (registered trademark) assigned to the learning object. Fig.15 1 is a diagram for explaining the teaching performed using a marker according to the first embodiment.

[0205] For example, in Fig.15 The upper part of shows an example in which teaching related to object recognition learning is performed using a tag M1 provided in an object O1 representing a “vacuum cleaner.” In this case, the recognition unit 120 may obtain a tag “vacuum cleaner” based on an image of the tag M1 taken by the input unit 110.

[0206] exist Fig.15 The lower part of shows an example in which teaching related to spatial area recognition learning is performed using a marker M2 provided in a door installed in a spatial area D1 representing a “front door”. Also in this case, the recognition unit 120 can also obtain the label “front door” based on the image of the marker M2 taken by the input unit 110.

[0207] In this way, as a result of using the marker according to the first embodiment, object or spatial region related teaching can be performed as an alternative to explicit teaching performed by the user, and the recognition capability of the autonomous mobile object 10 can be automatically enhanced.

[0208] Alternatively, the teaching according to the first embodiment may be performed based on information transmitted from the learning object using wireless communication. Fig.16 2 is a diagram for explaining the execution of teaching using wireless communication according to the first embodiment.

[0209] exist Fig.16 In the illustrated example, the object O1 representing a "vacuum cleaner" uses wireless communication to send the tag "vacuum cleaner" and the images 13a and 13b to the autonomous mobile body 10. Based on the received tag "vacuum cleaner" and the images 13a and 13b, the learning unit 130 according to the first embodiment can perform object recognition learning related to the object O1.

[0210] As a result of performing teaching using wireless communication, e.g. Fig.16 As shown, even when the object O1 representing the learning target cannot be photographed because it is stored in a closet, a pre-prepared image can be sent to the autonomous mobile body 10 together with a tag so that the autonomous mobile body 10 can perform object recognition learning related to the object O1.

[0211] As for the wireless communication according to the first embodiment, for example, NFC (Near Field Communication), or Bluetooth (registered trademark), or RFID (Radio Frequency Identification), or a beacon can be used.

[0212] Meanwhile, the teaching according to the first embodiment can also be implemented using inaudible sounds such as ultrasonic waves emitted by a learning object. Fig.17 is a diagram for explaining teaching using inaudible sound according to the first embodiment.

[0213] exist Fig.17 In the example shown, teaching is performed using a predetermined inaudible sound emitted by an object O5 representing a "washing machine" when operated. Fig.17 The upper part of shows an example in which, based on the inaudible sound NAS1 emitted by the object O5 during “draining”, the recognition unit 120 detects that the object O5 is a “washing machine” and detects that the object O5 is performing “draining”.

[0214] For example, in Fig.17 The lower part of shows an example in which, based on the inaudible sound NAS2 emitted by the object O5 during “drying”, the recognition unit 120 detects that the object O5 is a “washing machine” and detects that the object O5 is performing “drying”.

[0215] In this way, as a result of the teaching performed using inaudible sounds according to the first embodiment, not only the name of the object but also the operating state of the object can be taught. In addition, as a result of the teaching performed using inaudible sounds according to the first embodiment, for example, the learning unit 130 can also learn the audible sound AS1 representing the operating noise emitted by the object O5 during "draining", learn the audible sound AS2 representing the operating noise emitted by the object O5 during "drying", and learn the operating state of the object O5. As a result of performing such learning in a repetitive manner, the autonomous mobile body 10 according to the first embodiment becomes capable of gradually acquiring recognition capabilities regarding objects that do not emit audible sounds.

[0216] So far, explanations have been given about variations of the teaching performed according to the first embodiment. Explanations are given below about correction of associations between labels and learning objects according to the first embodiment. As described above, the autonomous mobile body 10 according to the first embodiment can perform pattern recognition learning based on various teachings.

[0217] However, for example, when learning is performed based on the user's teaching, it is also possible to conceive of a situation in which a learning object or a label is obtained by mistake. In this regard, the application control unit 240 of the information processing server 20 according to the first embodiment can provide a user interface that enables the user (or developer or service provider) to correct the association between the label and the learning object.

[0218] Fig.18 and Fig.19 is a diagram showing an example of a user interface according to the first embodiment. Fig.18 , there is shown an example of a user interface which is displayed in the information processing terminal 40 owned by the user by the application control unit 240 after the pattern recognition learning is performed based on the teaching.

[0219] exist Fig.18 In the example shown, although the user has specified "plastic bottle" by pointing at "plastic bottle" with a finger, and has performed teaching using voice, "glass" existing near "plastic bottle" is mistakenly recognized as a learning object, as shown in FIG. Fig.18 The left hand side of the figure is shown, and the label is incorrectly extracted as "Plastic Bottle".

[0220] At this time, if Fig.18 As shown on the right-hand side of , in the image I4 displayed in the user interface, the user selects the boundary recognition result about "plastic bottle" representing the correct learning object, and enters the correctly spelled "plastic bottle" in the field F1 corresponding to the label. Therefore, the incorrectly associated learning object and label can be easily corrected.

[0221] At the same time, the user can correct the association not only immediately after the learning based on the teaching but also from the history of the pattern recognition performed by the autonomous mobile body 10 .

[0222] For example, in Fig.19 An example of a user interface showing objects that the autonomous moving body 10 has recognized during its autonomous behavior is shown in FIG. Fig.19 The history shown on the center left hand side, where an object named "closet" was incorrectly identified as a "refrigerator".

[0223] At this point, the user can select the misidentified history, enter the correct label, and correct the association, e.g. Fig.19 In this way, as a result of providing the user interface under the control of the application control unit 240 according to the first embodiment, the details of any erroneous learning can be discovered immediately after learning or at any subsequent time point, and correction can be easily made.

[0224] At the same time, or alternatively, the correction of the learning result according to the first embodiment may be automatically performed by the autonomous mobile body 10. For example, the learning unit 130 according to the first embodiment may compare the learning result stored in the information processing server 20 with the learning result obtained by the corresponding autonomous mobile body 10; detect a mismatch between the collective knowledge body and the learning knowledge of the corresponding autonomous mobile body 10; and automatically correct a label or a learning object.

[0225] So far, explanations have been given about learning based on the teaching according to the first embodiment and about correction of the learning results. As a result of the above-mentioned functions of the autonomous mobile body 10 and the information processing server 20 according to the first embodiment; even if learning data is not provided in advance, learning data can be dynamically collected and learned on a daily basis, thereby enabling the enhancement of the recognition capability of the autonomous mobile body 10 with less burden and higher efficiency.

[0226] Meanwhile, in order to collect learning data more efficiently, the operation control unit 150 according to the first embodiment may instruct the autonomous mobile body 10 to perform a guiding operation to guide the user to teach.

[0227] Fig. 20 1 is a diagram for explaining the guidance operation according to the first embodiment. For example, when the autonomous mobile body 10 is performing an autonomous action, if the recognition unit 120 detects an unknown object Ox that cannot be recognized, the operation control unit 150 according to the first embodiment can instruct the autonomous mobile body 10 to perform the guidance operation to guide the user to teach about the object Ox.

[0228] exist Fig. 20In the illustrated example, based on the detection of the unrecognizable object Ox by the recognition unit 120 , the operation control unit 150 instructs the autonomous mobile object 10 to perform a guiding operation in the form of barking at the object Ox.

[0229] In this way, as a result of performing the guidance operation according to the first embodiment, the possibility of the user being taught about the unknown object Ox can be increased; and collection of learning data with higher efficiency can be expected.

[0230] Examples of the guidance operation according to the first embodiment include barking, sniffing, leaning, tilting the head to one side, alternately looking at the user and the object, and becoming afraid.

[0231] Furthermore, in addition to instructing the autonomous mobile body 10 to perform the guiding operation, the operation control unit 150 according to the first embodiment may instruct the autonomous mobile body 10 to take various other actions with respect to the learned event.

[0232] For example, by taking action on a newly recognized object such as a furniture item, the operation control unit 150 can prompt the user to use the furniture item. For example, based on the fact that the "vacuum cleaner" has not been operated for more than a predetermined period of time, the operation control unit 150 can instruct the autonomous mobile body 10 to take action as described above.

[0233] In addition, for example, regarding an object defined as a favorite object among the objects learned by the autonomous mobile body 10, the operation control unit 150 can instruct the autonomous mobile body 10 to express the favorite by hugging or embracing the favorite object. Such a favorite object can be defined based on, for example, a recognition count or an object identification (e.g., a manufacturer).

[0234] In addition, for example, the operation control unit 150 can instruct the autonomous mobile body 10 to imitate a newly learned action. At this time, based on the sequential data generated from the captured image of the action, the operation control unit 150 can instruct the autonomous mobile body 10 to take the learned action. In this article, the sequential data can be information containing the time-sequential changes in the rotational position of the joint area of ​​the autonomous mobile body 10 and containing control signals related to eyeball expressions and audio output. For example, the sequential data is generated based on the estimation of the joint position of the object performing the action and based on the estimation of the bending action amplitude.

[0235] In addition, for example, when the learning unit 130 newly learns an action such as an exercise performed by a user; based on the learning data of the exercise stored as a collective knowledge body in the information processing server 20, the operation control unit 150 can instruct the autonomous mobile body 10 to exhibit a behavior related to the user's exercise evaluation. At this time, for example, the evaluation can be performed based on the difference from the average value of the learning data stored as a collective knowledge body.

[0236] Furthermore, for example, when the learning unit 130 has newly learned the “front door,” the operation control unit 150 may control the drive unit 160 and the output unit 170 in such a manner that the autonomous moving body 10 greets the user at the “front door” when the user returns home.

[0237] In addition, for example, when the learning unit 130 has also learned the operating state of the object; if an abnormality is detected from the operating sound, the operation control unit 150 can instruct the autonomous mobile body 10 to notify the abnormality. In addition, for example, the operation control unit 150 can instruct the autonomous mobile body 10 to perform an action according to the recognized operating state of the object. For example, when it is recognized that the microwave oven is operating, the operation control unit 150 can instruct the autonomous mobile body 10 to keep the dishes ready.

[0238] <<1.7. Operation process>>

[0239] A detailed explanation is given below of the flow of operations performed by the autonomous mobile object 10 according to the first embodiment. Fig.21 1 is a flowchart for explaining the flow of operations performed by the autonomous mobile object 10 according to the first embodiment.

[0240] refer to Fig.21 First, the recognition unit 120 detects teaching (S1101). For example, the recognition unit 120 may detect the start of teaching based on user voice, or based on detection of a QR code, or based on information reception via wireless communication, or based on detection of an inaudible sound.

[0241] Then, based on the teaching start detected in step S1101, the operation control unit 150 instructs the autonomous mobile body 10 to perform an action instructing the start of pattern recognition learning (S1102). Examples of such actions include barking, moving ears or tail, and changing iris color.

[0242] Subsequently, the recognition unit 120 recognizes the learning object (S1103). For example, the recognition unit 120 may recognize the learning object based on a gesture such as pointing with a finger performed by the user, or may recognize the learning object based on information obtained from the learning object.

[0243] In addition, the recognition unit 120 obtains a label (S1104). For example, the recognition unit 120 may extract a label from the user's voice, or may obtain a label from information obtained from the learning object.

[0244] Then, the operation control unit 150 controls the driving unit 160 in such a manner that the autonomous mobile body 10 is in a position and a posture capable of capturing the learning object recognized in step S1103 as a whole ( S1105 ).

[0245] Subsequently, the operation control unit 150 controls the driving unit 160 and the input unit 110 in such a manner that the learning object is photographed from a plurality of angles ( S1106 ).

[0246] Then, the learning unit 130 performs pattern recognition learning based on the label obtained in step S1104 and the image captured in step S1106 ( S1107 ).

[0247] Subsequently, the operation control unit 150 instructs the autonomous mobile body 10 to perform an action indicating the end of pattern recognition learning (S1108). Examples of such an action include barking, moving ears or tail, and changing the color of the iris.

[0248] <2. Second Embodiment>

[0249] <<2.1. Overview>>

[0250] So far, explanations have been given about the first embodiment. Explanations of the second embodiment are given below. In the above-mentioned first embodiment, teaching related to pattern recognition learning is performed based on the user's voice or gesture or based on information obtained from a learning object. In the second embodiment, explanations are given about performing teaching mainly using an environment map.

[0251] The environmental map according to the second embodiment refers to a map including information about the placement of objects in the real space where the autonomous mobile body 10 exists, and including information about the structure of the building. For example, the environmental map according to the second embodiment can be a map that enables a user to visualize information obtained using SLAM (a map that enables a user to visualize is sometimes simply referred to as a SLAM map). As described above, the autonomous mobile body 10 according to the second embodiment can implement SLAM using an image captured by a wide-angle camera (i.e., camera 520) installed in the waist area.

[0252] In the second embodiment of the present application, the environment map generated in the above-described manner is presented to the user, and the user marks an arbitrary designated area in the environment map, and teaching related to pattern recognition learning of the spatial area is performed.

[0253] To this end, one feature of the application control unit 240 of the information processing server 20 according to the second embodiment is to control a user interface that enables the user to perform teaching using an environment map. For example, the application control unit 240 according to the second embodiment provides a user interface as a function of a network application, and the user interface can be displayed in an information processing terminal 40 (e.g., a smartphone or a PC (personal computer)) owned by the user. At the same time, or alternatively, the control function for the user interface can be implemented as a function of the autonomous mobile body 10.

[0254] A detailed explanation of performing teaching using an environment map according to the second embodiment is given below. The following explanation mainly gives the difference from the first embodiment, and the same functions and effects as the first embodiment are not explained in detail.

[0255] <<2.2. Details of instructions for using environment maps>>

[0256] exist Fig. 22 , an example of a user interface UI2 for teaching using an environment map according to the second embodiment is shown. Fig. 22 As shown, in the user interface UI2 according to the second embodiment, an environment map EM generated based on the sensor information collected by the autonomous mobile body 10 is presented to the user. As mentioned above, the environment map EM according to the second embodiment can be, for example, a SLAM map. The application control unit 240 can put the SLAM map received from the autonomous mobile body 10 in the user interface UI2. Alternatively, the information processing server 20 can generate an environment map EM, for example, a SLAM map, based on the sensor information (including position information and images) collected from the autonomous mobile body 10. The following explanation is given under the assumption that the environment map according to the second embodiment is a SLAM map.

[0257] In the environment map EM according to the second embodiment, for example, information on the real space recognized by the autonomous mobile body 10 may be indicated in units of blocks. Fig. 22 In the illustrated example, in the environment map EM, a plurality of blocks defined by predetermined units (distances) are placed in a two-dimensional coordinate system corresponding to a horizontal plane in the real space.

[0258] A block represents an attribute of a corresponding spatial region in the real space. For example, a block B1 represents a spatial region (e.g., a wall) to which the autonomous mobile body 10 cannot move (enter). In addition, a block B2 represents a spatial region to which the autonomous mobile body 10 actually moved in the past (i.e., a spatial region to which movement is allowed). In addition, a block B3 represents a spatial region to which the autonomous mobile body 10 has not yet moved. In addition, a block B4 represents an area to which it is difficult to determine whether it is possible to move or whether it has not yet moved.

[0259] With regard to such a spatial region represented by blocks B1 to B4, for example, the attributes can be represented using differences in color, pattern, and shape. Fig. 22 , an example of using color differences to express attributes is shown.

[0260] In addition, if Fig. 22As shown, in the environment map EM according to the second embodiment, the current position CP of the autonomous mobile body 10 and the position of the charging station CS for supplying electric power to the autonomous mobile body 10 may also be indicated.

[0261] In the environment map EM displayed in the user interface UI2 , the user can designate and mark any designated area, and can perform teaching related to the space area in the real space corresponding to the designated area.

[0262] At this time, first, the user selects an arbitrary designated area by performing a drag operation in the environment map EM. Fig. 22 In the illustrated illustration, the area selected and designated by the user is indicated by a dotted line. The user then assigns a label T (i.e., a label including the name of the spatial area) to the selected designated area. Fig. 22 An example is shown in FIG. 5 , in which a user assigns a tag T including a name “kitchen” to a selected designated area.

[0263] In this way, one feature of the application control unit 240 according to the second embodiment is to provide the user with a way to specify an arbitrary specified area of ​​the environment map EM in the user interface UI2 and to mark the specified area. As a result of this feature of the application control unit 240 according to the second embodiment, pattern recognition learning about a spatial area can be easily performed with high accuracy and using the environment map EM in a visualized form.

[0264] Furthermore, the application control unit 240 according to the second embodiment may also have a function of presenting candidates for a label (mark) that can be assigned to a designated area by the user in the user interface UI2 .

[0265] exist Fig. 22 In the illustrated example, the application control unit 240 displays candidate items OP such as “kitchen”, “bedroom”, and “living room” in the user interface UI2 as labels (tags) assignable by the user.

[0266] In this case, the user can drag a desired label from among the presented candidates OP in the environment map EM and easily mark the designated area.

[0267] As a result of pre-presenting such a plurality of common labels as described above, the application control unit 240 becomes able to absorb the variation in the naming given by each user and keep checking the naming type of the label. As a result, learning data having high versatility and shareable among a plurality of autonomous mobile bodies 10 can be efficiently collected, and pattern recognition learning can be performed with high accuracy.

[0268] Regarding the generation of the environment map EM according to the second embodiment, the following refers to Figure 23 to Figure 27 Give a detailed explanation. Figure 23 to Figure 27 is a diagram showing an example of a generation process for generating an environment map EM according to the second embodiment. As described above, the environment map EM according to the second embodiment may be a SLAM map generated for the user based on sensor information collected by the autonomous mobile body 10. Figure 23 to Figure 27 Although the reference numerals of the blocks B1 to B4, the current position CP of the autonomous moving body 10, and the charging station CS are not shown, it is assumed that Figure 22 to Figure 27 Shared visual expression and Fig. 22 The reference numerals shown are identical.

[0269] exist Fig.23 , an example of the environment map EM at the start of the generation of the environment map EM is shown. At this time, since sensor information capable of determining the attributes of the spatial region has not yet been sufficiently stored, only the current position CP of the autonomous mobile body 10 is shown.

[0270] exist Fig.24 In the figure, it is shown that Fig.23 An example of an environment map EM after a certain amount of time has passed since the time point shown. Fig.24 , the attributes of the spatial area are determined based on the sensor information collected accompanying the autonomous movement of the autonomous moving body 10; and it can be understood that the visual information such as blocks B1 to B4 has been added to the environment map EM. In addition, as Fig.24 As shown, in the environment map EM according to the second embodiment, in addition to indicating the current position CP of the autonomous moving object 10, its movement trajectory may be indicated using, for example, a line extending from the current position CP.

[0271] exist Fig.25 In the figure, it is shown that Fig.24 An example of an environment map EM after a greater amount of time has passed since the time point shown in FIG. Fig.25 and Fig.24 As a result of the comparison, it can be understood that the density of the blocks has increased because more sensor information has been stored over time. In this way, the environment map EM according to the second embodiment can be dynamically updated based on the sensor information collected along with the autonomous movement of the autonomous moving body 10.

[0272] exist Fig.26 In the figure, it is shown that Fig.25 Example of an environment map EM after a greater amount of time has passed since the time point shown in . Fig.26 It can be understood that the scale of the environment map EM changes due to the expansion of the moving range of the autonomous mobile body 10, and the environment map EM is updated as a map corresponding to a wider real space. In addition, as more sensor information is stored, the density of the block further increases.

[0273] exist Fig. 27 In the figure, it is shown that Fig.26 An example of an environment map EM after a greater amount of time has passed since the time point shown in FIG. Fig. 27 and Fig.26 The results of the comparison are Fig.26 Compared to the environmental map EM shown, Fig. 27 The scale of the environment map EM shown is further changed, and the environment map EM corresponds to a wider spatial area. In addition, it can be understood that due to the expansion of the movement range of the autonomous moving body 10, blocks are added in new areas.

[0274] So far, an explanation has been given about the generation process for generating an environment map EM according to the second embodiment. As described above, based on the sensor information (position information and images) collected accompanying the autonomous movement of the autonomous mobile body 10, the autonomous mobile body 10 according to the second embodiment can generate an environment map EM such as a SLAM map, and can be dynamically updated. In addition, the application control unit 240 according to the second embodiment displays the environment map EM in the user interface UI2 so that the user can understand the range of the spatial area in the real space recognized by the autonomous mobile body 10, and can use it for subsequent learning.

[0275] A detailed explanation of the movement control and information acquisition performed for moving to a spatial area corresponding to a designated area according to the second embodiment is given below. As described above, using the environment map EM displayed in the user interface UI2, the user can assign a tag to the designated area. At this time, the operation control unit 150 according to the second embodiment can move the autonomous mobile body 10 to a spatial area corresponding to the designated area that has been marked in the real space, and can instruct the autonomous mobile body 10 to obtain information about the spatial area.

[0276] Fig.28 : is a diagram for explaining movement control and information acquisition performed to move to a spatial area corresponding to a designated area according to the second embodiment. Fig.28 The left hand side of the diagram shows the environment map EM, and Fig.28 The right-hand side of shows an example of an image captured by a wide-angle camera (camera 520 ) mounted at the tip of the nose of the autonomous mobile object 10 .

[0277] For example, based on the marking performed in the environment map EM, or based on the user's voice such as "go to this place" after the marking is performed, the operation control unit 150 according to the second embodiment can move the autonomous mobile body 10 to a spatial area corresponding to the designated area that has been marked. In addition, in addition to performing the above-mentioned movement control, the operation control unit 150 according to the second embodiment instructs the autonomous mobile body 10 to capture an image of the relevant spatial area.

[0278] In this way, due to the operation control unit 150 according to the second embodiment, information related to the spatial area in the real space corresponding to the designated area that has been marked by the user in the environment map EM can be collected with high accuracy. At the same time, it is not always necessary to perform image capture immediately after marking. Alternatively, for example, after marking has been performed, when the autonomous mobile body 10 reaches the relevant spatial area during autonomous movement, an image can be captured.

[0279] The various information collected in the above manner is used for pattern recognition learning of the spatial region performed by the autonomous mobile body 10. For example, the learning unit 130 of the autonomous mobile body 10 can perform pattern recognition learning by associating the label assigned via the interface UI2 with the image of the spatial region taken in the above manner. In addition, the learning unit 130 can learn images taken in time sequence with movement to the spatial region and related time sequence information (or can learn frames of moving images), and thus can accurately learn the features that produce the relevant spatial region.

[0280] As a result of implementing the above-described set of techniques, it is possible to efficiently store annotated teaching learning data that can be shared among a plurality of autonomous mobile bodies 10, and in which nomenclature that does not change is associated with an image of a spatial region actually viewed from the viewpoint of the autonomous mobile body 10 (i.e., an image captured by the camera 520 mounted on the nose tip); and the teaching learning data can be used to enhance the accuracy of environment recognition in the autonomous mobile body 10. The information processing server 20 can share the teaching learning data among a plurality of autonomous mobile bodies 10 by collecting and distributing the teaching learning data.

[0281] As a result of this learning method, even if the autonomous mobile body 10 moves to an unlabeled spatial region, the name of the spatial region can be estimated based on the learning result.

[0282] Meanwhile, so far, explanation has been mainly given about the case where the operation control unit 150 instructs the autonomous mobile body 10 to capture an image of a spatial area. However, the information collection control according to the second embodiment is not limited to this example. That is, the operation control unit 150 according to the second embodiment can instruct the autonomous mobile body 10 to also collect various sensor information related to the spatial area.

[0283] For example, the operation control unit 150 according to the second embodiment can instruct the autonomous mobile body 10 to collect audio data related to the spatial area. In this case, the learning unit 130 can perform learning by associating the audio data and the label. As a result, for example, based on the detection of the sound of flowing water, the recognition unit 120 of the autonomous mobile body 10 can estimate that the spatial area of ​​the current position is a riverside.

[0284] In addition, for example, the operation control unit 150 according to the second embodiment can instruct the autonomous mobile body 10 to collect vibration data related to the spatial area. In this case, the learning unit 130 can perform learning by associating the vibration data and the label. As a result, for example, based on the detection of the vibration characteristics of the vehicle, the recognition unit 120 of the autonomous mobile body 10 can estimate that the spatial area of ​​the current position is a highway.

[0285] In addition, for example, the operation control unit 150 according to the second embodiment may instruct the autonomous mobile body 10 to collect sensor information related to the intensity of odor in the spatial area. In this case, the learning unit 130 may perform learning by associating the odor-related sensor information and the label. As a result, for example, based on the detection of a strong odor, the recognition unit 120 of the autonomous mobile body 10 is able to estimate that the spatial area of ​​the current position is a toilet.

[0286] Meanwhile, although an explanation has been given so far about the case where the environment map EM is a two-dimensional map, the environment map EM can be essentially extended to multiple dimensions. For example, when the autonomous mobile body 10 is an unmanned aerial vehicle such as a drone capable of flying, the autonomous mobile body 10 can generate a SLAM map including altitude information. In this case, the user can select a designated area with a designated altitude and then mark the designated area.

[0287] In addition, the environment map EM according to the second embodiment may also include time information. In this case, the user can specify the time and area, and then assign a label (for example, the sea at night). In addition, the environment map EM according to the second embodiment may be defined together with metadata. For example, the environment map EM according to the second embodiment may also contain six-dimensional information, such as the coordinates (x, y, z) of room F on floor E in building D. In addition, multiple environment maps EM according to the second embodiment may also be generated. In this case, the user can label all environment maps EM (for example, the label "first floor" may be assigned to environment map EM1, and the label "second floor" may be assigned to environment map EM2).

[0288] In addition, the following can be displayed in real time in the user interface UI2: Fig.28The image shown or the moving trajectory of the autonomous mobile body 10 in the environment map EM. As a result of confirming such information, the user can understand whether the designated area has been correctly identified, can understand whether the autonomous mobile body 10 is correctly moving in the spatial area corresponding to the designated area, and can understand the way in which the autonomous mobile body 10 "looks at" the spatial area corresponding to the designated area. At the same time, the real-time display of the image and the environment map EM is suitable for, for example, a monitoring system or a surveillance system.

[0289] 《The process of instruction-based learning using environment map EM》

[0290] A detailed explanation is given below of the flow of teaching-based learning using the environment map according to the second embodiment. Fig.29 is a flowchart for explaining an example of the flow of teaching-based learning using the environment map EM according to the second embodiment.

[0291] refer to Fig.29 First, based on the sensor information collected by the recognition unit 120 of the autonomous mobile body 10, an environment map EM is generated (S2101). Herein, the environment map EM according to the second embodiment may be generated by the information processing server 20.

[0292] Then, the application control unit 240 of the information processing server 20 presents the environment map EM generated in step S2101 to the user via the user interface UI2. The user selects any designated area in the environment map EM and marks the designated area (S2102).

[0293] Subsequently, the operation control unit 150 of the autonomous mobile body 10 instructs the autonomous mobile body 10 to move to such a space area in the real space corresponding to the designated area selected in step S2101 and collects sensor information (image) related to the space area ( S2103 ).

[0294] Then, the learning unit 130 of the autonomous mobile body 10 associates the label assigned in step S2102 and the sensor information collected in step S2103 , and performs pattern recognition learning about the spatial area ( S2104 ).

[0295] <3. Exemplary Hardware Configuration>

[0296] An explanation is given below of an exemplary hardware configuration of the information processing server 20 according to an embodiment of the present application. Fig.30 2 is a block diagram showing an exemplary hardware configuration of the information processing server 20 according to an embodiment of the present application. Fig.30, the information processing server 20 includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a memory 880, a driver 881, a connection port 882, and a communication device 883. However, this hardware configuration is merely exemplary, and some components may be omitted. In addition, some other components other than the components shown may be included.

[0297] (Processor 871)

[0298] The processor 871 functions as, for example, an arithmetic processing device or a control device, and controls operations of other constituent elements partially or entirely based on various programs recorded in the ROM 872 , the RAM 873 , the storage 880 , or the removable recording medium 901 .

[0299] (ROM 872 and RAM 873)

[0300] The ROM 872 is used to store programs read by the processor 871 and store data used in operation. For example, the RAM 873 is used to temporarily or permanently store programs read by the processor 871 and store various parameters that undergo appropriate changes during program execution.

[0301] (Host bus 874, bridge 875, external bus 876 and interface 877)

[0302] The processor 871, the ROM 872, and the RAM 873 are connected to each other via, for example, a host bus 874 capable of high-speed data transmission. In addition, the host bus 874 is connected to an external bus 876 having a relatively slow data transmission speed via, for example, a bridge 875. In addition, the external bus 876 is connected to various constituent elements via an interface 877.

[0303] (Input device 878)

[0304] Examples of the input device 878 include a mouse, a keyboard, a touch-sensitive panel, a button switch, and a joystick. In addition, as the input device 878, a remote controller capable of transmitting a control signal using infrared or other radio waves is sometimes used. In addition, the input device 878 includes an audio input device such as a microphone.

[0305] (Output device 879)

[0306] The output device 879 can notify the user of the obtained information in a visual or auditory manner, and examples of the output device 879 include a display device, such as a CRT (cathode ray tube), LCD, or organic EL; an audio output device, such as a speaker or headphones; a printer; a mobile phone; and a fax machine. In addition, the output device 879 according to the present application includes various vibration devices capable of outputting tactile stimulation.

[0307] (Memory 880)

[0308] The memory 880 is used to store various data. Examples of the memory 880 include a magnetic storage device, such as a hard disk drive (HDD); a semiconductor storage device; an optical storage device; and a magneto-optical storage device.

[0309] (Drive 881)

[0310] The drive 881 reads information recorded in the removable recording medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory; and writes information in the removable recording medium 901 .

[0311] (Removable Recording Medium 901)

[0312] Examples of the removable recording medium 901 include DVD media, Blu-ray (registered trademark) media, HD DVD media, and various semiconductor storage media. Of course, the removable recording medium 901 may be an IC card mounted with a non-contact type IC chip, or may be an electronic device.

[0313] (Connection port 882)

[0314] The connection port 882 is a port capable of establishing connection with the external connection device 902, for example, a USB (Universal Serial Bus) port, an IEEE1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal.

[0315] (External connection device 902)

[0316] Examples of the externally connected device 902 include a printer, a portable music player, a digital camera, a digital video camera, and an IC recorder.

[0317] (communication device 883)

[0318] The communication device 883 is capable of establishing a connection with a network, and examples of the communication device 883 include communication cards for wired LAN or wireless LAN, or Bluetooth (registered trademark), or WUSB (wireless USB); optical communication routers; ADSL (Asymmetric Digital Subscriber Line) routers; and various communication modems.

[0319] <4. Conclusion>

[0320] As described above, the autonomous mobile body 10 according to the embodiment of the present application includes the operation control unit 150 that controls the operation of the autonomous mobile body 10. As a feature of the operation control unit 150 according to the embodiment of the present application, when the start of teaching related to pattern recognition learning is detected, the autonomous mobile body 10 is instructed to obtain information about a learning object learned in a manner corresponding to the taught label. With this configuration, learning related to pattern recognition can be performed more efficiently.

[0321] Although the present application is described in detail in the form of embodiments with reference to the accompanying drawings, the technical scope of the present application is not limited to the above embodiments. That is, the present application will be interpreted as embodying all modifications that may be thought of by those skilled in the art and fall completely within the basic teachings set forth herein, such as other embodiments, additions, alternative constructions, and deletions. In any form thereof, as long as the functions / effects of the present application are achieved, the modifications are included in the scope of the present application.

[0322] The effects described in this written description are merely illustrative and exemplary and are not limited in scope. That is, in addition to or in place of the above effects, the technology disclosed in this application can achieve other effects that may be conceived by those skilled in the art.

[0323] Meanwhile, it is also possible to create a program for causing computer hardware including a CPU, a ROM, and a RAM to realize a function equivalent to the configuration of the autonomous mobile body 10 or the information processing server 20. Furthermore, a computer-readable recording medium recording the program may be provided.

[0324] Meanwhile, the steps of the operation performed by the autonomous mobile body 10 in this written description need not be processed in chronological order according to the order given in the sequence diagram and the flowchart. For example, the steps of the operation performed by the autonomous mobile body 10 may be processed in an order different from that given in the flowchart, or may be processed in parallel.

[0325] Meanwhile, the configuration described below also falls within the technical scope of the present application.

[0326] (1) An information processing device includes an operation control unit that controls an operation of an autonomous moving body that performs an action according to a recognition operation, wherein:

[0327] Based on detection of the start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile body to obtain information on a learning object learned in a manner corresponding to the taught label.

[0328] (2) The information processing device according to (1), wherein:

[0329] The learning object includes a spatial region, and

[0330] The operation control unit instructs the autonomous mobile body to obtain information about the spatial area based on a label assigned in an environment map generated based on sensor information collected by the autonomous mobile body.

[0331] (3) The information processing device according to (2), wherein the operation control unit instructs the autonomous mobile body to move to a spatial area in the real space, the spatial area corresponding to the designated area to which the tag is assigned in the environment map.

[0332] (4) The information processing device according to (3), wherein the operation control unit instructs the autonomous mobile object to photograph the spatial area corresponding to the designated area.

[0333] (5) The information processing device according to any one of (2) to (4), wherein the environment map is generated or updated using SLAM technology.

[0334] (6) The information processing device according to (1), wherein the operation control unit controls the position or posture of the autonomous mobile body in such a manner that the learning object is photographed from a plurality of angles.

[0335] (7) The information processing device according to (1) or (2), wherein the operation control unit controls the position or posture of the autonomous mobile body in such a manner that the learning object is captured as a whole.

[0336] (8) The information processing device according to (7), wherein the operation control unit controls the position or posture of the autonomous mobile body based on a result of boundary recognition performed on the learning object.

[0337] (9) The information processing device according to any one of (1) or (6) to (10), wherein:

[0338] Teaching is performed by the user, and

[0339] The operation control unit instructs the autonomous mobile body to photograph the learning object recognized based on an instruction of a user.

[0340] (10) The information processing device according to (9), wherein the user's instruction is given using a gesture.

[0341] (11) The information processing device according to any one of (1) or (6) to (10), wherein the teaching is performed based on information obtained from the learning object.

[0342] (12) The information processing device according to (11), wherein the teaching is performed by photographing a mark assigned to the learning object.

[0343] (13) The information processing device according to (11) or (12), wherein the teaching is performed based on information transmitted from the learning object using wireless communication.

[0344] (14) The information processing device according to any one of (11) to (13), wherein the teaching is performed based on an inaudible sound emitted from the learning object.

[0345] (15) The information processing device according to (9) or (10), wherein the label is extracted from the user's speech.

[0346] (16) The information processing device according to any one of (1) or (6) to (15), wherein:

[0347] Learning objects include objects, and

[0348] The operation control unit instructs the autonomous mobile object to capture an object recognized based on the teaching.

[0349] (17) The information processing device according to any one of (1) or (6) to (16), wherein:

[0350] Learning objects include the actions of objects, and

[0351] The operation control unit instructs the autonomous mobile body to obtain information on the motion of the object recognized based on the teaching.

[0352] (18) The information processing device according to any one of (1) or (6) to (17), wherein:

[0353] The learning object includes a spatial region, and

[0354] The operation control unit instructs the autonomous mobile body to obtain information on the spatial area recognized based on the teaching.

[0355] (19) The information processing device according to any one of (1) or (6) to (18), wherein the operation control unit instructs the autonomous mobile body to perform an action indicating that pattern recognition learning has started or ended.

[0356] (20) The information processing device according to any one of (1) or (6) to (19), wherein the operation control unit instructs the autonomous mobile body to perform a guiding operation to guide the user to perform the teaching.

[0357] (21) The information processing device according to any one of (1) to (20) further includes an application control unit that controls a user interface used when associating a label and a learning object regarding pattern recognition learning.

[0358] (22) The information processing device according to (21), wherein the application control unit provides a user with a way of designating any designated area in the environment map and assigning a label to the designated area in the user interface.

[0359] (23) The information processing device according to (22), wherein the application control unit presents candidates for a label that can be assigned to the designated area by a user in the user interface.

[0360] (24) The information processing device according to (21), wherein the application control unit provides a user with a way of correcting the association between the label and the learning object in the user interface.

[0361] (25) The information processing device according to any one of (1) to (24), further including a learning unit that performs pattern recognition learning based on the label and based on the image of the learning object.

[0362] (26) An information processing method implemented in a processor, the method comprising:

[0363] controlling the operation of an autonomous moving object to perform an action according to the recognition operation, wherein:

[0364] Based on detecting the start of teaching related to pattern recognition learning, the control operation includes instructing the autonomous mobile body to obtain information on a learning object learned in a manner corresponding to the taught label.

[0365] (27) A program for causing a computer to function as an information processing device, the information processing device including an operation control unit that controls an operation of an autonomous moving body that performs an action according to a recognition operation, wherein:

[0366] Based on detection of the start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile body to obtain information on a learning object learned in a manner corresponding to the taught label.

[0367] Reference numerals list

[0368] 10 Autonomous moving objects

[0369] 110 Input unit

[0370] 120 Identification Units

[0371] 130 study units

[0372] 140 Action Planning Unit

[0373] 150 Operation control unit

[0374] 160 Driver Unit

[0375] 170 Output Unit

[0376] 510 Display

[0377] 570 Actuator

[0378] 20 Information processing server

[0379] 210 Identification Unit

[0380] 220 study units

[0381] 230 storage units

[0382] 240 Application control unit.

Claims

1. An information processing device, comprising an operation control unit that controls an operation of an autonomous moving body that performs an action according to a recognition operation, wherein: Based on detection of the start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile body to obtain information about a learning object to be learned in a manner corresponding to a taught label and presented in a direction indicated by a finger of a user, and extracts a label from a voice uttered by the user after taking the object indicated by the finger of the user as a learning object; and performs the pattern recognition learning by associating the extracted label with a captured image of the learning object; instructing the autonomous mobile object to perform a first action to indicate completion of the pattern recognition learning; determining that an identified learning object identified by the pattern recognition learning has not been used by the user for more than a certain period of time; and In response to the identified learning object not being used by the user for more than a certain period of time, the autonomous mobile object is instructed to take a second action to prompt the user.

2. The information processing device according to claim 1, wherein: The learning object includes a spatial region, and The operation control unit instructs the autonomous mobile object to obtain information about the spatial area based on the label assigned in an environment map generated from sensor information collected by the autonomous mobile object.

3. The information processing device according to claim 2, wherein: The operation control unit instructs the autonomous mobile body to move in the real space to the spatial area corresponding to the designated area to which the label is assigned in the environment map.

4. The information processing device according to claim 3, wherein: The operation control unit instructs the autonomous mobile object to capture the spatial area corresponding to the designated area.

5. The information processing device according to claim 2, wherein: The environment map is generated or updated using a simultaneous localization and mapping technique.

6. The information processing device according to claim 1, wherein: The operation control unit controls the position or posture of the autonomous mobile body in such a manner that the learning object is photographed from a plurality of angles.

7. The information processing device according to claim 1, wherein: The operation control unit controls the position or posture of the autonomous mobile body in such a manner that the learning object is captured as a whole.

8. The information processing device according to claim 7, wherein: The operation control unit controls a position or a posture of the autonomous mobile body according to a result of boundary recognition performed on the learning object.

9. The information processing device according to claim 1, wherein: The teaching is performed by the user, and The operation control unit instructs the autonomous mobile object to photograph the learning object recognized based on an instruction from the user.

10. The information processing device according to claim 9, wherein: The user's instructions are given using gestures.

11. The information processing device according to claim 1, wherein: The teaching is performed based on information obtained from the learning object.

12. The information processing device according to claim 11, wherein: The teaching is performed by photographing a marking assigned to the learning object.

13. The information processing device according to claim 11, wherein: The teaching is performed based on information transmitted from the learning object using wireless communication.

14. The information processing device according to claim 11, wherein: The teaching is performed based on an inaudible sound emitted from the learning object.

15. The information processing device according to claim 9, wherein: The tag is extracted from the user's speech.

16. The information processing device according to claim 1, wherein: The learning object includes an object, and The operation control unit instructs the autonomous mobile object to capture an image of the object recognized based on the teaching.

17. The information processing device according to claim 1, wherein: The learning object includes an action of an object, and The operation control unit instructs the autonomous mobile object to obtain information on the motion of the object recognized based on the teaching.

18. The information processing device according to claim 1, wherein: The learning object includes a spatial region, and The operation control unit instructs the autonomous mobile object to obtain information on the spatial area recognized based on the teaching.

19. The information processing device according to claim 1, wherein: The operation control unit instructs the autonomous mobile object to perform an action indicating that the pattern recognition learning has started or ended.

20. The information processing device according to claim 1, wherein: The operation control unit instructs the autonomous mobile object to perform a guiding operation to guide a user to perform the teaching. 21 . The information processing apparatus according to claim 1 , further comprising an application control unit that controls a user interface used when associating a label regarding the pattern recognition learning and the learning object.

22. The information processing device according to claim 21, wherein: The application control unit provides a method for the user to designate an arbitrary designated area in the environment map and assign the tag to the designated area in the user interface.

23. The information processing device according to claim 22, wherein: The application control unit presents, in the user interface, candidates for the label that the user can assign to the designated area.

24. The information processing device according to claim 21, wherein: The application control unit provides a method for the user to correct the association between the label and the learning object in the user interface. 25 . The information processing apparatus according to claim 1 , further comprising a learning unit that performs the pattern recognition learning based on the label and the image of the learning object.

26. An information processing method, comprising: The processor controls the operation of the autonomous moving body to perform an action according to the recognition operation, Wherein, controlling the operation further includes: instructing the autonomous mobile body to obtain information about a learning object to be learned in a manner corresponding to a taught label and presented in a direction indicated by a finger of a user based on detecting the start of teaching related to pattern recognition learning, After taking an object indicated by a finger of a user as a learning object, extracting a label from a voice uttered by the user; performing the pattern recognition learning by associating the extracted label with a captured image of the learning object; instructing the autonomous mobile object to perform a first action to indicate completion of the pattern recognition learning; determining that an identified learning object identified by the pattern recognition learning has not been used by the user for more than a certain period of time; and In response to the identified learning object not being used by the user for more than a certain period of time, the autonomous mobile object is instructed to take a second action to prompt the user.

27. A computer-readable storage medium storing a program which, when executed, causes a computer to function as an information processing device, the information processing device comprising an operation control unit which controls an operation of an autonomous mobile body that performs an action according to a recognition operation, wherein upon detecting the start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile body to obtain information on a learning object to be learned in a manner corresponding to a taught label and presented in a direction indicated by a finger of a user, After taking an object indicated by a finger of a user as a learning object, extracting a label from a voice uttered by the user; performing the pattern recognition learning by associating the extracted label with a captured image of the learning object; instructing the autonomous mobile object to perform a first action to indicate completion of the pattern recognition learning; determining that an identified learning object identified by the pattern recognition learning has not been used by the user for more than a certain period of time; and In response to the identified learning object not being used by the user for more than a certain period of time, the autonomous mobile object is instructed to take a second action to prompt the user.

Images