Information processing device, information processing method, and program

Abstract

This information processing device is provided with: a sensor unit which senses environment information about a prescribed area; a storage unit which stores event information that includes event feature data pertaining to a prescribed event and meta information including spatial information about the prescribed event, which is associated with the event feature data; and a control unit which retrieves event information from the storage unit on the basis of the sensing result obtained by the sensor unit and acquires the spatial information included in the event information.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] For example, in the field of robot and agent technologies that operate in environments such as homes, technologies that recognize environmental information within the home and around the device have been developed and introduced. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-218955 [Patent Document 2] International Publication No. 2014 / 167700 Summary of the Invention [Problem to be solved by the invention]

[0004] However, not only do mobile robots operating in a given environment have significantly different visual information, but this visual information also changes from moment to moment. Furthermore, the spatial area observable by a mobile robot is limited by objects present in the given environment. Therefore, a recognition system that is not affected by differences or changes in the environment or visual occlusion in a given environment such as a house is desired.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that are capable of recognizing a specific event from a portion of information in a specific environment. [Means for solving the problem]

[0006] An information processing device of one embodiment according to the present disclosure includes a sensor unit that senses environmental information of a specified area, a memory unit that stores event information including event feature data regarding a specified event and meta-information including spatial information of the specified event associated with the event feature data, and a control unit that searches for the event information from the memory unit based on a sensing result by the sensor unit and acquires the spatial information included in the event information.

[0007] According to one aspect of the embodiment, a specific event can be recognized from a portion of information in a predetermined environment. Note that the effects described herein are not necessarily limited, and may be any of the effects described in this disclosure. Furthermore, the effects described herein are merely examples and are not limited, and other effects may be present. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an overview of a learning phase of an information processing system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a diagram illustrating an overview of a recall phase of an information processing system according to an embodiment of the present disclosure. [Diagram 3] 1 is a block diagram showing a configuration example of an information processing device according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating extraction and feature amounts of voice features according to an embodiment of the present disclosure. [Figure 5A] FIG. 1 illustrates an image sequence for the learning phase according to one embodiment of the present disclosure. [Figure 5B] FIG. 13 illustrates masking of an object region according to an embodiment of the present disclosure. [Figure 5C] FIG. 2 illustrates the extraction of object features according to one embodiment of the present disclosure. [Figure 5D] FIG. 1 is a diagram illustrating feature amounts of object features according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a diagram illustrating an example of event feature data according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a diagram illustrating a specific example of an event DB according to an embodiment of the present disclosure. [Figure 8] FIG. 13 is a diagram illustrating an example of updated event feature data according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart illustrating an example of an information processing method executed by an information processing device according to an embodiment of the present disclosure. [Figure 10] FIG. 2 is a diagram for explaining an example of initial setting of an information processing device according to an embodiment of the present disclosure. [Figure 11] FIG. 2 is a diagram for explaining an example of initial setting of an information processing device according to an embodiment of the present disclosure. [Figure 12] FIG. 13 is a diagram showing a specific example of an event DB according to a modified example of the present disclosure. [Figure 13A] FIG. 11 is a diagram showing a specific example of an information processing method according to a modified example of the present disclosure. [Figure 13B] 11A and 11B are diagrams showing specific examples of positions of an information processing device according to a modified example of the present disclosure. [Figure 13C] 11A and 11B are diagrams illustrating a specific example of movement of an information processing device according to a modified example of the present disclosure. [Figure 14] FIG. 13 is a schematic diagram of an information processing device according to a modified example. [Figure 15] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0010] (One embodiment) [System configuration according to one embodiment] As described above, in recent years, a technology has been developed for a mobile robot, such as a pet robot, to recognize predetermined environmental information in a house or around a device.

[0011] On the other hand, in environments such as homes, visual information differs greatly depending on the environment and changes from moment to moment. That is, the recognition of daily events in a given environment such as a home varies greatly depending on the environment, and the characteristics of the events differ for each environment. Therefore, a mobile robot needs to learn events for each environment. Furthermore, the space that the above-mentioned mobile robot can observe is partial at each time. That is, since the space observed by the sensors of a mobile robot differs at each time, it is necessary for the mobile robot to recognize an event by integrating the characteristics from the inputs of multiple sensors and complementing the incomplete information even if the observation is incomplete. Therefore, there has been a demand for a recognition system that is not affected by differences or changes in the environment within a home or visual occlusion.

[0012] Therefore, in the present disclosure, daily events are defined by event feature data and event meta information obtained based on inputs from multiple sensors, and are three-dimensionally mapped. This enables event recognition processing with higher robustness against differences in a given environment, such as inside a house H, and against visual occlusion. Here, the event feature data is visual information and auditory information that characterize the event itself. Specifically, the event feature data includes object feature data indicating the feature amount of an object, and audio feature data indicating the feature amount of an audio. The event meta information includes location information indicating a given location as spatial information.

[0013] First, the learning phase of the information processing device according to this embodiment will be described. FIG. 1 is a diagram showing an example of an overview of the learning phase of the information processing device according to this embodiment. As shown in FIG. 1, the information processing device 1 according to this embodiment exists in an environment of a predetermined area such as a house H. Note that, although the following describes a case where the information processing device 1 is a pet-type robot, the information processing device 1 may be a humanoid robot, a drone, or the like instead of a pet-type robot.

[0014] In the example of FIG. 1, the information processing device 1 is, for example, a pet-type robot equipped with a pet-shaped housing, and is capable of moving within the house H. The information processing device 1 as a pet-type robot learns events within the house H based on the detection results of a sensor that senses environmental information within the house H. In the example shown in FIG. 1, a case is described in which a person's visit, specifically a man's return home, is learned as an event, but the event is not necessarily limited to the visit of a person to the house H described above, and various events can be detected. Although specific details will be described later, the information processing device 1 includes a sensor unit 2 and is capable of sensing environmental information of a predetermined area. Specifically, the information processing device 1 includes a microphone sensor 21 and a camera sensor 22. The information processing device 1 senses events within the house H based on audio data as part of the sensing result collected by the microphone sensor 21 and video data as part of the sensing result captured by the camera sensor 22.

[0015] In the example shown in FIG. 1, when a male resident of a house H returns home, audio data is acquired by a microphone sensor 21, and video data is acquired by a camera sensor 22. Note that video data includes the concept of image data, since the video data is data composed of a plurality of image data. The information processing device 1 learns the correlation between the audio data and the video data and converts it into event feature data. In addition, the information processing device 1 maps the location where the learned event feature data was obtained, and learns it as event meta information including three-dimensional position information as spatial information. In the example shown in FIG. 1, the position information of the front door D in the house H is mapped as the position where the learned event feature data occurred, and is converted into event meta information.

[0016] FIG. 2 is a diagram showing an example of an outline of the recollection phase of the information processing device according to the present embodiment. As shown in FIG. 2, when the information processing device 1 acquires voice data by the microphone sensor 21, it retrieves event feature data based on the acquired voice data and retrieves location information included in the related event meta information. In the example shown in FIG. 2, first, based on the voice data of, for example, a man's footsteps and the sound of a door D opening acquired by the microphone sensor 21, the voice feature data "EA0011" is retrieved. When the voice feature data "EA0011" is retrieved, the related event feature ID "E001" and event meta information "EM001" can be retrieved. The information processing device 1 moves to the location of the location information included in the retrieved event meta information "EM001" by a driving mechanism described later. As a result, the information processing device 1, which is a pet-type robot, can automatically move to the location of a similar event that has occurred currently based on an event that has occurred in the past.

[0017] Furthermore, in the above-mentioned learning phase, when audio data and video data are acquired simultaneously, the information processing device 1 can gradually refine the event information by successively updating the event DB. Therefore, according to the information processing device 1, the user does not need to set detailed event information in advance, and the event information can be optimized with a simple operation, making it easy to optimize the event information.

[0018] [Configuration of information processing device according to embodiment] Next, a configuration example of the information processing device 1 according to an embodiment will be described. Fig. 3 is a block diagram showing a configuration example of the information processing device 1 according to the present embodiment. As shown in Fig. 2, the information processing device 1 includes a sensor unit 2, a communication unit 3, a storage unit 4, and a control unit 5.

[0019] The sensor unit 2 has a sensor that senses environmental information in a predetermined area (inside the house H). In the example shown in FIG. 2, the sensor unit 2 includes a microphone sensor 21, a camera sensor 22, and a depth sensor 23. The microphone sensor 21 is a device that collects surrounding sounds and outputs audio data converted into a digital signal via an amplifier and an ADC (Analog Digital Converter). That is, the microphone sensor 21 is assumed to be a sensor capable of receiving audio input, such as a microphone. The camera sensor 22 is an imaging device, such as an RGB camera, that has a lens system and an imaging element and captures an image (still image or moving image). It is assumed that the information acquired by the camera sensor 22, that is, the input, is an image having single color or multiple color information. The depth sensor 23 is a device that acquires depth information, such as an infrared distance measuring device, an ultrasonic distance measuring device, a LiDAR (Laser Imaging Detection and Ranging), or a stereo camera. That is, the depth sensor 23 is a so-called 3D sensor that measures the distance to a subject. The information processing device 1 may acquire sensing results of a predetermined area from a sensor unit 2 provided separately from the information processing device 1.

[0020] The communication unit 3 is a communication module that transmits and receives data to and from other devices with which it can communicate via a predetermined network. The communication unit 3 includes a receiving unit 31 and a transmitting unit 32. The receiving unit 31 receives predetermined information from other devices and outputs the information to the control unit 5. The transmitting unit 32 transmits the predetermined information to other devices via the network.

[0021] The memory unit 4 is a storage device for recording at least event information, and stores a voice feature database (DB) 41, an object mask DB 42, an object feature DB 43, an event meta information DB 44, an event feature DB 45, an event DB 46, a threshold DB 47, and recalled event meta information 48.

[0022] The voice feature data stored in the voice feature DB is information related to the feature amount of the voice data acquired by the information processing device 1. The voice feature data is, for example, a feature amount extracted by the control unit 5 described later based on the voice data acquired by the microphone sensor 21. FIG. 4 is a diagram showing a specific example of the voice feature data according to the present embodiment. As shown in FIG. 4, the voice feature data 210 in which a predetermined feature amount is extracted from the voice data 200 acquired by the microphone sensor 21 is stored in the voice feature DB 41 in the storage unit 4. Note that in the following description, the voice feature data is shown abstractly as "EA0015" and the like, but "EA0015" and the like as the voice feature data is concrete voice feature data as shown in FIG. 4.

[0023] Returning to FIG. 3, the object mask information stored in the object mask DB 42 is information of an object mask for estimating an object region for the video data acquired by the camera sensor 22. The object mask information is information that serves as a detection criterion for an object. FIG. 5A is a diagram showing a specific example of an image sequence according to this embodiment, and FIG. 5B and FIG. 5C are specific examples of object mask information obtained by detecting an object 101 included in the image sequence of the image data 100 in FIG. 5A and estimating an area in which the object exists. The storage unit 4 shown in FIG. 3 stores various object mask information shown in FIG. 5C. FIG. 5D is a diagram showing object feature data that is a feature amount of each object extracted based on each object mask information stored in the object mask DB 42 of the storage unit 4. Note that each object feature data obtained from the object mask information shown in FIG. 5C is abstracted as "EB001" in the following description, but the object feature data "EB001" is specific object feature data shown in FIG. 5D. The specific object feature data is, for example, 256-dimensional vector data. These object feature data are stored in the object feature DB 43 of the storage unit 4 shown in FIG.

[0024] 6 is a diagram showing an example of event feature data. As shown in FIG. 6, the event feature data includes object feature data and voice feature data linked to an event feature ID. The object feature data is stored in the event feature DB 45 of the storage unit 4 shown in FIG.

[0025] The event meta information also includes at least two-dimensional or three-dimensional position information. The event meta information may also include time information. In this embodiment, the event meta information is meta information including position information and time information related to a specific event. The event meta information may further include information necessary for the behavior of the mobile robot. The information necessary for the behavior of the mobile robot is, for example, category information, occurrence frequency information, and occurrence date and time information related to the event.

[0026] The event feature DB 45 is a database in which the above-mentioned voice feature data and object feature data are associated with each other and stored as event feature data. FIG. 6 is a diagram showing a specific example of event feature data. As shown in FIG. 6, the event feature data is data configured by associating object feature data and voice feature data, which are associated with each other, with an event feature ID. In the example shown in FIG. 6, object feature data "EB003" and "EB005" (see FIG. 5C and FIG. 5D) and voice feature data "EA0015" (see FIG. 4) related thereto are configured by being associated with the event feature ID "E001".

[0027] Furthermore, the event DB 46 is a database in which the above-mentioned event feature data and event meta information are associated with each other and stored as event information. FIG. 7 is a diagram showing a specific example of event information. As shown in FIG. 7, the event information is data in which an event feature ID and event meta information associated with each other are associated with an event ID. In the example shown in FIG. 7, the event information is configured by assigning an event ID "001" to an event feature ID "EM001" and event meta information "EM001" related thereto. Also, the event information is configured by assigning an event ID "002" to an event feature ID "EM002" and event meta information "EM002" related thereto.

[0028] The threshold DB 47 includes information on a threshold for determining the degree of matching between the voice data acquired by the microphone sensor 21 and the video data acquired by the camera sensor 22. This threshold for the degree of matching is referred to as a matching threshold in this specification, and is a threshold for the degree of matching between the voice feature data obtained from the voice data and the object feature data obtained from the video data. The matching threshold is also information on a threshold for determining whether or not to enter a learning phase, in other words, a threshold for determining whether or not an event should be registered. When the input voice data or video data exceeds the matching threshold, the learning phase is entered, and when it is equal to or less than the matching threshold, the recall phase is entered. Here, the learning phase is a process in which the event DB 46 is changed by a registration process or an update process performed by the control unit 5. The recall phase is a process in which the control unit 5 outputs event meta information included in predetermined event information from the event DB 46 under a predetermined condition.

[0029] The threshold DB 47 includes threshold information for determining the degree of similarity between the voice feature data registered in the voice feature DB 41 and the voice data acquired by the microphone sensor 21. The threshold DB 47 also includes threshold information for determining the degree of similarity between the object feature data registered in the object feature DB 43 and the video data acquired by the camera sensor 22. These thresholds are referred to as recall thresholds in this specification. In other words, the recall threshold is a threshold for determining whether or not event feature data including voice feature data or object feature data similar to the feature amount of the input voice data or video data is present in the event information stored in the event DB 46.

[0030] The recollected event meta information 48 is event meta information included in the event information retrieved from the event DB 46. The information processing device 1 makes an action plan based on the recollected event meta information 48.

[0031] Next, the control unit 5 will be described. The control unit 5 has a function of controlling each component included in the information processing device 1. As shown in Fig. 3, the control unit 5 includes a voice feature extraction unit 51, an object region estimation unit 52, an object feature extraction unit 53, a sound source estimation unit 54, a spatial position information acquisition unit 55, a time information acquisition unit 56, a learning and recall unit 57, and a behavior plan control unit 58.

[0032] The voice feature extraction unit 51 extracts highly abstract features from the voice data input from the microphone sensor 21 and converts them into voice feature data. The conversion process from voice data to voice feature data can be realized by a technique such as Fourier transform processing.

[0033] The object region estimation unit 52 estimates a region in which the object 101 exists as shown in FIG. 5B for the multiple image data 100 included in the video data shown in FIG. 5A acquired by the camera sensor 22, and outputs object mask information indicating the region of the object 101. As a result, as shown in FIG. 5C, the individual objects 101 included in the image data 100 are distinguished, and the respective object mask information is stored in the storage unit 4. The object feature extraction unit 53 specifies the region of each object 101 from the multiple image data 100 included in the input video data and the object mask information. The object feature extraction unit 53 extracts a highly abstract feature amount for each region of the object 101 from the specified region of the object 101 as shown in FIG. 5D, and converts it into object feature data. The object feature extraction unit 53 is stored in the storage unit 4. Note that the process of specifying the region of the object 101 and converting it into object feature data can be realized by existing technology.

[0034] Returning to FIG. 3, the sound source object estimation unit 54 calculates the degree of coincidence between the voice feature data obtained by the voice feature extraction unit 51 and each object feature data obtained by the object feature extraction unit 53. The sound source object estimation unit 54 estimates the source of the voice data detected by the voice feature extraction unit 51, i.e., the object 101 that is the sound source, based on the calculation of the degree of coincidence. Specifically, the sound source object estimation unit 54 estimates the direction in which the sound source is generated by using various direction calculation algorithms such as the MUSIC (Multiple Signal Classification) method for the voice data, and estimates the position of the object that is the sound source. The sound source object estimation unit 54 associates the object feature data and the voice feature data that are estimated to have a high degree of coincidence, and outputs them as event feature data. A method for calculating the degree of coincidence between the voice feature data and the object feature data includes, but is not necessarily limited to, a method of calculating the inner product of the object feature data and the voice feature data. The degree of match between the voice feature data and the object feature data can be calculated by, for example, a neural network obtained by machine learning. In the example shown in Fig. 5C and Fig. 6, the event feature data is assigned an event feature ID "E001", and the object feature data "EB003" and "EB005" are associated with the voice feature data "EA0015".

[0035] The microphone sensor 21, camera sensor 22, audio feature extraction unit 51, object region estimation unit 52, object feature extraction unit 53, sound source object estimation unit 54, audio feature DB 41, object mask DB 42, and object feature DB 43 constitute a feature extraction unit 70. The feature extraction unit 70 generally extracts event feature data from various data such as input audio data and video data. Meanwhile, the feature extraction unit 70 calculates the degree of coincidence between the object feature data and audio feature data, and calculates whether or not the video data contains an object that is a sound source.

[0036] The spatial position information acquisition unit 55 creates a map of a predetermined area (inside the house H) based on the depth information detected by the depth sensor 23, and stores the map information in the storage unit 4 as map information that is the basis of the event meta information. The spatial position information acquisition unit 55 can generate map information by SLAM (Simulation Localization and Mapping). The spatial position information acquisition unit 55 may update the map information at a predetermined period, assuming that furniture in the house H will be rearranged, or may generate a map every time the information processing device 1 moves. The information processing device 1 may store a map generated by another device as map information. The spatial position information acquisition unit 55 can calculate specific position information by comparing the depth information obtained by the depth sensor 23 with the map information stored in the storage unit 4. The following processing can be given as a method for acquiring the specific position information by the spatial position information acquisition unit 55. That is, it is a process of acquiring coordinate information on the earth using a positioning system such as a GPS (Global Positioning System) system, or a process of self-location estimation that acquires a relative position from a specified starting point using video data such as VisualSLAM.

[0037] The time information acquisition unit 56 is, for example, a timing mechanism such as a clock, or a time information receiving mechanism that receives time information from a server that outputs time information via a predetermined network.

[0038] The spatial position information acquisition unit 55 outputs the position information associated with the observed event as a part of the event meta information. The event meta information is stored in an event meta information database in the storage unit 4. The event meta information includes at least the position information of the event. Here, the position information of the event refers to a coordinate expression by two or more numerical values ​​with an arbitrary position as the origin. The position information can be expressed, for example, as a relative position from a predetermined origin in a map of the environment, that is, spatial information such as an XYZ position in a world coordinate system or coordinate information of the world geodetic system obtained from a GPS satellite. In addition, the time information acquired by the time information acquisition unit 56 may be associated with the position information calculated by the spatial position information acquisition unit 55 as time information of the time when the event occurred and be included as a part of the event meta information.

[0039] The depth sensor 23, spatial position information acquisition unit 55, time information acquisition unit 56, and event meta information DB 44 constitute an event meta information acquisition unit 80. Generally, the event meta information acquisition unit 80 outputs information necessary for retrieving event information and for the behavior of the mobile robot based on the input from the depth sensor 23 as event meta information and stores it in the storage unit 4.

[0040] The learning and recollection unit 57 as a part of the generation unit associates the event feature data obtained by the feature extraction unit 70 with the event meta information obtained by the event meta information acquisition unit 80 to generate event information, and stores the event information in the event DB 46 of the storage unit 4. Note that in this embodiment, the event feature data is stored in the event feature DB 45, and the event information is stored in the event DB 46, but this is not necessarily limited to this. In other words, instead of using a database, a system capable of outputting related information from a specific input, such as a Boltzmann machine or a self-organizing map, may be used.

[0041] The learning and recollection unit 57 determines whether to perform a registration process, an update process, or a recall process for the event information based on the event feature data output from the feature extraction unit 70 and the matching threshold and the recall threshold stored in the threshold DB 47.

[0042] The learning and recollection unit 57, the event feature DB 45, the event DB 46, and the threshold DB 47 constitute an event memory unit 90. Generally, the event memory unit 90 selects one of the processes of registration, update, and recollection for the event information, and also generates the event information and stores it in the storage unit 4.

[0043] The behavior plan control unit 58 has a function of planning an action to be taken by the information processing device 1 based on the information acquired by the sensor unit 2 and various data stored in the storage unit 4. The behavior plan control unit 58 according to this embodiment first retrieves, from the voice data acquired by the microphone sensor 21, event meta information stored in the event meta information DB 44 that corresponds to the voice data. The behavior plan control unit 58 then determines, based on the position information included in the retrieved event meta information, to execute an action of moving to a position specified by the position information.

[0044] Moreover, the behavior plan control unit 58 has a function of controlling the operation of the drive unit 6. The drive unit 6 has a function of driving the physical configuration in the information processing device 1. The drive unit 6 has a function for moving the position of the information processing device 1. The drive unit 6 is, for example, an actuator driven by a motor 61. The behavior plan control unit 58 controls the motor 61 of the drive unit 6 based on, for example, the above-mentioned behavior plan to drive the actuators of each joint provided in the drive unit 6. Note that the drive unit 6 may have any configuration as long as the information processing device 1 can realize a desired operation. The drive unit 6 may have any configuration as long as the information processing device 1 can realize the movement of the position, etc. When the information processing device 1 has a moving mechanism such as a caterpillar or a tire, the drive unit 6 drives the caterpillar or the tire. The drive unit 6 may further include a sensor, such as a GPS receiver or an acceleration sensor, necessary for controlling the mobile robot.

[0045] [Information processing method according to the embodiment] Next, a process procedure executed by the information processing device 1 according to the present embodiment will be described below. Fig. 9 is a flowchart showing a process procedure executed by the information processing device 1 according to the embodiment.

[0046] As shown in FIG. 9, first, in step ST1, the feature extraction unit 70 of the information processing device 1 acquires an event feature. Specifically, first, the microphone sensor 21 acquires voice data, and the camera sensor 22 acquires video data. The camera sensor 22 may acquire a plurality of image data instead of acquiring video data. The voice feature extraction unit 51 of the control unit 5 extracts voice feature data from the acquired voice data and stores it in the voice feature DB 41. The object region estimation unit 52 and the object feature extraction unit 53 extract object feature data from the video data using object mask data, and store it in the object feature DB 43. The sound source object estimation unit 54 estimates an object that is the sound source of the acquired voice data from the voice feature data and the object feature data. The voice feature data and the object feature data are combined to generate event feature data. If voice data is not obtained or object feature data cannot be extracted from the video data, the event feature data may be composed of only voice feature data or only object feature data. In parallel with the generation of the event feature data, the event meta information acquisition unit 80 generates event meta information for the location where the audio data and video data were acquired, and stores the event meta information in the event meta information DB 44 .

[0047] Next, when the process proceeds to step ST2, the event memory unit 90 of the information processing device 1 judges whether or not the generated event feature data exceeds the matching threshold. Specifically, first, the sound source object estimation unit 54 calculates the degree of matching between the voice feature data and the object feature data included in the event feature data, and outputs the degree of matching to the learning and recalling unit 57. When the learning and recalling unit 57 judges that the input degree of matching exceeds the matching threshold (step ST2: Yes), the process proceeds to step ST3. When the degree of matching between the voice feature data and the object feature data is high, the object that output the voice data is imaged by the camera sensor 22 at approximately the same time as the voice data is acquired by the microphone sensor 21. In this case, as described above, the process of the information processing device 1 enters the learning phase.

[0048] Next, in step ST3 as a learning phase, the control unit 5 of the information processing device 1 recalls events based on the event feature data. Specifically, the learning recall unit 57 of the control unit 5 retrieves event information stored in the event DB 46 based on the acquired event feature data. The event DB 46 stores event feature IDs and event meta information associated with event IDs, for example, as shown in FIG. 7.

[0049] Next, the process proceeds to step ST4, where the learning and recalling unit 57 determines whether or not there is event information having event feature data whose similarity to the acquired event feature data exceeds a predetermined recall threshold. The learning and recalling unit 57 may use a threshold based on other information included in the event meta information or a threshold based on the occurrence frequency or occurrence date and time as a recall threshold for similarity, in addition to the threshold for similarity for the event feature data. If the learning and recalling unit 57 determines that there is event information containing event feature data that exceeds the predetermined recall threshold (step ST4: Yes), the process proceeds to step ST5. The following description will be given assuming that the event feature data included in the retrieved event information is the event feature data of the event feature ID "E001" shown in FIG. 6.

[0050] In step ST5, the learning and recalling unit 57 updates the retrieved event feature data. Specifically, the learning and recalling unit 57 updates the event feature data included in the retrieved event information to the acquired event feature data. That is, for example, among the event feature data with the event feature ID "E001", the voice feature data is updated from the voice feature data "EA0015" shown in FIG. 6 to the voice feature data "EA0024" shown in FIG. 8. Note that the object feature data may be updated as necessary. The updated event feature data with the event feature ID "E001" is stored in the event ID "001" shown in FIG. 7, and the event information is updated. With the above, the learning phase executed by the information processing device 1 is completed.

[0051] Furthermore, if the learning and recalling unit 57 determines in step ST4 that there is no event information including event feature data exceeding the predetermined recall threshold (step ST4: No), the process proceeds to step ST6. In step ST6, the control unit 5 registers the event. Specifically, the learning and recalling unit 57 generates event feature data from the voice feature data and object feature data output from the feature extraction unit 70. Meanwhile, the learning and recalling unit 57 acquires event meta information output from the event meta information acquisition unit 80. The learning and recalling unit 57 associates the event feature data with the event meta information, assigns an event ID, and stores the data in the event DB 46. This ends the learning phase executed by the information processing device 1.

[0052] Furthermore, if the learning and recalling unit 57 determines in step ST2 that the calculated degree of matching is equal to or lower than the matching threshold (step ST2: No), the process proceeds to step ST7. If the degree of matching between the voice feature data and the object feature data is equal to or lower than the matching threshold, the object that output the voice data has not been imaged by the camera sensor 22 at the time the voice data was acquired by the microphone sensor 21. In this case, as described above, the process of the information processing device 1 enters the recall phase.

[0053] Next, in step ST7 as a recollection phase, the control unit 5 of the information processing device 1 recalls an event based on the voice feature data. Specifically, the learning and recalling unit 57 of the control unit 5 retrieves event information stored in the event DB 46 based on the acquired voice feature data. Note that the learning and recalling unit 57 may retrieve event information based on the acquired object feature data. The event DB 46 stores, for example, an event feature ID and event meta information associated with an event ID as shown in FIG. 7.

[0054] Next, in step ST8, the learning and recalling unit 57 judges whether or not there is event information in which the similarity between the voice feature data included in the retrieved event information and the acquired voice feature data exceeds a predetermined recall threshold. If the learning and recalling unit 57 judges that there is event information in which the voice feature data in which the similarity with the acquired voice feature data exceeds the recall threshold (step ST8: Yes), the process proceeds to step ST9. In the following explanation, for example, the case where the acquired voice feature data is "EA0015" is taken as an example.

[0055] In step ST9, the control unit 5 outputs the event meta information of the corresponding event. Specifically, the learning and recalling unit 57 first retrieves the event feature data "E001" (see FIG. 6) including the audio feature data "EA0015", and retrieves the event information of the event ID "001" shown in FIG. 7. Next, the learning and recalling unit 57 reads out the event meta information "EM001" included in the retrieved event information of the event ID "001". The learning and recalling unit 57 outputs the read event meta information "EM001" to the behavior plan control unit 58 as recalled event meta information 48. With this, the recall phase executed by the information processing device 1 ends.

[0056] The behavior plan control unit 58 to which the recollected event meta information 48 has been input executes a behavior plan based on the position information included in the recollected event meta information 48, and controls the driving unit 6. As a result, the information processing device 1 moves to a location indicated by the position information included in the recollected event meta information 48.

[0057] On the other hand, in step ST8, if the learning and recall unit 57 determines that there is no event information containing voice feature data whose similarity to the acquired voice feature data exceeds the recall threshold (step ST8: No), the recall phase executed by the information processing device 1 is terminated.

[0058] [Example] Next, a specific example of the information processing device 1 according to the embodiment will be described. In this example, a case where a husband or father returns home to the house H will be described as an example. First, as a preliminary preparation, as shown in FIG. 10, the spatial position information acquisition unit 55 of the information processing device 1 creates a map of a predetermined area (inside the house H) by, for example, VisualSLAM based on the depth information detected by the depth sensor 23. In this case, the position of, for example, a door D that emits a sound is also included in the map information. The created map is stored in the event meta information DB 44 of the storage unit 4 as map information that is the base of the event meta information. This enables the information processing device 1 to always estimate its own position from the starting point.

[0059] 11, the object feature extraction unit 53 of the information processing device 1 detects people and objects such as residents of the house H by the camera sensor 22. The detected objects 102, 103, and 104 are converted into object feature data and stored in an object feature DB in the storage unit 4. The detection and identification of objects by the object feature extraction unit 53 can be realized by utilizing known machine learning and pattern recognition techniques such as a boosting method, a neural network, and a Hidden Markov Model (HMM) method.

[0060] Thereafter, as shown in FIG. 1, when a resident of the house H returns home from outside and opens the front door D, a sound is generated at the door D. The sound may be, for example, the sound of the front door D opening and closing, or the voice of the resident returning home. The information processing device 1 acquires the sound generated at the door D as sound data. If the information processing device 1 can capture the sound generation situation as video data at approximately the same time that the resident opens the door D, the control unit 5 determines that there is a high degree of match between the video data and the audio data. In this case, the processing of the information processing device 1 transitions to the learning phase described above.

[0061] On the other hand, as shown in FIG. 2, there are cases where the information processing device 1 cannot capture the sound generation situation as video data at the time when the resident opens the front door D. In this case, the control unit 5 of the information processing device 1 determines that the degree of match between the video data and the sound data is low, and transitions to the recall phase. When transitioning to the recall phase, the information processing device 1 retrieves event information based on the input sound data and reads the recall event meta information 48. The information processing device 1 executes an action plan based on the read recall event meta information 48, and moves to a position indicated by the position information included in the recall event meta information 48. This allows the information processing device 1 to create a situation in which it responds to the generated sound and welcomes the resident who has returned home.

[0062] In this embodiment, the information processing device 1 searches for event information including similar audio data based on the acquired audio data, and moves to a position based on the associated event meta information, but may also search for the acquired video data. For example, the information processing device 1 that acquires a flash of lightning as video data may search for event information including object feature data based on the similar video data, and move to a position based on the associated event meta information.

[0063] As described above, the information processing device 1 consisting of a mobile robot can generate new event information not stored in the event DB 46 only when it acquires corresponding audio data and video data at approximately the same time and transitions to the learning phase. In this case, the generation of event information depends on chance. Therefore, various methods can be adopted to easily acquire related audio data and video data at the same time. In the above-mentioned embodiment, for example, when the position of the entrance door D is not mapped, an application installed in a mobile terminal device carried by a resident may be linked to GPS information provided by the mobile terminal device. First, the application of the mobile terminal device is set so that the resident can be notified of information about the mobile robot and the location information of the resident can be transmitted to the mobile robot. Then, when the resident approaches the house H, the mobile robot is controlled to move to a random location and wait there. In addition, when there is no resident in the house H, the mobile robot may wait at a different location each time. In addition, a behavior plan for moving in the direction of a sound may be added to the microphone sensor 21 using beamforming. Furthermore, an app on a mobile terminal device may be used to prompt a resident who is not out to greet a resident returning home together with the mobile robot.

[0064] [Variations] Next, a modified example of the above-mentioned embodiment will be described. Fig. 12 is a diagram showing a specific example of an event DB according to a modified example of the present disclosure. Fig. 13A, Fig. 13B, and Fig. 13C are diagrams showing a specific example of an information processing method and movement of an information processing device according to a modified example of the present disclosure, respectively. Here, the information processing device 1A according to the modified example is, for example, a mobile robot that operates a household appliance (hereinafter, home appliance).

[0065] As shown in Fig. 12, in the modified example, the event feature data of the object feature data and the voice feature data of each home appliance are stored in the event DB 46 of the storage unit 4 in association with event meta information including the location information of the home appliance. In the example shown in Fig. 12, for example, the event IDs "010", "011", and "012" are set corresponding to a water heater, a dishwasher, and a microwave oven, respectively, and stored in the event DB 46. Specifically, for example, for the microwave oven, the event ID "010", the event feature ID "E012", the object feature data "EB012", the voice feature data "EA0050", and the event meta information "EM012" are stored in association with each other in the event DB 46.

[0066] In the state where the event information is stored in the storage unit 4 of the information processing device 1A as described above, as shown in FIG. 13A, the information processing device 1A acquires, for example, a signal issued by a home appliance. The information processing device 1A extracts voice feature data from the acquired voice data, searches for voice feature data having a high similarity to the extracted voice feature data, and searches for event meta information associated with the voice feature data. In the example shown in FIG. 13A, voice feature data "EA0050" having a high similarity to the acquired and extracted voice feature data is searched for, and associated event meta information "EM012". This allows the information processing device 1A to recognize the position of the microwave oven, which is a home appliance. Based on the position information included in the searched event meta information "EM012", the information processing device 1A moves from the position shown in FIG. 13B to the position shown in FIG. 13C, and operates the home appliance.

[0067] [Summary of the modified example] In the above-described embodiment, the information processing device 1, 1A is installed in a predetermined area (house H), but the present invention is not limited to this. For example, the information processing device 1 can be configured as a server device.

[0068] Fig. 14 is a schematic diagram of an information processing device 300 according to a modified example. Note that Fig. 14 shows a simplified version of the information processing device 300. As shown in Fig. 14, the information processing device 300 according to the modified example is a server device, and includes an event meta information DB 144, an event feature DB 145, and an event DB 146.

[0069] The information processing device 300 receives, for example, audio data and video data as a sensing result of the environmental information transmitted from the pet robot 400. The pet robot 400 includes a sensor unit 2, a driving unit 6 capable of moving to a position specified by input position information, and a driving control unit for driving the driving unit 6. The information processing device 300 controls the behavior of the pet robot 400 based on the event meta information stored in the event meta information DB and the event feature data stored in the event feature DB 145. The information processing device 300 transmits position information to which the pet robot 400 should move based on the received audio data or video data. The pet robot 400 that has received the position information moves to a position included in the received position information. Note that, although the case where the information processing device 300 receives the sensing result from the pet robot 400 has been described, the present invention is not necessarily limited to this.

[0070] Furthermore, the information processing device 300 and a mobile terminal device 500 carried by a user may be made to be capable of communicating with each other, so that the movement of the pet robot 400 may be controlled by the mobile terminal device 500 .

[0071] [Other variations] Incidentally, in the above embodiment, the predetermined area has been described as the house H, but the present invention is not limited to this, and any area can be set as the predetermined area.

[0072] Information devices such as the information processing device, HMD, and controller according to each of the above-mentioned embodiments are realized by a computer 1000 having a configuration as shown in FIG. 15, for example. The information processing device 1 according to the embodiment will be described below as an example. FIG. 15 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 1. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0073] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0074] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0075] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a program according to the present disclosure, which is an example of program data 1450.

[0076] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0077] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0078] For example, when the computer 1000 functions as the information processing device 1 according to the embodiment, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize functions such as the spatial position information acquisition unit 55. The HDD 1400 stores the program according to the present disclosure and data in the storage unit 4. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may acquire these programs from other devices via an external network 1550.

[0079] Up until now, it has been difficult to use information processing devices such as mobile robots in environments such as inside a house H because they cannot recall both images and sounds in association with spatial information or because there are limitations on the information that can be input. In contrast, according to the embodiment described above, both image data and sound data can be stored in a mutually recallable state by linking them with spatial information. As a result, by acquiring only sound data or only video data, all other information, specifically sound data, image data, and event meta information, can be retrieved and used for controlling the behavior of the mobile robot. Furthermore, even if the information processing device 1, 1A such as a mobile robot is in an environment where it cannot acquire video data, it can move to a location where an event occurs if it is in a situation where it can acquire sound data. Similarly, even if the information processing device 1, 1A such as a mobile robot is in an environment where it cannot acquire sound data, it can move to a location where an event occurs if it is in a situation where it can acquire video data. Furthermore, since event information is registered or continuously updated at a timing where it can acquire sound data and video data simultaneously, the information processing device 1, 1A can operate robustly against changes in the environment. In addition, since the environment, such as objects within the house H, changes from moment to moment, by transitioning to the learning phase when audio data and video data are acquired simultaneously, the robot can adapt to changes in the environment from the next time onwards.

[0080] The present technology can also be configured as follows. (1) A sensor unit that senses environmental information of a predetermined area; A storage unit in which event information including event feature data relating to a predetermined event and meta information including spatial information of the predetermined event associated with the event feature data is stored; a control unit that searches for the event information from the storage unit based on a sensing result by the sensor unit, and acquires the spatial information included in the event information; An information processing device comprising: (2) The control unit is determining a similarity between a sensing result obtained by the sensor unit and the event feature data stored in the storage unit; When the similarity exceeds a predetermined recall threshold, event information including event feature data exceeding the recall threshold is retrieved from the storage unit. The information processing device according to (1). (3) The event feature data includes object feature data obtained from an object that can be sensed by the sensor unit, and voice feature data obtained from a voice that can be sensed by the sensor unit. The information processing device according to (1) or (2). (4) The control unit is Based on the voice feature data obtained from the voice sensed by the sensor unit, event information including voice feature data whose similarity to the voice feature data exceeds a predetermined recall threshold is retrieved from the storage unit. The information processing device according to (3). (5) The control unit is Based on object feature data obtained from an object sensed by the sensor unit, event information including object feature data whose similarity to the object feature data exceeds a predetermined recall threshold is retrieved from the storage unit. The information processing device according to (3). (6) the object feature data is a feature amount of an object sensed by the sensor unit, The voice feature data is a feature quantity of a voice emitted from an object sensed by the sensor unit. The information processing device according to any one of (3) to (5). (7) The mobile robot is configured to be controllable, the mobile robot being provided with a drive unit that moves a housing, The control unit is A behavior plan is made based on the acquired spatial information, and the mobile robot is controlled to behave in accordance with the behavior plan. The information processing device according to any one of (1) to (6). (8) It is a mobile robot. The information processing device according to any one of (1) to (7). (9) The computer Based on a sensing result by a sensor unit that senses environmental information of a predetermined area, the event information is retrieved from a storage unit in which event information including event feature data related to a predetermined event and meta-information including spatial information of the predetermined event associated with the event feature data is stored, and the spatial information included in the event information is output. Information processing methods. (10) Computer, A sensor unit that senses environmental information of a predetermined area; A storage unit in which event information including event feature data relating to a predetermined event and meta information including spatial information of the predetermined event associated with the event feature data is stored; a control unit that searches for the event information from the storage unit based on a sensing result by the sensor unit, and outputs the spatial information included in the event information; A program that functions as a (11) A sensor unit that senses environmental information of a predetermined area; a generation unit that generates event information by associating event feature data related to a predetermined event obtained based on a sensing result by the sensor unit with meta information including spatial information of the predetermined event obtained based on the sensing result; and An information processing device comprising: (12) the event feature data includes object feature data obtained from an object that can be sensed by the sensor unit, and sound feature data obtained from a sound that can be sensed by the sensor unit, The control unit is determining a degree of match between the object feature data and the voice feature data obtained based on the sensing result; When the degree of coincidence exceeds a predetermined coincidence threshold, the generating unit generates the event information. The information processing device according to (10). (13) The computer Event feature data relating to a predetermined event, obtained based on a sensing result by a sensor unit that senses environmental information of a predetermined area, and meta information including spatial information of the predetermined event, obtained based on the sensing result, are associated with each other to generate event information. Information processing methods. (14) Computer, A sensor unit that senses environmental information of a predetermined area; a generation unit that generates event information by associating event feature data related to a predetermined event obtained based on a sensing result by the sensor unit with meta information including spatial information of the predetermined event obtained based on the sensing result; and A program that functions as a [Explanation of symbols]

[0081] 1,1A,300 Information processing equipment 2 Sensor section 3. Communications Department 4 Storage section 5. Control section 6 Drive unit 21 Microphone Sensor 22 Camera Sensor 23 Depth Sensor 41 Audio feature DB 42 Object Mask DB 43 Object Feature DB 44,144 Event Metadata DB 45,145 Event Features DB 46,146 Event DB 47 Threshold DB 48 Recall Event Meta Information 51 Voice feature extraction unit 52 Object region estimation section 53 Object feature extraction unit 54 Sound source object estimation unit 55 Spatial position information acquisition unit 56 Time information acquisition section 57 Learning and Recall Section 58 Action Plan Control Unit 70 Feature Extraction Unit 80 Event Meta Information Acquisition Unit 90 Event Memory Section

Claims

1. A sensor unit that senses environmental information of a predetermined area; A storage unit in which event information including event feature data relating to a predetermined event and meta information including spatial information of the predetermined event associated with the event feature data is stored; a control unit that searches for the event information from the storage unit based on a sensing result by the sensor unit, and acquires the spatial information included in the event information; Equipped with The control unit is determining a similarity between a sensing result obtained by the sensor unit and the event feature data stored in the storage unit; when the degree of similarity exceeds a predetermined recall threshold, retrieves event information including event feature data exceeding the recall threshold from the storage unit, and updates the event feature data included in the retrieved event information based on a sensing result obtained by the sensor unit. Information processing device.

2. The control unit: updating the event feature data included in the retrieved event information to event feature data based on a sensing result sensed by the sensor unit; The information processing device according to claim 1 .

3. The event feature data includes object feature data obtained from an object that can be sensed by the sensor unit, and voice feature data obtained from a voice that can be sensed by the sensor unit. The information processing device according to claim 1 .

4. The control unit is Based on the voice feature data obtained from the voice sensed by the sensor unit, event information including voice feature data whose similarity to the voice feature data exceeds a predetermined recall threshold is retrieved from the storage unit. The information processing device according to claim 3 .

5. The control unit is Based on object feature data obtained from an object sensed by the sensor unit, event information including object feature data whose similarity to the object feature data exceeds a predetermined recall threshold is retrieved from the storage unit. The information processing device according to claim 3 .

6. the object feature data is a feature amount of an object sensed by the sensor unit, The voice feature data is a feature quantity of a voice emitted from an object sensed by the sensor unit. The information processing device according to claim 3 .

7. The mobile robot is configured to be controllable, the mobile robot being provided with a drive unit that moves a housing, The control unit is A behavior plan is made based on the acquired spatial information, and the mobile robot is controlled to behave in accordance with the behavior plan. The information processing device according to claim 1 .

8. It is a mobile robot. The information processing device according to claim 1 .

9. The computer based on a sensing result by a sensor unit that senses environmental information of a predetermined area, searching for the event information from a storage unit in which event information including event feature data relating to a predetermined event and meta-information including spatial information of the predetermined event associated with the event feature data is stored, and outputting the spatial information included in the event information; determining a similarity between a sensing result obtained by the sensor unit and the event feature data stored in the storage unit; when the degree of similarity exceeds a predetermined recall threshold, retrieves event information including event feature data exceeding the recall threshold from the storage unit, and updates the event feature data included in the retrieved event information based on a sensing result obtained by the sensor unit. Information processing methods.

10. Computer, A sensor unit that senses environmental information of a predetermined area; A storage unit in which event information including event feature data relating to a predetermined event and meta information including spatial information of the predetermined event associated with the event feature data is stored; a control unit that searches for the event information from the storage unit based on a sensing result by the sensor unit, and outputs the spatial information included in the event information; Function as a The control unit is determining a similarity between a sensing result obtained by the sensor unit and the event feature data stored in the storage unit; when the degree of similarity exceeds a predetermined recall threshold, retrieves event information including event feature data exceeding the recall threshold from the storage unit, and updates the event feature data included in the retrieved event information based on a sensing result obtained by the sensor unit. program.

Images