Information processing method, information processing device, and information processing program

Abstract

To provide an information processing method capable of causing a self-propelled apparatus to perform a predetermined operation according to the situation in which an article has been damaged, an information processing apparatus, and an information processing program.SOLUTION: An information processing method in a server device 3 includes the steps of: obtaining a piece of first information obtained in at least one of the one or more sensors installed in a space; detecting damage of an article existing in the space based on the first information; estimating the situation where the article is damaged based on the first information; and outputting a piece of second information for causing the self-propelled cleaner 21 to execute a predetermined operation in the space according to the estimated situation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing method, an information processing device, and an information processing program for causing a device to execute a predetermined operation. [Background technology]

[0002] Conventionally, there is known a vacuum cleaner that performs image recognition by comparing an image captured by an imaging unit with an image of a registered foreign object stored in a memory unit during a cleaning operation, thereby recognizing the registered foreign object stored in the memory unit (see, for example, Patent Document 1). When this vacuum cleaner recognizes a registered foreign object, it controls the suction drive unit based on a control mode stored in the memory unit corresponding to the recognized foreign object, and displays an indication on the display screen that identifies what the recognized foreign object is. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5771885 specification Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology does not estimate the circumstances under which damage to an item occurs, and further improvement is needed.

[0005] The present disclosure has been made to solve the above problems, and aims to provide an information processing method, an information processing device, and an information processing program that can cause a self-propelled device to perform a predetermined operation depending on the situation in which damage to an item occurs. [Means for solving the problem]

[0006] An information processing method according to one embodiment of the present disclosure is an information processing method in an information processing device, which acquires first information acquired by at least one of one or more sensors installed in a space, detects damage to an item present in the space based on the first information, estimates the circumstances under which the damage to the item occurred based on the first information, and outputs second information for causing a self-propelled device to perform a predetermined operation within the space according to the estimated circumstances. Effect of the Invention

[0007] According to the present disclosure, when an item present in a space is damaged, the self-propelled device can be caused to perform a predetermined operation depending on the circumstances under which the damage to the item occurred. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing a configuration of a device control system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of a server device according to the first embodiment of the present disclosure. [Diagram 3] 4 is a diagram showing an example of appliance operation information stored in an appliance operation information storage unit in the first embodiment. FIG. [Figure 4] 4 is a diagram showing an example of product information stored in an product information storage unit in the first embodiment. FIG. [Diagram 5] FIG. 2 is a diagram showing an example of movie information stored in a service information storage unit in the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of restaurant information stored in a service information storage unit in the first embodiment. [Figure 7] 4 is a first flowchart illustrating an operation of the server device according to the first embodiment of the present disclosure. [Figure 8] 11 is a second flowchart illustrating an operation of the server device according to the first embodiment of the present disclosure. [Figure 9]FIG. 11 is a diagram for explaining the operation of a device in a first situation in which an object has slipped out of a person's hand during daily activity in the first embodiment. [Figure 10] FIG. 11 is a diagram for explaining the operation of devices in a second situation in which a plurality of people are fighting in the first embodiment. [Figure 11] FIG. 11 is a diagram for explaining the operation of the device in a third situation in which a suspicious person has intruded in the first embodiment. [Figure 12] FIG. 11 is a diagram illustrating a configuration of a first server device according to a second embodiment of the present disclosure. [Figure 13] FIG. 11 is a diagram illustrating a configuration of a second server device according to the second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (Findings on which this disclosure is based) In the above-mentioned conventional technology, images of foreign objects other than the object to be cleaned are registered in advance in a storage unit, and the image captured during the cleaning operation is compared with the image of the foreign object to recognize the registered foreign object, and a display that identifies what the recognized foreign object is is displayed on the display screen. From the viewpoint of preventing the vacuum cleaner from breaking down or being damaged, the foreign object includes "plastic bags," "documents," "cords," and "screws," while from the viewpoint of ensuring the cleanliness of the foreign object to be sucked in and preventing it from being soiled or damaged, the foreign object includes "documents," "microSD cards," "banknotes," and "jewelry." In other words, in the conventional technology, foreign objects other than the object to be cleaned are recognized, but cleaning objects such as, for example, broken mugs are not recognized.

[0010] In addition, in the prior art, when an item is damaged, the circumstances under which the damage occurred are not estimated, and therefore the prior art does not disclose or suggest how to cause a vacuum cleaner to perform a predetermined operation in accordance with the circumstances under which the damage occurred.

[0011] In order to solve the above problems, an information processing method according to one embodiment of the present disclosure is an information processing method in an information processing device, which acquires first information acquired by at least one of one or more sensors installed in a space, detects damage to an item present in the space based on the first information, estimates the circumstances under which the damage to the item occurred based on the first information, and outputs second information for causing a self-propelled device to perform a predetermined operation within the space according to the estimated circumstances.

[0012] According to this configuration, the situation in which the damage to the item occurred is estimated based on the first information acquired by at least one of the one or more sensors installed in the space, and second information for causing the self-propelled device to execute a predetermined operation in the space is output according to the estimated situation. Therefore, when an item present in the space is damaged, it is possible to cause the self-propelled device to execute a predetermined operation according to the situation in which the damage to the item occurred.

[0013] In the above information processing method, the second information may be information for causing the self-propelled device to output a predetermined sound within the space in accordance with the estimated situation.

[0014] According to this configuration, the self-propelled device can be made to output a predetermined sound within the space depending on the situation in which the damage to the item occurred. For example, if the situation in which the damage to the item occurred is a situation in which multiple people are fighting, the self-propelled device can be made to perform an operation of moving the self-propelled device while outputting a sound to calm the multiple people who are fighting.

[0015] In the above information processing method, third information may be output to cause a presentation device to present information for changing the estimated situation.

[0016] According to this configuration, the third information for causing the presentation device to present information for changing the estimated situation is output, so that the situation in which the damage to the item occurred can be changed. For example, if the situation in which the damage to the item occurred is a situation in which multiple people are fighting, information suitable for the multiple people fighting to make up can be presented.

[0017] In addition, in the above information processing method, the self-propelled device may be a self-propelled vacuum cleaner, and the second information may be information for causing the self-propelled vacuum cleaner to clean the damaged items within the space depending on the estimated situation.

[0018] According to this configuration, the self-propelled device is a self-propelled vacuum cleaner, and the self-propelled vacuum cleaner can be made to clean the damaged item in the space depending on the situation in which the damaged item occurred. For example, if the situation in which the damaged item occurred is a situation in which the item slipped out of a person's hand during daily activities, the self-propelled vacuum cleaner can be made to perform an operation to clean the damaged item.

[0019] In addition, in the above information processing method, the estimated situation may be a situation in which a suspicious person has intruded into the space, and the second information may be information for causing the self-propelled device to perform an operation to disrupt the suspicious person within the space.

[0020] According to this configuration, in the case where damage to an article occurs due to a suspicious person entering a space, it is possible to cause the self-propelled device to perform an operation to disrupt the suspicious person within the space.

[0021] In addition, in the above information processing method, image data of the suspicious person may be acquired from an imaging device arranged within the space, and the acquired image data and notification information for notifying the presence of the suspicious person may be transmitted.

[0022] According to this configuration, image data of a suspicious person is obtained from an imaging device placed within the space, and the obtained image data and notification information for notifying the presence of a suspicious person are transmitted, thereby making it possible to notify others of the presence of a suspicious person.

[0023] In the above information processing method, when damage information indicating that the self-propelled device is damaged is acquired, fourth information for requesting repair of the self-propelled device may be output.

[0024] According to this configuration, if the self-propelled device is damaged, it is possible to automatically request repair of the self-propelled device.

[0025] In the above information processing method, fifth information may be output to cause a presentation device to present information about an article for suppressing the occurrence of the estimated situation.

[0026] According to this configuration, since the information on the item for suppressing the occurrence of a situation in which the item is damaged is presented by the presentation device, it is possible to suppress the occurrence of a situation in which the item is damaged. For example, if the situation in which the item is damaged is a situation in which a suspicious person has entered a space, it is possible to present information on the security product as the item for suppressing the occurrence of the situation in which a suspicious person has entered a space.

[0027] In addition, in the above information processing method, the one or more sensors may include at least one of a microphone device and an imaging device installed in the space, the first information may include at least one of sound data acquired by the microphone device and image data acquired by the imaging device, and the estimation of the situation may be based on at least one of the sound data and the image data, and the estimation of the situation in which damage to the item occurred may be based on at least one of the sound data and the image data.

[0028] With this configuration, the circumstances under which damage to an item occurred can be estimated with high accuracy based on at least one of sound data acquired by a microphone device installed in the space and image data acquired by an imaging device installed in the space.

[0029] In addition, in the above information processing method, the first information may be obtained at a predetermined time interval, and the situation may be estimated based on a plurality of first information obtained within a predetermined period based on the time when the damage to the item occurred, to estimate the situation in which the damage to the item occurred.

[0030] According to this configuration, the first information is acquired at a predetermined time interval, and the circumstances under which the damage to the item occurred are estimated based on multiple pieces of first information acquired within a predetermined period based on the time when the damage to the item occurred. Therefore, for example, the circumstances under which the damage to the item occurred can be estimated with greater accuracy using image data from a predetermined period of time ago from the time when the damage to the item occurred.

[0031] An information processing device according to another aspect of the present disclosure includes an acquisition unit that acquires first information acquired by at least one of one or more sensors installed within a space, a detection unit that detects damage to an item present within the space based on the first information, an estimation unit that estimates the circumstances under which the damage to the item occurred based on the first information, and an output unit that outputs second information for causing a self-propelled device to perform a predetermined operation within the space in accordance with the estimated circumstances.

[0032] According to this configuration, the situation in which the damage to the item occurred is estimated based on the first information acquired by at least one of the one or more sensors installed in the space, and second information for causing the self-propelled device to execute a predetermined operation in the space is output according to the estimated situation. Therefore, when an item present in the space is damaged, it is possible to cause the self-propelled device to execute a predetermined operation according to the situation in which the damage to the item occurred.

[0033] An information processing program according to another aspect of the present disclosure causes a computer to acquire first information obtained by at least one of one or more sensors installed within a space, detect damage to an item present within the space based on the first information, estimate the circumstances under which the damage to the item occurred based on the first information, and output second information for causing a self-propelled device to perform a predetermined operation within the space depending on the estimated circumstances.

[0034] According to this configuration, the situation in which the damage to the item occurred is estimated based on the first information acquired by at least one of the one or more sensors installed in the space, and second information for causing the self-propelled device to execute a predetermined operation in the space is output according to the estimated situation. Therefore, when an item present in the space is damaged, it is possible to cause the self-propelled device to execute a predetermined operation according to the situation in which the damage to the item occurred.

[0035] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments are examples of the present disclosure and do not limit the technical scope of the present disclosure.

[0036] (Embodiment 1) 1 is a diagram showing a configuration of a device control system according to the first embodiment of the present disclosure. As shown in FIG 1, the device control system includes a server device 3, a gateway (GW) 5, a first sensor 11, a second sensor 12, a self-propelled vacuum cleaner 21, and a display device 22.

[0037] The gateway 5, the first sensor 11, the second sensor 12, the self-propelled vacuum cleaner 21, and the display device 22 are disposed in a house 10. The gateway 5 is connected to the first sensor 11, the second sensor 12, the self-propelled vacuum cleaner 21, and the display device 22 so as to be able to communicate wirelessly. The gateway 5 is also connected to the server device 3 via a network 4 so as to be able to communicate with each other. The network 4 is, for example, the Internet.

[0038] The first sensor 11, the second sensor 12, the self-propelled vacuum cleaner 21, and the display device 22 are connected to the server device 3 via the gateway 5 so as to be able to communicate with each other. Note that the first sensor 11, the second sensor 12, the self-propelled vacuum cleaner 21, and the display device 22 may be connected to the server device 3 so as to be able to communicate directly with each other without going through the gateway 5.

[0039] Fig. 2 is a diagram showing a configuration of a server device in the first embodiment of the present disclosure. The server device 3 is communicatively connected to a sensor group 1 including a plurality of sensors arranged in a house 10, and an equipment group 2 including a plurality of equipment arranged in the house 10. The sensor group 1 includes various sensors such as a first sensor 11 and a second sensor 12. The equipment group 2 includes various equipment such as a self-propelled vacuum cleaner 21, a display device 22, and an information device 23. Note that the gateway 5 is omitted in Fig. 2.

[0040] The first sensor 11 is, for example, a microphone device that collects sounds inside the house 10 and transmits the sound data to the server device 3. The second sensor 12 is, for example, an imaging device that captures images inside the house 10 and transmits the image data to the server device 3. The sensor group 1 may include a thermal image sensor and a vibration sensor. The sensors that make up the sensor group 1 may be installed on the walls, floors, and furniture of the house 10, or may be mounted on any of the devices in the device group 2.

[0041] The self-propelled vacuum cleaner 21 is an example of a self-propelled device, and performs suction cleaning while moving autonomously. The self-propelled vacuum cleaner 21 cleans the floor surface while moving autonomously on the floor surface in the house 10. Normally, the self-propelled vacuum cleaner 21 is connected to a charging device (not shown) installed at a predetermined location in the house 10, and when a cleaning start button provided on the main body of the self-propelled vacuum cleaner 21 is pressed by a user or when cleaning instruction information is received from the server device 3, the self-propelled vacuum cleaner 21 leaves the charging device and starts cleaning. The self-propelled vacuum cleaner 21 includes a control unit, a camera, a speaker, a driving unit, a cleaning unit, and a communication unit, all of which are not shown.

[0042] The control unit controls the cleaning operation by the self-propelled vacuum cleaner 21. The drive unit moves the self-propelled vacuum cleaner 21. The drive unit includes drive wheels for moving the self-propelled vacuum cleaner 21 and a motor for driving the drive wheels. The drive wheels are installed at the bottom of the self-propelled vacuum cleaner 21. The cleaning unit is installed at the bottom of the self-propelled vacuum cleaner 21 and sucks up objects to be sucked.

[0043] The camera captures an image in the traveling direction of the self-propelled vacuum cleaner 21. The communication unit transmits image data captured by the camera to the server device 3. The communication unit also receives cleaning instruction information for starting cleaning from the server device 3. The control unit starts cleaning when the communication unit receives the cleaning instruction information. The cleaning instruction information includes a damage position where an item 6 is damaged in the house 10. The damage position is a position where an item 6 such as a mug or a plate is damaged. After moving to the damage position, the self-propelled vacuum cleaner 21 captures an image of the suction target present at the damage position and transmits the captured image data to the server device 3. The self-propelled vacuum cleaner 21 then cleans the damage position and returns to the charging device.

[0044] The speaker outputs a predetermined sound depending on the situation in which the damage to the item occurred. For example, if the damage to the item occurred because multiple people were fighting, the speaker outputs a sound to calm the multiple people fighting.

[0045] In the first embodiment, the appliance control system includes the self-propelled vacuum cleaner 21 as an example of a self-propelled device, but the present disclosure is not particularly limited thereto, and the appliance control system may include a self-propelled robot such as a pet-type robot as an example of a self-propelled device. The self-propelled robot has a function other than the cleaning function of the self-propelled vacuum cleaner 21.

[0046] The display device 22 is arranged on a wall of a predetermined room in the house 10. The device control system in the first embodiment may include a plurality of display devices 22. The plurality of display devices 22 may be arranged on the walls of each room, such as a living room, kitchen, bedroom, bathroom, toilet, and entrance hall. The display device 22 may be an information terminal such as a smartphone or a tablet computer. The display device 22 includes a communication unit, a display unit, and an input unit, which are not shown.

[0047] The communication unit receives information indicating the state of the devices from the devices installed in the house 10. The communication unit also receives presentation information from the server device 3.

[0048] The display unit is, for example, a liquid crystal display device, and displays various information. The display unit displays information related to the devices arranged in the house 10. For example, the display unit displays the current status of a washing machine or the current status of an air conditioner. The display unit also displays presentation information received by the communication unit.

[0049] The input unit is, for example, a touch panel, and accepts input operations by a user. The input unit accepts input of operation instructions for devices arranged in the house 10. The input unit accepts input of operation instructions for air conditioning devices and input of operation instructions for lighting devices, for example. The communication unit transmits the operation instructions input by the input unit to the devices.

[0050] The information device 23 is, for example, a smartphone, a tablet computer, a personal computer, or a mobile phone, and has a function of communicating with the outside. The information device 23 includes a communication unit (not shown). The communication unit receives image data of a suspicious person who has invaded the house 10 and notification information for notifying the presence of the suspicious person from the server device 3, and transmits the received image data and notification information to a server device managed by the police. The server device 3 transmits the image data and notification information to the server device managed by the police via the information device 23 in the house 10, so that the police can identify the sender of the image data and notification information.

[0051] In this embodiment 1, the server device 3 transmits image data and notification information to a server device managed by the police via the information device 23, but the present disclosure is not limited to this, and the server device 3 may transmit the image data and notification information directly to the server device managed by the police without going through the information device 23.

[0052] The device group 2 includes a washing machine, lighting equipment, air conditioner, electric shutter, electric lock, air purifier, etc. in addition to the self-propelled vacuum cleaner 21, the display device 22, and the information device 23. The devices constituting the device group 2 include, for example, home appliances, information devices, and home equipment devices.

[0053] The server device 3 includes a communication unit 31, a processor 32, and a memory 33.

[0054] The communication unit 31 includes a sensor data receiving unit 311, a control information transmitting unit 312, a presentation information transmitting unit 313, and a notification information transmitting unit 314. The processor 32 includes a damage detecting unit 321, a situation estimating unit 322, a damage position identifying unit 323, an appliance operation determining unit 324, a damaged item identifying unit 325, a substitute item identifying unit 326, a presentation information generating unit 327, a service identifying unit 328, and a notification information generating unit 329. The memory 33 includes an appliance operation information storage unit 331, an item information storage unit 332, and a service information storage unit 333.

[0055] The sensor data receiving unit 311 obtains sensor data (first information) obtained by at least one of one or more sensors installed in the house 10 (space). The sensor data receiving unit 311 receives sensor data from each sensor of the sensor group 1. The sensor data (first information) includes sound data obtained by the first sensor 11 (microphone device) and image data obtained by the second sensor 12 (imaging device). The sensor data receiving unit 311 receives the sound data from the first sensor 11 as sensor data, and receives the image data from the second sensor 12 as sensor data.

[0056] Furthermore, the sensor data receiving unit 311 receives sensor data from each device in the device group 2. Some devices in the device group 2 are equipped with sensors. The devices equipped with sensors transmit the sensor data to the server device 3. As described above, the self-propelled vacuum cleaner 21 is equipped with a camera. Therefore, the sensor data receiving unit 311 receives image data from the self-propelled vacuum cleaner 21 as sensor data. Furthermore, the display device 22 may be equipped with a microphone and a camera, and the sensor data receiving unit 311 may receive sound data and image data from the display device 22 as sensor data.

[0057] The damage detection unit 321 detects damage to an item present in the house 10 based on the sensor data (first information) received by the sensor data receiving unit 311. The damage detection unit 321 detects that an item has been damaged when the sound data received from the first sensor 11 contains characteristics of a sound generated when an item is damaged. The memory 33 may store in advance frequency components of a plurality of damage sounds, such as the sound of ceramic or glass breaking. The damage detection unit 321 compares the frequency components of the sound data received from the first sensor 11 with the frequency components of the plurality of damage sounds stored in the memory 33, and detects that an item has been damaged when the two frequency components match.

[0058] The damage detection unit 321 may estimate that damage to the article has occurred from the sound data received from the first sensor 11, using a prediction model that has been machine-learned using the sound data at the time when the damage to the article occurred and the occurrence of the damage to the article as training data. In this case, the prediction model is stored in the memory 33 in advance.

[0059] Furthermore, the damage detection unit 321 may detect that an article has been damaged from image data captured by the second sensor 12. For example, the sensor data receiving unit 311 may acquire temporally continuous image data from the second sensor 12. The damage detection unit 321 may analyze the acquired image data, and detect that the article has been damaged when the acquired image data includes a state in which an article has fallen from the hand of a person in the house 10 and broken on the floor.

[0060] The damage detection unit 321 may also detect that an article is damaged using sensor data from other sensors such as a vibration sensor. The damage detection unit 321 may also detect that an article is damaged using sensor data from multiple sensors in the sensor group 1.

[0061] The situation estimation unit 322 estimates the situation in which the damage to the item occurred based on the sensor data (first information). The situation estimation unit 322 estimates the situation in which the damage to the item occurred based on at least one of the sound data and the image data. That is, the sensor data receiving unit 311 acquires image data at a predetermined time interval. The situation estimation unit 322 estimates the situation in which the damage to the item occurred based on a plurality of image data acquired within a predetermined period based on the time point when the damage to the item occurred. The damage detection unit 321 can identify the time point when the damage to the item occurred by recognizing a characteristic component of the sound of the damage to the item from the sound data. The situation estimation unit 322 estimates the situation in which the damage to the item occurred based on a plurality of image data acquired within a predetermined period in the past from the time point when the specified damage to the item occurred.

[0062] For example, if an object slips out of a person's hand during a daily activity, the object is damaged. Also, for example, if multiple people are fighting, one of the multiple people throws an object, which damages the object. Also, for example, if a suspicious person intrudes into the house 10, the suspicious person destroys the object, which damages the object. Therefore, as situations in which damage to an object occurs, there are, for example, a first situation in which an object slips out of a person's hand during a daily activity, a second situation in which multiple people are fighting, and a third situation in which a suspicious person intrudes. The situation estimation unit 322 estimates whether the situation in which damage to an object occurs is one of the first situation in which an object slips out of a person's hand during a daily activity, the second situation in which multiple people are fighting, and the third situation in which a suspicious person intrudes.

[0063] The situation estimation unit 322 analyzes a plurality of image data immediately before the time when the damage to the item occurs, and recognizes the person's hand and the item contained in the plurality of image data. Then, when the item falls from the hand of the person, the situation estimation unit 322 estimates that the situation in which the damage to the item occurs is the first situation in which the item slips out of the person's hand during a daily movement. Note that the situation estimation unit 322 may acquire sound data immediately before the time when the damage to the item occurs, and when the sound data includes a voice of the person being surprised, estimate that the situation in which the damage to the item occurs is the first situation in which the item slips out of the person's hand during a daily movement.

[0064] Moreover, the situation estimation unit 322 analyzes a plurality of image data immediately before the time when the damage to the item occurs, and recognizes the hands of the plurality of people and the item contained in the plurality of image data. Then, when one of the plurality of people throws an item, the situation estimation unit 322 estimates that the situation in which the damage to the item occurs is the second situation in which the plurality of people are fighting. Note that the situation estimation unit 322 may acquire sound data immediately before the time when the damage to the item occurs, and when the sound data includes the voices of the plurality of people arguing, estimate that the situation in which the damage to the item occurs is the second situation in which the plurality of people are fighting.

[0065] The situation estimation unit 322 may also acquire sound data immediately before the time when the damage to the item occurs, and if the sound data includes a volume threshold or more for the voices of the multiple people, estimate that the situation in which the damage to the item occurs is a second situation in which multiple people are fighting. The situation estimation unit 322 may also acquire multiple image data immediately before the damage to the item occurs, and recognize the movements of the multiple people fighting included in the multiple image data. The situation estimation unit 322 may also detect vibrations generated when multiple people are fighting using a vibration sensor.

[0066] Furthermore, the situation estimation unit 322 analyzes a plurality of image data immediately before the time when the damage to the item occurred, and recognizes a person included in the plurality of image data. Then, when the situation estimation unit 322 recognizes a person who is not a resident of the pre-registered house 10, it estimates that the situation in which the damage to the item occurred is the third situation in which a suspicious person has intruded. Note that, when the situation estimation unit 322 recognizes a person who is not a resident of the pre-registered house 10 and does not recognize a resident of the pre-registered house 10, it may estimate that the situation in which the damage to the item occurred is the third situation in which a suspicious person has intruded.

[0067] The situation estimation unit 322 may estimate the situation in which the damage to the article occurred from the image data immediately before the damage to the article occurred, using a prediction model that has been machine-learned using the image data immediately before the damage to the article and the situation in which the damage to the article occurred as training data. The prediction model is stored in the memory 33 in advance.

[0068] It should be noted that the circumstances in which damage to an article occurs in the present embodiment 1 are not limited to the above first to third circumstances.

[0069] The damage position identifying unit 323 identifies the damage position of the item in the house 10. The memory 33 may store in advance, for example, a floor plan of the house 10 expressed in a two-dimensional coordinate space. The self-propelled vacuum cleaner 21 may create the floor plan by moving around the house 10 and transmit the created floor plan to the server device 3. The damage position identifying unit 323 identifies the coordinates on the floor plan of the source of the sound that has damaged the item as the damage position.

[0070] The damage position identifying unit 323 can identify the source of the sound of the damage to the item more accurately by collecting the sound of the damage to the item using multiple microphones. The damage position identifying unit 323 may also identify the position of the damage to the item from image data captured by the second sensor 12.

[0071] The appliance operation determination unit 324 determines a predetermined operation to be performed by the self-propelled vacuum cleaner 21 (self-propelled device) in the house 10 (space) according to the situation estimated by the situation estimation unit 322. The appliance operation determination unit 324 may also determine a predetermined operation to be performed by appliances other than the self-propelled vacuum cleaner 21 according to the situation estimated by the situation estimation unit 322. The appliance operation determination unit 324 may also determine a predetermined operation to be performed by the sensors constituting the sensor group 1 according to the situation estimated by the situation estimation unit 322, in addition to the appliances constituting the appliance group 2.

[0072] The appliance operation information storage unit 331 prestores appliance operation information that associates the situation in which the damage to the item occurred with the operation to be executed by the appliance.

[0073] FIG. 3 is a diagram showing an example of appliance operation information stored in the appliance operation information storage unit in the first embodiment.

[0074] As shown in Fig. 3, the situation in which an article is damaged is associated with an operation to be executed by the device. The first situation in which an article slips out of a person's hand during daily activities is associated with an operation to make the self-propelled vacuum cleaner 21 suck up the damaged article and an operation to make the display device 22 present a substitute article for the damaged article. The second situation in which multiple people are fighting is associated with an operation to make the self-propelled vacuum cleaner 21 move while outputting a voice to soothe the multiple people fighting, and an operation to make the display device 22 present a restaurant or movie suitable for reconciliation. The third situation in which a suspicious person has intruded is associated with an operation to make the self-propelled vacuum cleaner 21 move while disturbing the feet of the suspicious person, an operation to make the imaging device (second sensor 12) capture the suspicious person, and an operation to make the information device 23 transmit image data of the suspicious person and notification information for notifying the police of the presence of the suspicious person.

[0075] It should be noted that the operation of the device associated with the situation in which damage to an item occurs is not limited to the above.

[0076] The appliance operation determination unit 324 refers to the appliance operation information storage unit 331 and determines a predetermined operation to be executed by the appliance associated with the situation estimated by the situation estimation unit 322 .

[0077] When a first situation in which an object has slipped out of a person's hand during a daily activity is estimated, the device operation determination unit 324 determines the operation of the self-propelled vacuum cleaner 21 to suck up the broken object and the operation of the display device 22 to present a substitute object for the broken object, when a second situation in which multiple people are fighting is estimated, the device operation determination unit 324 determines the operation of the self-propelled vacuum cleaner 21 to move while outputting a voice to soothe the multiple people fighting, and the operation of the display device 22 to present a restaurant or movie suitable for reconciliation, when a third situation in which a suspicious person has intruded is estimated, the device operation determination unit 324 determines the operation of the self-propelled vacuum cleaner 21 to move while disturbing the suspicious person's feet, the operation of the imaging device (second sensor 12) to image the suspicious person, and the operation of the information device 23 to transmit image data of the suspicious person and notification information to notify the police of the presence of the suspicious person to the police.

[0078] Further, the appliance operation determination unit 324 controls the operation of the self-propelled vacuum cleaner 21. The appliance operation determination unit 324 generates control information (second information) for causing the self-propelled vacuum cleaner 21 (self-propelled device) to execute a predetermined operation in the space according to the estimated situation. Here, the control information is cleaning instruction information for causing the self-propelled vacuum cleaner 21 to clean damaged items in the house 10 (space) according to the estimated situation. When the appliance operation determination unit 324 determines the operation of the self-propelled vacuum cleaner 21 to suck up damaged items, it moves the self-propelled vacuum cleaner 21 to the damage position identified by the damage position identification unit 323 and generates cleaning instruction information for causing the self-propelled vacuum cleaner 21 to clean the damaged items at the damage position.

[0079] The control information transmission unit 312 outputs control information (second information) for causing the self-propelled vacuum cleaner 21 (self-propelled device) to execute a predetermined operation in the house 10 (space) according to the estimated situation. The control information transmission unit 312 transmits the cleaning instruction information generated by the appliance operation determination unit 324 to the self-propelled vacuum cleaner 21. When the self-propelled vacuum cleaner 21 receives the cleaning instruction information, it moves to the damage position, captures an image of the suction target, which is a damaged item, at the damage position, transmits the captured image data to the server device 3, and sucks up the suction target. The sensor data reception unit 311 acquires information on the suction target by the self-propelled vacuum cleaner 21. For example, the information on the suction target is information on the appearance of the suction target. The information on the appearance includes an image captured by a camera provided in the self-propelled vacuum cleaner 21. The image includes the suction target. The sensor data reception unit 311 receives the image data of the suction target transmitted by the self-propelled vacuum cleaner 21.

[0080] When the device operation determination unit 324 specifies the operation of the display device 22 to present a substitute for the damaged item, the damaged item identification unit 325 specifies the damaged item constituted by the suction object based on the image data including the suction object received from the self-propelled vacuum cleaner 21. The suction object is a damaged item. The damaged item identification unit 325 specifies the damaged item based on the appearance of the suction object. The damaged item identification unit 325 recognizes an image including the suction object and identifies the damaged item from the recognized image. The memory 33 may previously store a table in which images of a plurality of items are associated with the names (product names) of each of the plurality of items. The damaged item identification unit 325 compares the image data of the suction object with the images of the plurality of items stored in the memory 33, and specifies the name of the item associated with the image of the item that partially matches the image of the suction object as the name of the damaged item.

[0081] The product information storage unit 332 stores product information relating to products.

[0082] FIG. 4 is a diagram showing an example of the product information stored in the product information storage unit in the first embodiment.

[0083] 4, the product information includes a product number for identifying the product, the product name, the type of the product, the category of the product, the color of the product, the size of the product, the weight of the product, the material of the product, the price of the product, the manufacturer of the product, and the store selling the product. The product information storage unit 332 stores product information that associates the product number, the product name, the type of the product, the category of the product, the color of the product, the size of the product, the weight of the product, the material of the product, the price of the product, the manufacturer of the product, and the store selling the product.

[0084] 4 is an example, and may include other information such as images of the items. All of the item information may be managed in one table, or may be managed in a distributed manner across multiple tables.

[0085] The replacement item identification unit 326 identifies a replacement item related to the damaged item based on the item information on the damaged item. The replacement item identification unit 326 obtains the item information of the damaged item from the item information storage unit 332.

[0086] For example, the replacement item may be the same item as the damaged item. In this case, the replacement item identification unit 326 identifies the same item as the damaged item as the replacement item.

[0087] Also, for example, the replacement item may be an item with the same attributes as the damaged item. In this case, the replacement item identification unit 326 identifies an item with the same attributes as the damaged item as the replacement item. The attributes are, for example, the color, size, weight, or material of the item. The replacement item identification unit 326 identifies an item with the same color, size, weight, and material as the damaged item as the replacement item. Note that the replacement item identification unit 326 may identify an item with at least one of the color, size, weight, and material as the damaged item as the replacement item.

[0088] Also, for example, the replacement item may be an item with attributes similar to those of the damaged item. In this case, the replacement item identification unit 326 identifies an item with attributes similar to those of the damaged item as a replacement item. The attributes are, for example, the color, size, or weight of the item. The replacement item identification unit 326 identifies an item similar to at least one of the color, size, and weight of the damaged item as a replacement item. For example, a color similar to blue is blue-purple, and similar colors are associated with each color and stored in advance. Also, an item with a size similar to that of the damaged item is, for example, an item with a size within a range of -1 centimeter to +1 centimeter of the width, depth, and height of the damaged item. Note that an item with a size similar to that of the damaged item is not limited to an item with a size within a predetermined value range as described above, but may be an item with a size within a predetermined ratio range, such as an item with a size within a range of -10% to +10% of the width, depth, and height of the damaged item. Furthermore, an item with a weight similar to that of the damaged item is, for example, an item with a weight within a range of -10 grams to +10 grams of the weight of the damaged item. Note that an item with a weight similar to that of the damaged item is not limited to an item with a size within a predetermined range as described above, but may be an item with a size within a predetermined ratio range, such as an item with a weight within a range of -10% to +10% of the weight of the damaged item.

[0089] Also, for example, the replacement item may be an item that has the same attributes as the damaged item and is made of a material that is stronger than the damaged item. In this case, the replacement item identification unit 326 identifies an item that has the same attributes as the damaged item and is made of a material that is stronger than the damaged item as the replacement item. The attribute is, for example, the color of the item. If the material of the damaged item is pottery, the replacement item identification unit 326 identifies an item that has the same color as the damaged item and is made of metal that is stronger than the damaged item as the replacement item.

[0090] The memory 33 may include a user information storage unit that stores user information related to the user. The user information includes a user ID for identifying the user, the user's name, the user's address, the user's date of birth, the user's blood type, the user's family structure, and the user's owned items. The replacement item identification unit 326 may identify a user who owns the damaged item identified by the damaged item identification unit 325, and obtain user information of the identified user from the user information storage unit. Then, the replacement item identification unit 326 may identify a replacement item related to the damaged item based on the item information related to the damaged item and the user information related to the owner of the damaged item.

[0091] For example, the user information may include owned item information indicating a plurality of owned items owned by the owner. The substitute item identification unit 326 may identify an item of a different type from the damaged item from among the plurality of owned items indicated by the owned item information, and may identify an item of the same type as the damaged item and having at least one attribute identical to the identified item from among a plurality of items being sold, as a substitute item. The user information may also include residence information indicating the location of the owner's residence. The substitute item identification unit 326 may identify a substitute item that can be purchased at a store within a predetermined range from the location of the residence indicated by the residence information.

[0092] The presentation information generating unit 327 generates presentation information regarding the substitute item identified by the substitute item identifying unit 326. The presentation information includes an image showing the appearance of the substitute item. The presentation information may also include an object image for ordering the substitute item together with the image showing the appearance of the substitute item.

[0093] The presentation information transmitting unit 313 outputs presentation information related to the substitute item identified by the substitute item identifying unit 326. That is, the presentation information transmitting unit 313 transmits the presentation information generated by the presentation information generating unit 327 to the display device 22. The display device 22 displays the received presentation information in a predetermined format.

[0094] Furthermore, when a second situation in which multiple people are fighting is estimated, the control information generated by the appliance operation determination unit 324 is audio output instruction information for causing the self-propelled vacuum cleaner 21 (self-propelled device) to output a predetermined sound in the house 10 (space) according to the estimated situation. When the appliance operation determination unit 324 determines the operation of the self-propelled vacuum cleaner 21 to move to the damage position identified by the damage position identification unit 323 while outputting a sound that soothes the multiple people who are fighting, the appliance operation determination unit 324 generates audio output instruction information for moving the self-propelled vacuum cleaner 21 to the damage position identified by the damage position identification unit 323 while outputting a sound that soothes the multiple people who are fighting.

[0095] The control information transmitting unit 312 transmits the sound output instruction information generated by the appliance operation determining unit 324 to the self-propelled vacuum cleaner 21. When the self-propelled vacuum cleaner 21 receives the sound output instruction information, it moves from the charging position to the damage position and outputs a sound to calm down the multiple people fighting at the damage position. At this time, the self-propelled vacuum cleaner 21 may output a sound to calm down the multiple people fighting and may also suck up the suction object (broken item). Then, after the suction of the suction object is completed or after a predetermined time has elapsed since the start of the sound output, the self-propelled vacuum cleaner 21 returns to the charging position.

[0096] When the equipment operation determination unit 324 has specified an operation of the display device 22 for presenting restaurants or movies suitable for reconciliation, the service specification unit 328 refers to the service information storage unit 333 to specify a restaurant or movie to present to the multiple people who are fighting.

[0097] The service information storage unit 333 prestores service information including movie information about movies currently being shown and restaurant information about restaurants.

[0098] FIG. 5 is a diagram showing an example of movie information stored in the service information storage unit in the first embodiment.

[0099] 5, the movie information includes the movie title, the movie genre, the screening location, the screening schedule, and the vacant seat information. The service information storage unit 333 stores the movie information in which the movie title, the movie genre, the screening location, the screening schedule, and the vacant seat information are associated with each other.

[0100] 5 is an example, and may include other information such as actors appearing in the movie. Also, all of the movie information may be managed in one table, or may be managed in a distributed manner across multiple tables.

[0101] The service identification unit 328 identifies a movie to be presented to the multiple people who are fighting. The service identification unit 328 identifies, for example, a movie that is in a genre suitable for reconciliation and is viewable. It is preferable that whether or not a movie is in a genre suitable for reconciliation is included in advance in the movie information. Also, a viewable movie is a movie whose screening start time is after the current time and has vacant seats. Also, when the memory 33 stores in advance user information related to each of the multiple people who are fighting and the user information includes information related to a favorite movie genre, the service identification unit 328 may refer to the user information and identify a movie that corresponds to a favorite movie genre common to the multiple people who are fighting.

[0102] FIG. 6 is a diagram showing an example of restaurant information stored in the service information storage unit in the first embodiment.

[0103] 6, the restaurant information includes the restaurant name, the food genre, the restaurant location, the restaurant business hours, and the vacant seat information. The service information storage unit 333 stores the restaurant information in which the restaurant name, the food genre, the restaurant location, the restaurant business hours, and the vacant seat information are associated with each other.

[0104] 6 is an example, and may include other information such as menus and images of the restaurant interior, etc. All restaurant information may be managed in one table, or may be managed in a distributed manner across multiple tables.

[0105] The service identification unit 328 identifies a restaurant to be presented to the multiple people who are fighting. The service identification unit 328 identifies, for example, a restaurant that is a genre suitable for reconciliation and where meals can be eaten. It is preferable that whether or not a restaurant is a genre suitable for reconciliation is included in advance in the restaurant information. In addition, a restaurant where meals can be eaten is a restaurant that is open at the current time and has vacant seats. In addition, when the memory 33 stores in advance user information related to each of the multiple people who are fighting, and the user information includes information related to a favorite food genre, the service identification unit 328 may refer to the user information and identify a restaurant that corresponds to a favorite food genre common to the multiple people who are fighting.

[0106] In addition, in this embodiment 1, the service identification unit 328 may identify either a restaurant or a movie to be presented to the multiple people who are fighting, or may identify both a restaurant and a movie to be presented to the multiple people who are fighting.

[0107] In addition, in the first embodiment, the appliance operation determination unit 324 determines the operation of the display device 22 to present a restaurant or movie suitable for reconciliation when the second situation in which a plurality of people are fighting is estimated, but the present disclosure is not particularly limited thereto, and the appliance operation determination unit 324 may determine the operation of the display device 22 to present a service suitable for reconciliation when the second situation in which a plurality of people are fighting is estimated. When the appliance operation determination unit 324 specifies the operation of the display device 22 to present a service suitable for reconciliation, the service specification unit 328 refers to the service information storage unit 333 and specifies a service to be presented to the plurality of people who are fighting.

[0108] The presentation information generating unit 327 generates presentation information (third information) for causing the display device 22 (presentation device) to present information for changing the estimated situation. The presentation information generating unit 327 generates presentation information related to a restaurant or a movie suitable for reconciliation identified by the service identifying unit 328.

[0109] The presentation information transmitting unit 313 outputs presentation information for causing the display device 22 (presentation device) to present information for changing the estimated situation. Here, the presentation information transmitting unit 313 transmits presentation information related to the restaurant or movie identified by the service identifying unit 328 to the display device 22. That is, the presentation information transmitting unit 313 transmits the presentation information generated by the presentation information generating unit 327 to the display device 22. The display device 22 displays the received presentation information in a predetermined format.

[0110] Furthermore, when a third situation in which a suspicious person has intruded is estimated, the control information generated by the appliance operation determination unit 324 is disturbance operation instruction information for causing the self-propelled vacuum cleaner 21 (self-propelled device) to execute an operation of disturbing the suspicious person in the house 10 (space). When the appliance operation determination unit 324 determines an operation of causing the self-propelled vacuum cleaner 21 to move to the damage position identified by the damage position identification unit 323 while disturbing the feet of the suspicious person, it generates disturbance operation instruction information for causing the self-propelled vacuum cleaner 21 to move to the damage position while disturbing the feet of the suspicious person at the damage position.

[0111] The control information transmitting unit 312 transmits the disturbance operation instruction information generated by the appliance operation determining unit 324 to the self-propelled vacuum cleaner 21. When the self-propelled vacuum cleaner 21 receives the disturbance operation instruction information, it moves from the charging position to the damage position and moves while disturbing the feet of the suspicious person at the damage position. At this time, the self-propelled vacuum cleaner 21 measures the distance from the suspicious person with a distance sensor and moves while maintaining a predetermined distance from the suspicious person. Then, after the suspicious person has left the house 10 (space), the self-propelled vacuum cleaner 21 sucks up the suction target (broken item) and returns to the charging position. The self-propelled vacuum cleaner 21 may also move so as to block the entrance and exit of the house 10 in order to keep the suspicious person inside the house 10 until the police arrive.

[0112] Furthermore, when an action of causing the imaging device (second sensor 12) to image a suspicious person is identified, the sensor data receiving unit 311 acquires image data of the suspicious person from the second sensor 12 arranged in the house 10 (space). The sensor data receiving unit 311 outputs the image data to the notification information generating unit 329.

[0113] The notification information generating unit 329 generates notification information for notifying that a suspicious person is present. The notification information includes, for example, information indicating the presence of a suspicious person and the address of the house 10. The notification information generating unit 329 outputs the image data acquired by the sensor data receiving unit 311 and the notification information to the notification information transmitting unit 314.

[0114] When it is determined that information device 23 should operate to transmit image data of an image of a suspicious person and notification information for notifying the police of the presence of a suspicious person, notification information transmitting unit 314 transmits the image data acquired by sensor data receiving unit 311 and the notification information for notifying the police of the presence of a suspicious person to information device 23. Information device 23 receives the image data and notification information from server device 3, and transmits the received image data and notification information to a server device managed by the police.

[0115] When the self-propelled vacuum cleaner 21 detects that it has been damaged, it may transmit damage information indicating that the self-propelled vacuum cleaner 21 has been damaged to the server device 3. The sensor data receiving unit 311 may receive the damage information transmitted by the self-propelled vacuum cleaner 21. When the damage information is received, the notification information generating unit 329 may generate notification information (fourth information) for requesting the manufacturer to repair the self-propelled vacuum cleaner 21. The notification information transmitting unit 314 may transmit the notification information for requesting the manufacturer to repair the self-propelled vacuum cleaner 21 to the information device 23. The information device 23 may receive the notification information for requesting the manufacturer to repair the self-propelled vacuum cleaner 21 from the server device 3, and transmit the received notification information to a server device managed by the manufacturer.

[0116] Fig. 7 is a first flowchart for explaining the operation of the server device in the first embodiment of the present disclosure, and Fig. 8 is a second flowchart for explaining the operation of the server device in the first embodiment of the present disclosure. Note that, although Fig. 7 explains an example of detecting damage based on sound data, damage may be detected based on other sensor data such as image data as described above.

[0117] First, the sensor data receiving unit 311 receives sound data as sensor data from the first sensor 11 (step S1).

[0118] Next, the damage detection unit 321 uses the sound data received by the sensor data receiving unit 311 to determine whether or not damage to an item in the house 10 has been detected (step S2). At this time, the damage detection unit 321 detects that an item has been damaged if the frequency components of the sound data received from the sensor data receiving unit 311 match the frequency components of pre-stored sound data indicating that an item has been damaged. Here, if it is determined that damage to an item has not been detected (NO in step S2), the process returns to step S1.

[0119] On the other hand, when it is determined that damage to the article has been detected (YES in step S2), the sensor data receiving unit 311 receives, as sensor data, from the second sensor 12, a plurality of image data sets acquired within a predetermined period of time from the time when it was detected that damage to the article occurred (step S3). The sensor data receiving unit 311 may request the second sensor 12 to transmit a plurality of image data sets acquired within a predetermined period of time from the time when it was detected that damage to the article occurred, and may receive a plurality of image data sets transmitted by the second sensor 12 in response to the request. The sensor data receiving unit 311 may also periodically receive image data from the second sensor 12 and store the received image data sets in the memory 33. Then, when it is detected that damage to the article has occurred, the situation estimating unit 322 may read out from the memory 33 a plurality of image data sets acquired within a predetermined period of time from the time when it was detected that damage to the article occurred.

[0120] Next, the situation estimation unit 322 estimates the situation in which the damage to the item occurred based on a plurality of image data acquired within a predetermined period of time from the time when the damage to the item was detected (step S4). In the first embodiment, the situation estimation unit 322 estimates whether the situation in which the damage to the item occurred is one of the first situation in which the item slipped out of a person's hand during a daily activity, the second situation in which a plurality of people are fighting, and the third situation in which a suspicious person has entered.

[0121] Next, the situation estimation unit 322 determines whether the estimation result is a first situation in which an object has slipped out of the person's hand during a daily movement (step S5).

[0122] Here, if it is determined that the estimation result is the first situation in which the item has slipped out of the person's hand (YES in step S5), the damage position identifying unit 323 identifies the damage position of the item within the house 10 (step S6).

[0123] Next, the appliance operation determination unit 324 refers to the appliance operation information storage unit 331 and determines the operation of the appliance associated with the first situation estimated by the situation estimation unit 322 (step S7). Here, if the estimation result is the first situation in which an item has slipped out of a person's hand, the appliance operation determination unit 324 determines the operation of the self-propelled vacuum cleaner 21 to suck up the broken item and the operation of the display device 22 to present a replacement item for the broken item.

[0124] Next, the equipment operation determination unit 324 moves the self-propelled vacuum cleaner 21 to the damage position identified by the damage position identification unit 323, and generates cleaning instruction information for causing the self-propelled vacuum cleaner 21 to clean the damaged items at the damage position (step S8).

[0125] Next, the control information transmitting unit 312 transmits the cleaning instruction information generated by the appliance operation determining unit 324 to the self-propelled vacuum cleaner 21 (step S9). The self-propelled vacuum cleaner 21 receives the cleaning instruction information from the server device 3, and moves toward the damage position included in the cleaning instruction information. When the self-propelled vacuum cleaner 21 arrives at the damage position, it captures an image of the suction target with the camera and transmits the captured image data to the server device 3. After transmitting the image data including the suction target to the server device 3, the self-propelled vacuum cleaner 21 sucks up the suction target.

[0126] Next, the sensor data receiving unit 311 receives image data including the suction target from the self-propelled cleaner 21 as sensor data (step S10).

[0127] Next, the damaged item identifying unit 325 identifies a damaged item constituted by the suction object based on the image data including the suction object received from the self-propelled vacuum cleaner 21 (step S11). The damaged item identifying unit 325 compares images of a plurality of items stored in advance with the image of the suction object contained in the image data, and recognizes the damaged item constituted by the suction object. For example, if the suction object is a fragment of a ceramic mug, the damaged item identifying unit 325 recognizes an image of the item that partially matches the image of the fragment contained in the image data, and identifies the item corresponding to the recognized image of the item as the damaged item.

[0128] Next, the replacement item identification unit 326 acquires item information of the damaged item from the item information storage unit 332 (step S12).

[0129] Next, the replacement item specifying unit 326 specifies a replacement item related to the damaged item based on the item information on the damaged item (step S13). For example, the replacement item specifying unit 326 specifies an item identical to the damaged item as a replacement item.

[0130] Next, the presentation information generating unit 327 generates presentation information related to the substitute item identified by the substitute item identifying unit 326 (step S14).

[0131] Next, the presentation information transmitting unit 313 transmits the presentation information generated by the presentation information generating unit 327 to the display device 22 (step S15). The display device 22 receives the presentation information transmitted by the server device 3 and displays the received presentation information. The display device 22 displays the presentation information while the self-propelled vacuum cleaner 21 is sucking up the suction target. The display device 22 may display the presentation information when the self-propelled vacuum cleaner 21 starts to suck up the suction target. The display device 22 may continue to display the presentation information even after the self-propelled vacuum cleaner 21 finishes suctioning the suction target.

[0132] FIG. 9 is a diagram for explaining the operation of a device in a first situation in which an object has slipped out of a person's hand during daily activity in the first embodiment.

[0133] If the item 6 slips out of the person's hand during daily activities and is damaged, the self-propelled vacuum cleaner 21 moves to the damaged location and sucks up the damaged item 6. Also, the display device 22 installed in the room displays presented information 221 including an image 222 for confirming whether or not to purchase a replacement item identical to the damaged item 6.

[0134] As shown in FIG. 9, the presentation information 221 includes, for example, an image 222 including the sentence "Would you buy a new mug?", an image showing the appearance of the alternative item, and a button for switching to an order screen for ordering the alternative item.

[0135] When generating the cleaning instruction information, the appliance operation determination unit 324 may generate presentation information for notifying the user that cleaning will start. In this case, the control information transmission unit 312 may transmit the cleaning instruction information generated by the appliance operation determination unit 324 to the self-propelled vacuum cleaner 21 and also transmit the presentation information generated by the appliance operation determination unit 324 to the display device 22. The display device 22 may display the presentation information for notifying the user that cleaning will start. In this case, the presentation information includes, for example, a sentence such as "Are you okay? Not injured? I'm going to clean now."

[0136] On the other hand, if it is determined in step S5 of FIG. 7 that the inference result is not the first situation in which an object has slipped out of a person's hand (NO in step S5), the situation estimation unit 322 determines whether the inference result is the second situation in which multiple people are fighting (step S16).

[0137] Here, when it is determined that the estimation result is the second situation in which a plurality of people are fighting (YES in step S16), the damage position identifying unit 323 identifies the position of damage to the item in the house 10 (step S17).

[0138] Next, the appliance operation determination unit 324 refers to the appliance operation information storage unit 331 and determines the operation of the appliance associated with the second situation estimated by the situation estimation unit 322 (step S18). Here, if the estimation result is the second situation in which multiple people are fighting, the appliance operation determination unit 324 determines the operation of the self-propelled vacuum cleaner 21 to move the multiple people fighting while outputting sounds to calm them down, and the operation of the display device 22 to present restaurants or movies suitable for reconciliation.

[0139] Next, the appliance operation determination unit 324 moves the self-propelled vacuum cleaner 21 to the damage position identified by the damage position identification unit 323, and generates audio output instruction information for moving the self-propelled vacuum cleaner 21 to the damage position while outputting audio for calming down the multiple people who are fighting (step S19).

[0140] Next, the control information transmitting unit 312 transmits the audio output instruction information generated by the appliance operation determining unit 324 to the self-propelled vacuum cleaner 21 (step S20). Upon receiving the audio output instruction information, the self-propelled vacuum cleaner 21 moves from the charging position to the damage position and outputs audio to calm down the multiple people fighting at the damage position. Then, the self-propelled vacuum cleaner 21 sucks up the object to be sucked (broken item). After the suction of the object to be sucked is completed or after a predetermined time has elapsed since the start of audio output, the self-propelled vacuum cleaner 21 returns to the charging position.

[0141] Next, the service identification unit 328 refers to the service information storage unit 333 and identifies a restaurant to present to the multiple people who are fighting (step S21). The service identification unit 328 refers to pre-stored user information, identifies a genre of cuisine common to the multiple people who are fighting, and identifies a restaurant that corresponds to the identified genre and where meals can be eaten.

[0142] In the processing of FIG. 8, the service identification unit 328 identifies a restaurant to present to multiple people who are fighting, but the present disclosure is not limited to this, and the service identification unit 328 may identify a movie to present to multiple people who are fighting.

[0143] Next, the presentation information generating unit 327 generates presentation information related to the restaurant suitable for reconciliation identified by the service identifying unit 328 (step S22).

[0144] Next, the presentation information transmitting unit 313 transmits the presentation information generated by the presentation information generating unit 327 to the display device 22 (step S23). The display device 22 receives the presentation information transmitted by the server device 3 and displays the received presentation information. The display device 22 displays the presentation information while the self-propelled vacuum cleaner 21 is outputting sound. The display device 22 may display the presentation information when the self-propelled vacuum cleaner 21 starts outputting sound. The display device 22 may continue to display the presentation information even after the self-propelled vacuum cleaner 21 finishes outputting sound.

[0145] FIG. 10 is a diagram for explaining the operation of devices in a second situation in which a plurality of people are fighting in the first embodiment.

[0146] If an item 6 is damaged while multiple people 62, 63 are fighting, the self-propelled vacuum cleaner 21 moves to the location of the damage and outputs a voice to calm the multiple people 62, 63 who are fighting. In Fig. 10, for example, the self-propelled vacuum cleaner 21 outputs a voice saying "Calm down." In addition, the display device 22 installed in the room displays presentation information 223 for presenting restaurants suitable for the multiple people 62, 63 who are fighting to make up.

[0147] 10, the presented information 223 includes, for example, a sentence such as "Would you like to eat Italian food at M Restaurant?" and a reservation button for making a reservation at the restaurant. When the reservation button is pressed, the screen transitions to a reservation screen for making a reservation at the restaurant.

[0148] On the other hand, if it is determined in step S16 of FIG. 8 that the inference result is not the second situation in which multiple people are fighting (NO in step S16), the situation estimation unit 322 determines whether the inference result is the third situation in which a suspicious person has intruded (step S24).

[0149] Here, if it is determined that the estimation result is not the third situation where a suspicious person has intruded (NO in step S24), that is, if the situation where an item has been damaged cannot be estimated, the process ends. Note that, if it is determined that the estimation result is not the third situation where a suspicious person has intruded, the appliance operation determination unit 324 may determine the operation of the self-propelled vacuum cleaner 21 to suck up the damaged item.

[0150] On the other hand, if the estimation result is determined to be the third situation in which a suspicious person has intruded (YES in step S24), the damage position identifying unit 323 identifies the damage position of the item within the house 10 (step S25).

[0151] Next, the appliance operation determination unit 324 refers to the appliance operation information storage unit 331 and determines the operation of the appliance associated with the third situation estimated by the situation estimation unit 322 (step S26). Here, if the estimation result is the third situation in which a suspicious person has intruded, the appliance operation determination unit 324 determines the operation of the self-propelled vacuum cleaner 21 to move while disturbing the feet of the suspicious person, the operation of the imaging device (second sensor 12) to image the suspicious person, and the operation of the information device 23 to transmit image data of the image of the suspicious person and notification information to notify the police of the presence of the suspicious person.

[0152] Next, the equipment operation determination unit 324 moves the self-propelled vacuum cleaner 21 to the damage position identified by the damage position identification unit 323, and generates disturbance operation instruction information for causing the self-propelled vacuum cleaner 21 to move to the damage position while disturbing the feet of the suspicious person (step S27).

[0153] Next, the control information transmitting unit 312 transmits the disturbance operation instruction information generated by the appliance operation determining unit 324 to the self-propelled vacuum cleaner 21 (step S28). Upon receiving the disturbance operation instruction information, the self-propelled vacuum cleaner 21 moves from the charging position to the damage position, and moves while disturbing the feet of the suspicious person at the damage position. Then, after the suspicious person has left the house 10 (space), the self-propelled vacuum cleaner 21 sucks up the suction target (broken item) and returns to the charging position.

[0154] Next, the sensor data receiving unit 311 receives image data of the suspicious person captured by the second sensor 12 arranged in the house 10 (space) (step S29).

[0155] Next, the notification information generating unit 329 generates notification information for notifying that a suspicious individual is present (step S30).

[0156] Next, the notification information transmitting unit 314 transmits the image data acquired by the sensor data receiving unit 311 and the notification information generated by the notification information generating unit 329 to the information device 23 (step S31). The information device 23 receives the image data and the notification information from the server device 3, and transmits the received image data and notification information to a server device managed by the police.

[0157] FIG. 11 is a diagram for explaining the operation of the device in a third situation in which a suspicious person has intruded in the first embodiment.

[0158] If the article 6 is damaged when the suspicious person 64 intrudes, the self-propelled vacuum cleaner 21 moves to the damaged location, and moves while disturbing the feet of the suspicious person 64. In FIG. 11, for example, the self-propelled vacuum cleaner 21 moves around the suspicious person 64 while maintaining a predetermined distance from the suspicious person 64. In addition, the second sensor 12 transmits image data of the suspicious person 64 to the server device 3. Furthermore, the information device 23 receives image data of the suspicious person 64 and notification information for notifying the presence of the suspicious person 64 from the server device 3, and transmits the received image data and notification information to a server device managed by the police.

[0159] After the information device 23 transmits the image data and the notification information to a server device managed by the police, the display device 22 installed in the room may display presentation information 224 to indicate that the police have been notified. As shown in Fig. 11, the presentation information 224 includes, for example, a sentence such as "The police have been notified."

[0160] In this way, the situation in which the damage to the item occurred is estimated based on the first information acquired by at least one of the one or more sensors installed in the space, and second information for causing the self-propelled vacuum cleaner 21 to perform a predetermined operation in the space is output according to the estimated situation. Therefore, when an item present in the space is damaged, the self-propelled vacuum cleaner 21 can be caused to perform a predetermined operation according to the situation in which the damage to the item occurred.

[0161] The presentation information generating unit 327 may generate presentation information (fifth information) for causing the display device 22 (presentation device) to present information related to an article for suppressing the occurrence of the estimated situation. The presentation information transmitting unit 313 may transmit (output) presentation information (fifth information) for causing the display device 22 (presentation device) to present information related to an article for suppressing the occurrence of the estimated situation. For example, when a third situation in which a suspicious person has intruded is estimated, the presentation information generating unit 327 may generate presentation information related to a security product and transmit it to the display device 22.

[0162] (Embodiment 2) The device control system in the first embodiment includes one server device, whereas the device control system in the second embodiment includes two server devices.

[0163] FIG. 12 is a diagram showing a configuration of a first server device according to the second embodiment of the present disclosure, and FIG. 13 is a diagram showing a configuration of a second server device according to the second embodiment of the present disclosure.

[0164] The device control system in the second embodiment includes a first server device 3A, a second server device 3B, a gateway 5 (not shown), a first sensor 11, a second sensor 12, a self-propelled vacuum cleaner 21, a display device 22, and an information device 23. In the second embodiment, the same components as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted. The sensor group 1 includes various sensors such as the first sensor 11 and the second sensor 12. The device group 2 includes various devices such as the self-propelled vacuum cleaner 21, the display device 22, and the information device 23. The gateway 5 is omitted in Figs. 12 and 13.

[0165] The first server device 3A is communicatively connected to the sensor group 1, the device group 2, and the second server device 3B via a network. The second server device 3B is communicatively connected to the device group 2 and the first server device 3A via the network.

[0166] The first server device 3A is operated, for example, by a platform provider, and the second server device 3B is operated, for example, by a third party.

[0167] The first server device 3A includes a communication unit 31A, a processor 32A, and a memory 33A.

[0168] The communication unit 31A includes a sensor data receiving unit 311, a control information transmitting unit 312, a notification information transmitting unit 314, a damaged item information transmitting unit 315, and an appliance operation information transmitting unit 316. The processor 32A includes a damage detecting unit 321, a situation estimating unit 322, a damage position identifying unit 323, an appliance operation determining unit 324, a damaged item identifying unit 325, and a notification information generating unit 329. The memory 33A includes an appliance operation information storage unit 331.

[0169] The damaged item information transmitting section 315 transmits damaged item information indicating the damaged item identified by the damaged item identifying section 325 to the second server device 3B.

[0170] The appliance operation information transmitting section 316 transmits appliance operation information indicating the operation of the display device 22 that causes the display device 22 to present the restaurant or movie suitable for reconciliation determined by the appliance operation determining section 324 to the second server device 3B.

[0171] The second server device 3B includes a communication unit 31B, a processor 32B, and a memory 33B.

[0172] The communication unit 31B includes a presentation information transmission unit 313, a damaged item information reception unit 317, and an appliance operation information reception unit 318. The processor 32B includes a replacement item identification unit 326 and a presentation information generation unit 327. The memory 33B includes an item information storage unit 332.

[0173] The damaged item information receiving unit 317 receives the damaged item information transmitted by the first server device 3 A. The replacement item identifying unit 326 identifies a replacement item related to the damaged item based on the damaged item information received by the damaged item information receiving unit 317.

[0174] The device operation information receiving unit 318 receives the device operation information transmitted by the first server device 3 A. When the device operation information receiving unit 318 receives device operation information indicating an operation of the display device 22 to present a restaurant or movie suitable for reconciliation, the service identifying unit 328 refers to the service information storage unit 333 and identifies a restaurant or movie to present to the multiple people who are fighting.

[0175] In addition, in this embodiment 2, the first server device 3A transmits damaged item information and equipment operation information to the second server device 3B, but the present disclosure is not limited to this, and the second server device 3B may transmit a request requesting damaged item information and equipment operation information to the first server device 3A, and the first server device 3A may transmit the damaged item information and equipment operation information to the second server device 3B in response to the request.

[0176] In the second embodiment, the device control system may include a plurality of second servers 3B.

[0177] In each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0178] A part or all of the functions of the device according to the embodiment of the present disclosure are typically realized as an LSI (Large Scale Integration), which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip that includes some or all of the functions. The integrated circuit is not limited to an LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI, may be used.

[0179] Furthermore, some or all of the functions of the device according to the embodiment of the present disclosure may be realized by a processor such as a CPU executing a program.

[0180] Furthermore, all the numbers used above are merely examples for the purpose of specifically explaining the present disclosure, and the present disclosure is not limited to the numbers exemplified.

[0181] In addition, the order in which each step is performed shown in the above flowchart is merely an example for specifically explaining the present disclosure, and an order other than the above may be used as long as the same effect is obtained. In addition, some of the steps may be performed simultaneously (in parallel) with other steps. [Industrial Applicability]

[0182] The information processing method, information processing device, and information processing program disclosed herein can cause a self-propelled device to perform a specified operation depending on the situation in which damage to an item occurs, and are therefore useful as an information processing method, information processing device, and information processing program for causing equipment to perform a specified operation. [Explanation of symbols]

[0183] 1 Sensor group 2 equipment group 3. Server equipment 3A First server device 3B Second server device 4. Network 5. Gateway 6 Goods 10. Housing 11 First Sensor 12 Second Sensor 21 Self-propelled vacuum cleaner 22 Display device 23 Information equipment 31,31A,31B Communication Department 32, 32A, 32B processors 33, 33A, 33B Memory 311 Sensor data receiver 312 Control information transmission unit 313 Presentation information transmission unit 314 Notification information transmission unit 315 Damaged Goods Information Transmission Division 316 Equipment operation information transmission unit 317 Damaged Goods Information Receiving Department 318 Equipment operation information receiving unit 321 Damage Detection Unit 322 Situation Estimation Department 323 Damage location identification part 324 Device operation determination section 325 Damaged Goods Identification Department 326 Substitute Goods Identification Department 327 Presentation information generation unit 328 Service Specific Department 329 Notification information generation unit 331 Device operation information storage unit 332 Article information storage unit 333 Service information storage unit

Claims

1. An information processing method in an information processing device for causing a self-propelled device to perform a predetermined operation and presenting information on a presentation device, comprising: The information processing device, Acquiring first information acquired by at least one of one or more sensors installed in the space; When damage to an item present in the space is detected based on the first information, second information is output for causing the self-propelled device to execute a predetermined operation within the space; The second information includes information for moving the self-propelled device to a location where the item is damaged, for causing the self-propelled device to capture an image of the damaged item, and for transmitting the captured image data; Acquire the image data of the damaged item transmitted by the self-propelled device; Identifying the damaged item based on the image data; Identifying a replacement item having the same or similar attributes as the identified damaged item; outputting sixth information for causing a presentation device to present information about the identified substitute item; Information processing methods.

2. The information processing device further estimates a situation in which the damage to the item occurred based on the first information, The second information is information for causing the self-propelled device to output a predetermined sound in the space according to the estimated situation.

2. The information processing method according to claim 1.

3. The information processing device, Furthermore, a situation in which the damage to the item occurred is estimated based on the first information, and outputting third information for causing a presentation device to present information for changing the estimated situation.

3. The information processing method according to claim 1 or 2.

4. The information processing device further estimates a situation in which the damage to the item occurred based on the first information, The self-propelled device is a self-propelled vacuum cleaner, The second information is information for causing the self-propelled vacuum cleaner to clean the damaged item in the space according to the estimated situation. The information processing method according to any one of claims 1 to 3.

5. The information processing device further estimates a situation in which the damage to the item occurred based on the first information, The estimated situation is a situation in which a suspicious person has entered the space, The second information is information for causing the self-propelled device to execute an operation to disturb the suspicious person in the space. The information processing method according to any one of claims 1 to 4.

6. The information processing device, Furthermore, image data of the suspicious person is acquired from an imaging device disposed within the space, and transmitting the acquired image data and notification information for notifying the presence of the suspicious person.

6. The information processing method according to claim 5.

7. The information processing device further outputs fourth information for requesting repair of the self-propelled device when damage information indicating that the self-propelled device is damaged is acquired.

7. The information processing method according to claim 5 or 6.

8. The information processing device, Furthermore, a situation in which the damage to the item occurred is estimated based on the first information, and outputting fifth information for causing a presentation device to present information about an article for suppressing the occurrence of the estimated situation. The information processing method according to any one of claims 1 to 7.

9. the one or more sensors include at least one of a microphone device and an imaging device installed in the space; the first information includes at least one of sound data acquired by the microphone device and image data acquired by the imaging device; The information processing device further estimates a situation in which the damage to the item occurred based on at least one of the sound data and the image data. The information processing method according to any one of claims 1 to 8.

10. The information processing device further estimates a situation in which the damage to the item occurred based on the first information, The acquisition of the first information includes acquiring the first information at a predetermined time interval; The estimation of the situation includes estimating the situation in which the damage to the item occurred based on a plurality of first information items acquired within a predetermined period based on a time point when the damage to the item occurred. The information processing method according to any one of claims 1 to 9.

11. An information processing device for causing a self-propelled device to perform a predetermined operation and presenting information on a presentation device, a first acquisition unit that acquires first information acquired by at least one of one or more sensors installed in the space; A first output unit that outputs second information for causing a self-propelled device to execute a predetermined operation in the space when damage to an item present in the space is detected based on the first information; The second information includes information for moving the self-propelled device to a location where the item is damaged, for causing the self-propelled device to capture an image of the damaged item, and for transmitting the captured image data; A second acquisition unit that acquires the image data of the damaged item transmitted by the self-propelled device; a damaged item identification unit that identifies the damaged item based on the image data; A replacement item identification unit that identifies a replacement item having attributes identical to or similar to the identified damaged item; a second output unit that outputs sixth information for causing a presentation device to present information about the identified substitute item; An information processing device comprising:

12. An information processing program for causing a self-propelled device to perform a predetermined operation and presenting information on a presentation device, Acquiring first information acquired by at least one of one or more sensors installed in the space; When damage to an item present in the space is detected based on the first information, second information is output for causing the self-propelled device to execute a predetermined operation within the space; The second information includes information for moving the self-propelled device to a location where the item is damaged, for causing the self-propelled device to capture an image of the damaged item, and for transmitting the captured image data; Acquire the image data of the damaged item transmitted by the self-propelled device; Identifying the damaged item based on the image data; Identifying a replacement item having the same or similar attributes as the identified damaged item; An information processing program that causes a computer to function to output sixth information for causing a presentation device to present information regarding the identified alternative item.

Images