Electric cleaning device and program for electric cleaning device

Abstract

To provide an electric cleaning device that accurately notifies the user of foreign objects other than the object being cleaned. [Solution] The electric cleaning device comprises an electric vacuum cleaner having a suction unit with a suction port for sucking up objects to be cleaned, a suction drive unit for sucking up and taking in objects to be cleaned from the suction port, and a drive unit for moving the device on its own, and a separate device connected to the electric vacuum cleaner via a communication network. The device includes an image recognition unit for image recognition of foreign objects other than objects to be cleaned, a display image generation unit for generating a display image to inform the user of what the foreign object is when a foreign object is image recognized, and a display control unit for displaying the generated display image on the display screen of a display unit provided in the separate device. Each time a new foreign object is image recognized by the image recognition unit, a new display image is generated by the display image generation unit, and each time a new display image is generated, the new display image is added and displayed on the display screen of the display unit.

Description

Technical Field

[0001] This invention relates to an electric cleaning device and a program for an electric cleaning device.

Background Art

[0002] In recent years, very small cameras such as CCD (Charge Coupled Device) imaging devices and CMOS (Complementary Metal Oxide Semiconductor) imaging devices have become popular. An electric cleaner has been proposed in which this small camera is arranged near the suction port so that the sucked objects can be confirmed.

[0003] For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2003-164401) describes an electric cleaner that arranges a CCD camera near the suction port and has a display unit for displaying the captured image of this CCD camera near the handle. In the electric cleaner described in this Patent Document 1, the user can perform cleaning while looking at the captured image of the CCD camera displayed on the display screen of the display unit, and can work while confirming the dust to be sucked.

[0004] In addition, the electric cleaner described in this Patent Document 1 has a function of detecting and recording (storing) an object estimated to be a precious metal by measuring the luminance of the image captured by the CCD camera. After the cleaning is completed, the user can perform a playback instruction operation to play back the recorded image and confirm the precious metal accidentally sucked in by gazing at the playback image.

[0005] Also, Patent Document 2 (Japanese Patent Application Laid-Open No. 2006-141836) describes an electric cleaner that updates and records the captured image data output from an imaging means such as a CCD camera that captures an object sucked from the suction head at regular intervals, and records the captured image data when it is detected that an object of a certain size or more has been sucked in.

[0006] In the vacuum cleaner described in Patent Document 2, if the user feels that an object other than dust has been sucked up during the vacuuming operation, they can stop the suction operation by operating the suction stop button. In the vacuum cleaner described in Patent Document 2, in response to the user's operation for image playback, the captured image data is played back and the captured image is displayed on the display monitor. The user can confirm what has been mistakenly sucked up by the vacuum cleaner by looking at the captured image displayed on the display monitor. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2003-164401 [Patent Document 2] Japanese Patent Publication No. 2006-141836 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] However, in the case of the vacuum cleaner described in Patent Document 1, while it is possible to confirm the presence of dirt by taking an image of the top of a chest of drawers with a camera, it is unnatural for the user to clean while looking at the display screen during cleaning operations such as floor cleaning. It is more common to clean while looking at the area near the suction port. Therefore, it is not possible to recognize foreign objects other than the object being cleaned, such as precious metals, in advance and avoid sucking them up, and it is inevitable that they will be sucked up.

[0009] For example, foreign objects such as "plastic bags," "cash bags," "documents," "cords," "pins, screws, and nails" should not be vacuumed up because they may cause the vacuum cleaner to stop at the suction nozzle or cause malfunction or damage to the vacuum cleaner. However, such control has not been implemented in the past. Similarly, foreign objects such as "documents," "microSD cards," and "USB memory sticks" should not be vacuumed up to ensure the cleanliness of the foreign objects and to avoid contamination or damage. However, such control has not been implemented in the past.

[0010] Furthermore, it would be convenient if it were possible to determine whether a foreign object has been sucked into the dust collection chamber or is outside of it, but such a function has not been implemented in the past.

[0011] The purpose of this invention is to provide an electric cleaning device that can improve upon the above-mentioned problems. [Means for solving the problem]

[0012] To solve the above problems, this invention provides: An electric vacuum cleaner comprising: a suction unit having a suction port for sucking up objects to be cleaned; a suction drive unit for sucking up and taking in the objects to be cleaned from the suction port; a drive unit for moving the unit itself; and a separate device connected to the electric vacuum cleaner via a communication network, An image recognition unit that recognizes foreign objects other than the object to be cleaned, When the image recognition unit recognizes the foreign object, a display image generation unit generates a display image to inform the user what the recognized foreign object is. A display control unit that displays the display image generated by the display image generation unit on the display screen of a display unit provided in the separate device, Equipped with, Each time the image recognition unit recognizes a new foreign object, the display image generation unit generates a new display image to inform the user of what the recognized foreign object is. Each time a new display image is generated, the new display image is added to the display screen of the display unit. The present invention provides an electric cleaning device characterized by the following features.

[0013] In the electric cleaning device of this invention, each time a new foreign object is recognized by the image recognition unit, an image indicating what the new foreign object is is added to the display screen. [Effects of the Invention]

[0014] According to this invention, the electric cleaning device is extremely convenient because, each time a new foreign object is recognized by the image recognition unit, an image of what that new foreign object is is added to the display screen. [Brief explanation of the drawing]

[0015] [Figure 1] This figure shows a first configuration example of an electric cleaning device related to the electric cleaning device according to this invention. [Figure 2] This block diagram shows an example of the electrical configuration of the electric cleaning device shown in Figure 1. [Figure 3] This figure is used to explain the main components of the example configuration of the electric cleaning device shown in Figure 1. [Figure 4] This figure illustrates an example of the display screen in the configuration example of the electric cleaning device shown in Figure 1. [Figure 5] This figure shows a flowchart illustrating an example of the processing operation of the electric cleaning device shown in Figure 1. [Figure 6] This figure shows a second example of the configuration of an electric cleaning device related to the electric cleaning device according to this invention. [Figure 7] Figure 6 is a block diagram showing an example of the electrical configuration of an electric cleaning device. [Figure 8]It is a diagram showing a flowchart for explaining an example of the processing operation of the electric vacuum cleaner in the example of FIG. 6. [Figure 9] It is a diagram showing a flowchart for explaining an example of the processing operation of a mobile phone terminal as an example of a mobile device constituting the electric vacuum cleaner in the example of FIG. 6. [Figure 10] It is a block diagram showing an example of the electrical configuration of an embodiment of the electric vacuum cleaner according to this invention. [Figure 11] It is a diagram showing a part of a flowchart for explaining an example of the processing operation of a self-propelled electric vacuum cleaner constituting an embodiment of the electric vacuum cleaner according to this invention. [Figure 12] It is a diagram showing a part of a flowchart for explaining an example of the processing operation of a self-propelled electric vacuum cleaner constituting an embodiment of the electric vacuum cleaner according to this invention. [Figure 13] It is a diagram showing a flowchart for explaining an example of the processing operation of a server device constituting an embodiment of the electric vacuum cleaner according to this invention. [Embodiments for Carrying Out the Invention]

[0016] [First Configuration Example of an Electric Vacuum Cleaner Related to this Invention] The case where the first configuration example of the electric vacuum cleaner related to this invention is applied to an electric vacuum cleaner that performs cleaning while the user operates it will be described with reference to the drawings.

[0017] FIG. 1 is a diagram for explaining an overall configuration example of the electric vacuum cleaner 1 of this configuration example. The electric vacuum cleaner 1 of this example includes a cleaner main body 10, a suction hose 11, a handle 12, a joint pipe 13, a suction part 14, an operation part 15, a display part 16, and a control circuit part 20.

[0018] The cleaner main body 10 includes a dust collection chamber 101 and a suction drive part 102, and has a function of sucking cleaning objects such as dust and dirt and foreign objects other than the cleaning objects from the suction part 14 and collecting them in the dust collection chamber 101 by driving the suction drive part 102. The cleaner main body 10 includes a roller 103 that can move autonomously when the user performs cleaning.

[0019] The dust collection chamber 101 of the vacuum cleaner body 10 is connected to the suction hose 11. The handle 12 is connected to the side of the suction hose 11 opposite to the side connected to the vacuum cleaner body 10. Furthermore, the side of the handle 12 opposite to the side connected to the suction hose 11 is connected to a connecting pipe 13. A suction section 14 is provided at the tip of the connecting pipe 13.

[0020] The handle 12 is equipped with an operating unit 15, a display unit 16, and a control circuit unit 20. The user holds this handle 12 to perform cleaning. The operating unit 15 includes a number of key buttons. The display unit 16 is, for example, an LCD (Liquid Crystal Display) and has a display screen 16D capable of displaying images. In this example configuration, the operating unit 15 includes a touch panel superimposed on the display screen 16D of the display unit 16, although this is not shown in the illustration.

[0021] The user can operate the key buttons on the control unit 15 using the handle 12 when needed while cleaning. Furthermore, the user can view the display screen 16D of the display unit 16 while cleaning with the handle 12. The control circuit unit 20 is electrically connected to the control unit 15 and the display unit 16, as well as to the suction drive unit 102 of the vacuum cleaner body 10.

[0022] The suction unit 14 has a suction port 141 having a predetermined area (cross-sectional area). The suction port 141 is provided so as to have an opening on the side of the housing 142 of the suction unit 14 that faces the cleaning surface, such as the floor. When the suction drive unit 102 of the vacuum cleaner body 10 is driven, objects to be cleaned smaller than the cross-sectional area and foreign objects other than the objects to be cleaned are sucked in from the suction port 141 and sucked into the dust collection chamber 101 through the connecting pipe 13, handle 12, and suction hose 11. Note that long items such as shopping bags, plastic bags, and cords may not be able to be completely sucked in even if they are sucked into the suction port 141.

[0023] The electric cleaning device 1 in this example configuration is equipped with multiple camera units. Specifically, the housing 142 of the suction unit 14 faces the cleaning surface, such as the floor, and two cameras 171 and 172 are provided near the suction port 141. When the user performs the cleaning operation, the suction unit 14 is moved in the forward and backward directions relative to the user. Camera 171 is positioned in front of the suction port 141 in the direction of movement of the suction unit 14, and camera 172 is positioned behind the suction port 141. Cameras 171 and 172 photograph the object to be cleaned and any foreign objects other than the object to be cleaned on the cleaning surface, such as the floor. Although not shown in Figure 1, lighting, such as an LED (Light Emitting Diode), is provided near cameras 171 and 172 to brightly illuminate the shooting position.

[0024] Furthermore, in this configuration example, the suction unit 14 is equipped with a camera 173 in addition to the surface facing the cleaning surface such as the floor, so that it can photograph the object to be cleaned and any foreign objects other than the object to be cleaned on the cleaning surface such as the floor, in front of the user's forward movement direction when performing a cleaning operation. In the example in Figure 1, the camera 173 is mounted on the surface of the housing 142 of the suction unit 14 that intersects with the surface facing the cleaning surface such as the floor during the cleaning operation, and is on the front side in the forward movement direction during the cleaning operation.

[0025] Furthermore, a camera 174 is attached near the connection point between the suction hose 11 and the dust collection chamber 101 so as to be able to photograph the objects to be cleaned and other foreign objects that are sucked up and collected in the dust collection chamber 101.

[0026] The four cameras 171 to 174 described above are each composed of, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor and an imaging lens, and are electrically connected to the control circuit unit 20.

[0027] [Description of an example configuration of the control circuit unit 20] In this configuration example, the control circuit unit 20 includes an image recognition unit that performs image recognition on images captured from the four cameras 171 to 174 during the cleaning operation, and separates and recognizes at least the object to be cleaned from foreign objects other than the object to be cleaned. Furthermore, when the control circuit unit 20 detects a foreign object with the image recognition unit, it has a function to notify the user of this fact and to inform them of what the foreign object is. In addition, in this configuration example, the control circuit unit 20 also has a function to forcibly stop the cleaning operation depending on the detected foreign object.

[0028] Figure 2 is a block diagram showing an example of the internal configuration of the control circuit unit 20 and the connection relationships between this control circuit unit 20 and each other that are electrically connected to it.

[0029] As shown in Figure 2, the control circuit unit 20 is configured such that the following units are connected to the control unit 201, which consists of a microcomputer, via the system bus 200: a drive control unit 202, a display control unit 203, an operation unit interface 204, camera interfaces 205, 206, 207, 208, an image capture memory 209, an image recognition unit 210, an image registration memory 211, an external interface 212, a display image generation unit 213, and an alarm drive unit 214.

[0030] The drive control unit 202 is electrically connected to the suction drive unit 102, which is located in the vacuum cleaner body 10. The control circuit unit 201 supplies a drive start control signal to the suction drive unit 102 in response to a cleaning start operation via the user's operation unit 15, and supplies a drive stop control signal in response to a cleaning stop operation.

[0031] The display control unit 203 is equipped with video RAM (Random Access Memory) and has the function of a display controller for displaying images on the display screen 16D of the display unit 16.

[0032] The operation unit 15 is connected to the operation unit interface 204. As mentioned above, in this configuration example, the operation unit 15 includes the aforementioned multiple key buttons as well as a touch panel superimposed on the display screen 16D. Therefore, the operation unit interface 204 includes not only the function of an interface for the key buttons but also the function of a touch panel interface for detecting the touch operation position on the display screen 16D.

[0033] Camera interfaces 205, 206, 207, and 208 are connected to cameras 171, 172, 173, and 174, respectively. In this configuration example, as mentioned above, each of the cameras 171, 172, 173, and 174 is attached to a lighting unit 171L, 172L, 173L, and 174L, which consists of, for example, LEDs.

[0034] The control circuit unit 20 supplies camera control signals to cameras 171, 172, 173, and 174 through camera interfaces 205, 206, 207, and 208, respectively, to control the start and stop of image acquisition for each of these cameras 171, 172, 173, and 174. Upon receiving the image acquisition start control, each of the cameras 171, 172, 173, and 174, in the image acquisition operation state, supplies the captured image to the system bus 200 through camera interfaces 205, 206, 207, and 208, respectively.

[0035] The illumination units 171L, 172L, 173L, and 174L are controlled to light up by the control unit 201. In this case, the control unit 201 of the control circuit unit 20 measures the brightness of the captured images (such as the average value of the brightness of the captured images) from the cameras 171, 172, 173, and 174 through software processing, and controls the control circuit unit 20 to light up the illumination units via the camera interfaces 205, 206, 207, and 208 when the brightness of the captured images is below a predetermined level. Alternatively, the control unit 201 may control the illumination units 171L, 172L, 173L, and 174L to always light up when the cameras 171, 172, 173, and 174 are in the image capture operation state.

[0036] The image memory 209 temporarily stores the captured image information sent from cameras 171, 172, 173, and 174 via camera interfaces 205, 206, 207, and 208, respectively, and supplies this temporarily stored image information to the image recognition unit 210 for image recognition. Furthermore, as will be described later, when the image recognition unit 210 recognizes a foreign object other than the object to be cleaned, the image memory 209 stores and retains the captured image information based on the image recognition result.

[0037] In this case, the control circuit unit 20 stores the captured image information in a way that allows the captured images from each of the cameras 171, 172, 173, and 174 to be separated and identified from one another, by, for example, by pre-separating the memory area of ​​the captured image memory 209 for each of the cameras 171, 172, 173, and 174, or by storing the captured image information in association with the identification information of each of the cameras 171, 172, 173, and 174.

[0038] In this example, the image information temporarily stored in the image memory 209 is used for image recognition processing by the image recognition unit 210, and the number of frames is calculated by adding a buffer to the number of frames corresponding to the processing time required to obtain the image recognition result. Image information in which no foreign object is recognized by the image recognition unit 210 is discarded and replaced with new image information. When a foreign object is recognized by the image recognition unit 210, the image information in which the foreign object is recognized is transferred from the image memory 209 to the display image generation unit 213 based on the image recognition result of the image recognition unit 210, and is held in the display image generation unit 213. At this time, the image information in which the foreign object is recognized is associated with the foreign object identification information (foreign object ID) included in the image recognition result from the image recognition unit 210, and is held in the display image generation unit 213.

[0039] Furthermore, if the capacity of the image memory 209 is large, all of the image information captured during a single cleaning operation may be stored in the image memory 209, and the temporarily stored image may be erased when the cleaning operation is completed. Of course, all of the images captured during multiple cleaning operations may also be stored in the image memory 209. In addition, the image stored in the image memory 209 can be appropriately recorded to a storage medium such as a USB memory or SD card (SD memory card) via the external interface 212.

[0040] In this configuration example, the image registration memory 211 stores image information of objects to be recognized as foreign objects other than the object to be cleaned, in a manner described later. The image recognition unit 210 recognizes and detects foreign objects other than the object to be cleaned from the image information captured by cameras 171, 172, 173, and 174. Of the four cameras, the image information captured by three cameras 171, 172, and 173 is compared with images stored in the image registration memory 211 and pattern matching is performed. By detecting a match between the two or a similarity of a predetermined similarity level or higher, foreign objects other than the object to be cleaned are recognized.

[0041] Furthermore, while image recognition could be performed on the captured image information from camera 174 by comparing it with registered captured images using pattern matching, the image from camera 174 is captured at the joint between the suction hose 11, the vacuum cleaner body 10, and the dust collection chamber 101, making image recognition potentially difficult. Therefore, in this configuration example, the image recognition unit 210 determines whether the size of the object to be cleaned and any foreign objects other than the object to be cleaned that are sucked into the dust collection chamber 101 through the suction hose 11 are larger than or equal to a preset size, and also determines whether the luminosity of the reflected light when light is shone on it is above a predetermined level, such as with precious metals, in order to recognize whether or not an object is a foreign object other than the object to be cleaned.

[0042] The image registration memory 211 stores image information of objects other than the target object for cleaning, which have been pre-registered by the manufacturer of the electric cleaning device 1 before it leaves the factory. In this case, the image registration memory 211 stores multiple images of a single foreign object, each captured by cameras 171, 172, and 173. Furthermore, for each image captured by cameras 171, 172, and 173, the image registration memory 211 stores image information from multiple imaging directions, such as from the front, from the back, from the left side, and from the right side. This registered image information of foreign objects is associated with the identification information (hereinafter referred to as the foreign object ID) assigned to each foreign object and stored in the image registration memory 211.

[0043] In this configuration example, the image registration memory 211 stores text information, associated with the foreign object ID, for displaying the name of the foreign object along with an image of the foreign object on the display screen 16D. Furthermore, in this configuration example, the image registration memory 211 stores "Stop suction drive" or "Continue suction drive (no control required)" as modes of suction drive control for the suction drive unit 102 of the electric cleaning device 1 when a registered foreign object is recognized, associated with the foreign object ID of the registered foreign object. In addition, various other modes of suction drive control can be set, such as "Turbo suction drive," "Strong suction drive," "Weak suction drive," and "Weakest suction drive." Furthermore, for example, when a foreign object is detected, instead of stopping the suction drive, the system can be set to "weak suction drive" or "weakest suction drive." Then, if the user looks at the image on the display screen 16D and determines that the detected foreign object is safe to suck up, the system can be controlled to "turbo suction drive" or "strong suction drive" to suck up the foreign object. On the other hand, if the user looks at the image on the display screen 16D and determines that the foreign object should not be sucked up, the system can be controlled to stop the suction drive.

[0044] Furthermore, in this configuration example, in addition to image information prepared in advance by the manufacturer, the user can register images of objects that they want the electric cleaning device 1 to recognize as foreign objects other than the objects to be cleaned, and store them in the image registration memory 211.

[0045] In other words, the electric cleaning device 1 in this configuration example is equipped with a function for the user to register images of foreign objects other than the object to be cleaned. When the user selects this function to register images of foreign objects, the display screen 16D of the display unit 16 displays the procedure for registering the foreign object images with messages and animations. The user can perform the registration process for the foreign object images while viewing the registration procedure messages and animations on the display screen 16D.

[0046] For example, the display screen 16D first displays a screen prompting the user to take images of the foreign object to be registered using cameras 171, 172, and 173. Then, instructions are given regarding the positional relationship between each camera and the subject for each camera 171, 172, and 173, and the user is prompted to take images with each camera. When the user takes images with cameras 171, 172, and 173 according to the registration procedure displayed on the display screen 16D, these images are associated with the assigned foreign object ID and stored in the image registration memory 211. Finally, the display screen 16D displays instructions prompting the user to enter the name of the foreign object as text characters, and when the user enters the name of the foreign object to be registered, the entered text character information is associated with the foreign object ID of the registered foreign object and stored in the image registration memory 211.

[0047] Finally, the display screen 16D displays instructions prompting the user to input settings for suction drive control for the suction drive unit 102 of the electric cleaning device 1 when it recognizes a registered foreign object. When the user inputs "Stop suction drive" or "Continue suction drive (no control required)," the suction drive control mode is stored in the image registration memory 211, associated with the registered image of the foreign object.

[0048] The above description concerns the registration process of images of foreign objects using cameras 171-173 provided by the electric cleaning device 1. In this configuration example, the registration process of images of foreign objects is not limited to the method of capturing images of foreign objects using cameras 171-173 provided by the electric cleaning device 1, but is also configured to allow registration through an external interface 212, for example, via a storage medium such as a USB memory or SD card. In that case, the storage medium such as a USB memory or SD card stores the registered image captured in the same way as captured by cameras 171-173, the name of the foreign object, and its suction drive control mode, and this stored information is transferred to and stored in the image registration memory 211.

[0049] Furthermore, the manufacturer of the electric vacuum cleaner 1 can provide various images of foreign objects, their names, and their suction drive control modes on a website operated by the manufacturer. In this case, the user can access the website using, for example, a personal computer, obtain information on the foreign object they wish to register, store it on a USB memory stick or SD card, and transfer it to the image registration memory 211 via the external interface 212 to register the image of the foreign object. Of course, if the electric vacuum cleaner 1 has a network connection function, the user can access the website directly without a personal computer.

[0050] Figure 3 shows an example of a foreign object stored in the image registration memory 211. In this example in Figure 3, the image registration memory 211 stores the foreign object name, registered images V01, V02, V03, ..., thumbnail images SM01, SM02, SM03, ..., and the suction drive control mode, all associated with the foreign object identification information (foreign object ID) assigned to each foreign object. In this configuration example, the suction drive control mode for the suction drive unit 102 is set to "stop suction drive" or "continue suction drive (no control required)".

[0051] Each of the registered images V01, V02, V03, ... consists of image information from multiple captured images, as described above, and is associated with the foreign object ID and stored as a single set. The thumbnail images SM01, SM02, SM03, ... are used to display what the recognized foreign object is on the display screen 16D when the image recognition unit 210 recognizes the foreign object, and consist of reduced images of captured images from cameras 171 to 173 that are easily recognizable by the user as representing the foreign object in question.

[0052] Alternatively, instead of pre-registering thumbnail images in the image registration memory 211, when the image recognition unit 210 recognizes a foreign object, the display image generation unit 213 may retrieve the registered captured image for the recognized foreign object from the image registration memory 211 based on the recognition result and generate the thumbnail image.

[0053] In the example shown in Figure 3, the mode of suction drive control when each foreign object is recognized is stored as either "(suction operation) stop" or "(suction drive control) not required," both from the perspective of avoiding malfunction or damage to the electric cleaning device 1, and from the perspective of ensuring the cleanliness of the foreign objects that are sucked in and avoiding damage.

[0054] In other words, in the example shown in Figure 3, "plastic bags," "cash bags," "documents," "cords," "pins, screws, and nails" may cause the device to stop at the suction port 141 or get caught in the joint pipe 13 or suction hose 11, potentially damaging them. Therefore, to avoid malfunction or damage to the electric cleaning device 1, the suction drive control is set to "(suction operation) stopped." Although not illustrated in Figure 3, the same applies to "toothpicks and needles."

[0055] Furthermore, in the example in Figure 3, the suction drive control mode for "documents," "microSD cards," and "USB memory" is set to "(suction operation) stopped" from the perspective of ensuring the cleanliness of foreign objects that may be sucked in and avoiding contamination or damage. Although not illustrated in Figure 3, the same applies to "SD cards, SIM cards," etc. In the example in Figure 3, the suction drive control mode for "banknotes," "gold coins," and "jewelry" is set to "(suction drive control) not required," but it may also be set to "(suction operation) stopped" to avoid contamination or damage. Although not illustrated in Figure 3, the same applies to "checks, stamps," etc. Note that the setting of the suction drive control mode may be configured to be changeable by the user.

[0056] In this configuration example, the image recognition unit 210 performs image recognition by comparing each of the multiple registered images of foreign objects stored in the image registration memory 211, as described above, with the captured images received through the captured image memory 209 from each of the cameras 171, 172, and 173, using pattern matching processing. When the image recognition unit 210 recognizes a foreign object among the multiple registered images of a single foreign object that matches or shows a similarity level above a predetermined similarity level from the captured images from the cameras 171, 172, and 173, it determines that the foreign object in that registered image has been recognized and outputs the foreign object ID of that registered image as the image recognition result to the display image generation unit 213. At this time, the captured image from the camera that recognized the foreign object in the image recognition unit 210 is supplied from the captured image memory 209 to the display image generation unit 213 and is stored in association with the foreign object ID of the image recognition result.

[0057] Furthermore, when the image recognition unit 210 determines that it has recognized a foreign object, it also detects which of the cameras 171, 172, or 173 captured the image in which the foreign object was recognized. That is, the image recognition unit 210 determines the foreign object that was recognized in the image and which of the cameras 171, 172, or 173 captured the image in which the foreign object was recognized. Based on which camera captured the image in which the foreign object was recognized, the image recognition unit 210 then determines whether the recognized foreign object has been sucked into the dust collection chamber 101 of the vacuum cleaner body 10, or whether it is outside the electric vacuum cleaner 1 without being sucked in.

[0058] In other words, when the image recognition unit 210 determines that it has recognized a foreign object in the image captured by camera 171 or 172 located near the intake port 141, it determines that there is a high probability that the detected foreign object has been sucked into the dust collection chamber 101. In this configuration example, the image recognition unit 210 further refers to the image recognition result for the image captured by camera 174 and determines whether or not a foreign object has been detected in the image captured by camera 174, and then determines whether or not the recognized foreign object has actually been sucked into the dust collection chamber 101. When the image recognition unit 210 determines that it has recognized a foreign object in the image captured by camera 171 or 172 located near the intake port 141, it may determine that the detected foreign object has been sucked into the dust collection chamber 101 without referring to the recognition result from camera 174. However, by referring to the recognition result for the image captured by camera 174, it is possible to determine with a higher degree of accuracy whether or not the object has been sucked into the dust collection chamber 101.

[0059] Furthermore, when the image recognition unit 210 determines that it has recognized a foreign object in the image captured by the camera 173, which captures an image of the area in front of the suction unit 14, it also constantly refers to the image recognition result for the image captured by the camera 174 to determine whether or not a foreign object has been detected, and to determine whether or not a foreign object has been sucked into the dust collection chamber 101.

[0060] As described above, when the image recognition unit 210 recognizes a foreign object through image recognition, in addition to the foreign object ID of the recognized foreign object in the image recognition result, it determines whether the location of the foreign object is inside or outside, that is, whether it has been sucked into the dust collection chamber 101 (inside) or whether it is outside the electric cleaning device 1 without being sucked in (outside). The image recognition unit 210 then supplies the foreign object ID of the recognized foreign object, along with the location information of the foreign object (information on whether it is inside or outside), as the image recognition result to the display image generation unit 213.

[0061] Furthermore, in this configuration example, as described above, when the image recognition unit 210 recognizes a foreign object, the image captured by the camera 173 at that time is supplied from the image memory 209 to the display image generation unit 213. The display image generation unit 213 then stores the image captured by the camera 173, associating it with the foreign object ID of the image recognition result sent from the image recognition unit 210.

[0062] The image captured by the camera 173, which is stored in memory, is an image of the area in front of the suction unit 14 at the location where the foreign object was recognized, and serves as a surrounding image to determine where the foreign object was located. By looking at the surrounding image of the location where the foreign object was recognized, captured by the camera 173, the user can get a rough idea of ​​where in the room being cleaned the foreign object was detected.

[0063] The display image generation unit 213 generates display image information to be displayed on the display unit 16 via the display control unit 203. In this configuration example, in addition to the function of generating guide images during the foreign object registration process described above, the unit also generates images to notify the user of foreign objects recognized during the cleaning operation based on the image recognition result information sent from the image recognition unit 210. In this configuration example, the image of the foreign object recognized during the cleaning operation is a thumbnail image, which is a reduced image of the captured image stored in the image registration memory 211.

[0064] In other words, the display image generation unit 213 receives the foreign object ID as the result of foreign object recognition from the image recognition unit 210, its location information, and the image captured by the camera 173 at the time of foreign object recognition. Then, based on the foreign object ID obtained from the image recognition unit 210, the display image generation unit 213 obtains thumbnail image information SMi (i=01, 02, ...) associated with the foreign object ID from the image registration memory 211, and displays the thumbnail image 221 based on that thumbnail image information on the display screen 16D via the display control unit 203, as shown in Figure 4(A).

[0065] Furthermore, as shown in Figure 4(A), the display image generation unit 213 provides a foreign object name field 222 in the display area of ​​each thumbnail image 221. The display image generation unit 213 then receives the name of the foreign object from the image registration memory 211 based on the foreign object ID obtained from the image recognition unit 210, and displays the name of the foreign object in the foreign object name field 222.

[0066] Furthermore, as shown in Figure 4(A), the display image generation unit 213 provides a display field 223 for the location of the foreign object in the display field of each thumbnail image 221. The display image generation unit 213 then generates additional display information indicating whether the location of the foreign object is inside or outside from the location information sent from the image recognition unit 210 along with the foreign object ID, and displays the generated location information in the display field 223 as shown in Figure 4(A). In Figure 4(A), a foreign object displayed as "inside" in the display field 223 is located inside the dust collection chamber 101 of the vacuum cleaner body 10, and a foreign object displayed as "outside" is located outside and has not been sucked in.

[0067] The display image generation unit 213 performs the above operation each time it receives the recognition result of a foreign object from the image recognition unit 210 during the cleaning operation, and adds thumbnail images 221 of the recognized foreign objects to the display screen 16D, as shown in Figure 4(A). As a result, as shown in Figure 4(A), at the end of the cleaning operation, a list of foreign objects recognized during the cleaning operation is displayed on the display screen 16D as a list of thumbnail images.

[0068] As mentioned above, the suction drive control mode for each foreign object is registered and stored in the image registration memory 211. When the suction drive control mode for a foreign object recognized by the image recognition unit 210 is set to "(suction operation) stopped," the suction drive unit 102 is forcibly stopped. In this configuration example, when the suction drive control mode for a foreign object corresponding to the foreign object ID of the image recognition result from the image recognition unit 210 is set to "(suction operation) stopped," the display image generation unit 213 provides a display field 224 for suction drive stop control in the display field of the thumbnail image 221. The display field 224 displays the word "Stopped" to indicate that suction drive stop control has been performed, informing the user of what the foreign object was that caused the forced stop.

[0069] In this configuration example, when a user clicks on one of the thumbnail images displayed on the display screen 16D, the display image generation unit 213, based on the control of the control unit 201, recognizes the foreign object ID corresponding to the clicked thumbnail image, and displays the captured image 241 in which the foreign object was recognized by the image recognition unit 210, which stores the recognized foreign object ID, in half of the display screen 16D, as shown in Figure 4(B). Furthermore, the display image generation unit 213 displays the captured image (peripheral image) 242 from the camera 173, which stores the recognized foreign object ID, in the remaining area of ​​the display screen 16D, as shown in Figure 4(B).

[0070] The example in Figure 4(B) shows what happens when a thumbnail image of a ring is clicked by the user, and captured image 241 is an image of the ring taken by either camera 171 or 172. Captured image 242 is a peripheral image in front of the position where the electric cleaning device 1 sucked up the ring, and in the example in Figure 4(B), it shows the gap between two pieces of furniture being photographed.

[0071] By viewing the captured image 241, the user can accurately identify the foreign object that was recognized in the image, and by viewing the captured image 242, they can determine which area of ​​the cleaning process the foreign object was in when it was recognized in the image.

[0072] Next, the alarm drive unit 214 is connected to the alarm generation unit 215. When the image recognition unit 210 recognizes a foreign object, the alarm drive unit 214 controls the alarm generation unit 215 based on the image recognition result to generate an alarm and notify the user that a foreign object has been detected. As mentioned above, when the image recognition unit 210 recognizes a foreign object, a thumbnail image of the recognized foreign object is displayed on the display screen 16D based on the image recognition result, and the alarm generation unit 215 generates an alarm in conjunction with (synchronized with) the display of this thumbnail image of the foreign object. The user learns that the image recognition unit 210 has recognized a foreign object when the alarm is generated, and learns what the recognized foreign object is by looking at the thumbnail image displayed on the display screen 16D.

[0073] In this example, the alarm generation unit 215 consists of a buzzer sound generation unit. However, the alarm generation unit 215 is not limited to a buzzer sound generation unit; it may also consist of a vibration generation unit such as a vibrator, or a speaker that emits alarm sounds or alarm music. Furthermore, a combination of alarm sound and vibration may be used.

[0074] Furthermore, in this configuration example, the alarm generation unit 215 causes the thumbnail image of the recognized foreign object displayed on the display screen 16D to flash. This allows the user to easily recognize what the foreign object is at that particular moment, even if multiple thumbnail images of foreign objects are displayed on the display screen 16D, as will be described later.

[0075] In addition, in the configuration of the control circuit unit 20 shown in Figure 2, the image recognition unit 210, the display image generation unit 213, and the alarm drive unit 214 can also be implemented as software functions of the microcontroller that constitutes the control unit 201.

[0076] [Explanation of the processing operation in the control circuit section 20 of the electric cleaning device 1] Figure 5 is a flowchart illustrating the processing flow in the control circuit unit 20 of the electric cleaning device 1 in the first configuration example. In the following explanation of the flowchart in Figure 5, it is assumed that the control unit 201 implements the functions of the image recognition unit 210, the display image generation unit 213, and the alarm drive unit 214 as software functions.

[0077] First, the control unit 201 monitors the user's input through the operation unit 15 via the operation unit interface 204 and waits for the suction drive start button to be pressed (step S101). In step S101, when the control unit 201 determines that the suction drive start button has been pressed, it supplies a suction drive start control signal to the suction drive unit 102 via the drive control unit 202 to start the suction drive, and also supplies a camera shooting start control signal to each of the cameras 171 to 174 to start camera shooting (step S102).

[0078] Then, the control unit 201 performs the image recognition processing described above on the image information captured from cameras 171 to 174 (step S103) and determines whether or not a foreign object has been recognized (step S104). If, in step S104, it is determined that no foreign object has been recognized, the control unit 201 monitors the user's operation input through the operation unit 15 via the operation unit interface 204 and determines whether or not the suction drive stop button has been pressed (step S105).

[0079] Then, in step S105, if it is determined that the suction drive stop button has not been pressed, the control unit 201 returns to step S103 and repeats the process from step S103 onward. Also, if it is determined in step S105 that the suction drive stop button has been pressed, the control unit 201 supplies a suction drive stop control signal to the suction drive unit 102 through the drive control unit 202 to stop the suction drive, and also supplies camera shooting stop control signals to each of the cameras 171 to 174 to end camera shooting (step S106). Then, the control unit 201 returns to step S101 and repeats the process from step S101 onward.

[0080] Then, in step S104, when it is determined that a foreign object has been detected, the control unit 201 sounds an alarm using the alarm generation unit 215 and displays a thumbnail image of the recognized foreign object on the display screen 16D. At this time, the control unit 201 determines, as described above, whether the foreign object is in the dust collection chamber 101 or outside without being sucked in, and displays the result of this determination on the thumbnail image as shown in Figure 4(A) (step S107).

[0081] Next, the control unit 201 refers to the contents of the image registration memory 211 to obtain information on the suction drive control mode stored in correspondence with the foreign object ID of the recognized foreign object, and determines whether or not it is necessary to forcibly stop the suction operation (step S108).

[0082] If, in step S108, it is determined that it is not necessary to forcibly stop the suction operation, the control unit 201 continues the suction operation of the suction drive unit 102 (step S109), returns the process to step S105, and repeats the process from step S105 onward.

[0083] At this time, the electric cleaning device 1 is controlled to continue the suction operation, but the user can press the drive stop button if necessary after seeing the alarm generated in step S107 and the thumbnail image on the display screen 16D. Then, the control unit 201 detects that the drive stop button has been pressed in step S105 and stops the suction drive.

[0084] Next, in step S108, if it is determined that it is necessary to forcibly stop the suction operation, the control unit 201 supplies a control signal to the suction drive unit 102 to stop the suction operation, thereby forcibly stopping the suction operation. In this forcibly stopped state of the suction operation, the control unit 201 locks the control unit 15 so that operations such as restarting the suction operation through the control unit 15 cannot be performed (step S110).

[0085] This locked state can be released when the user removes any foreign objects near the suction port 141 or opens the dust collection chamber 101 and removes any foreign objects, and then inputs an unlock operation through the control panel 15.

[0086] After step S110, the control unit 201 determines whether or not the lock has been released (step S111). If it determines that the lock has not been released, it performs an insensitive process that ignores operation inputs such as restart (step S112). After step S112, the control unit 201 returns to step S111 and repeats the process from step S111 onward.

[0087] Furthermore, if it is determined in step S111 that the lock has been released, the control unit 201 returns to step S101 and repeats the process from step S101 onward.

[0088] In the above explanation, when the suction drive is stopped in response to the detection of a foreign object, the device is locked and becomes insensitive to normal operation using the start button. The device is then restarted after this lock state is released. However, depending on the detected foreign object, it may not be necessary to remove the foreign object near the suction port 141 or to open the dust collection chamber 101 and remove the foreign object. In this configuration example, the user can see the foreign object detected by the electric cleaning device 1 on the display screen 16D. Considering the above, the operation unit 15 of the electric cleaning device 1 may be equipped with a "continue cleaning" button that allows the user to restart the suction drive without having to perform the aforementioned unlock operation after confirming it on the display screen 16D.

[0089] [Effects of the first configuration example] As described above, the electric cleaning device 1 of the first configuration example detects foreign objects other than the intended cleaning target by image recognition during cleaning operation, displays a thumbnail image of the detected foreign object on the display screen 16D, and notifies the user of the detection of the foreign object by alarm. Therefore, the user can immediately find out what the foreign object is by hearing the alarm and looking at the thumbnail image on the display screen 16D. In this case, the user does not need to perform any playback operations as in the prior art described at the beginning, which is very convenient.

[0090] Furthermore, the user can conveniently determine whether it is better to stop the suction operation after checking the detected foreign object in the thumbnail image. Moreover, in this first configuration example, foreign objects for which it is better to forcibly stop the suction operation can be registered in advance, allowing the suction operation to be automatically stopped, preventing damage to the electric cleaning device 1 and preventing foreign objects that should be kept clean from being sucked into the dust collection chamber 101.

[0091] Furthermore, the thumbnail images of recognized foreign objects include additional indications showing whether the foreign object has been sucked into the dust collection chamber 101 or is located outside without being sucked in, allowing the user to easily determine the location of the foreign object.

[0092] Furthermore, in this configuration example, when a user clicks on a thumbnail image displayed on the display screen 16D, the actual image of the detected foreign object is displayed, along with an image of the surrounding area in front of the suction unit 14 at the cleaning location where the foreign object was image-recognized. This allows the user to accurately identify what the detected foreign object is and to know where it was located, which is convenient.

[0093] In particular, when foreign objects hidden in gaps between furniture or under beds are detected, the location of the foreign object can be determined from the surrounding image captured in front of the suction unit 14 at the cleaning location where the foreign object was image-recognized, even if the foreign object has not been sucked up. Furthermore, the electric vacuum cleaner 1 in this configuration example can recognize foreign objects in places that the electric vacuum cleaner 1 is not cleaning or places that cannot be cleaned, based on the recognition results of the image captured by the camera 173. Therefore, the electric vacuum cleaner 1 in this configuration example is very convenient because it can recognize something that the user thought was lost as a foreign object, regardless of whether it is on the floor or other surface being cleaned, and inform the user of its location. Needless to say, in places such as gaps between furniture or under beds, these areas are brightly illuminated by the lighting unit attached to the camera and captured by the camera.

[0094] Furthermore, while the vacuum cleaner described in Patent Document 1 at the beginning can stop suction when it determines that an object is larger than the area of ​​the suction opening, it may not stop suctioning string-like objects such as electrical outlet cords and AC adapter cables, or plastic bags, for which it is difficult to determine whether their size is larger than the area of ​​the suction opening. In addition, there is a problem that it cannot prevent the vacuum cleaner from sucking up foreign objects that are smaller than the area of ​​the suction opening but should not be sucked up, such as microSD cards, small media such as SD cards and SIM cards, or sharp screws and nails.

[0095] In contrast, with the first example of the electrical cleaning device described above, string-like objects such as electrical outlet cords and AC adapter cables, plastic bags, microSD cards, small media such as SD cards and SIM cards, screws and nails are registered in the image registration memory and defined to stop the suction operation. Therefore, these foreign objects that should stop the suction operation can be reliably recognized and stopped by image recognition.

[0096] Furthermore, with the electric cleaning device of the first configuration example described above, when the cleaning operation is completed, a list of thumbnail images of foreign objects recognized during the cleaning operation is displayed on the screen. Therefore, the user can easily determine whether or not there are any foreign objects that need to be removed from the dust collection chamber by looking at this list of images. Moreover, when a thumbnail image is clicked, the actual image of the captured foreign object is enlarged and displayed on the screen. Therefore, even if the details of a foreign object are unclear or uncertain in the thumbnail image, the enlarged image of the captured image allows for reliable confirmation of the foreign object.

[0097] [Second configuration example of an electric cleaning device related to this invention] A second configuration example of an electric vacuum cleaner related to this invention consists of a self-propelled electric vacuum cleaner and a portable device equipped with a display unit, such as a mobile phone terminal. In this second configuration example, the self-propelled electric vacuum cleaner and the mobile phone terminal are connected via a wireless communication path. In the following example, a short-range wireless communication path conforming to the Bluetooth® standard is used. However, the wireless communication path is not limited to this, and a wireless LAN (Local Area Network) such as Wi-Fi® or other wireless communication means can also be used.

[0098] [Overview of the 30-inch electric vacuum cleaner] Figure 6 is a diagram illustrating the self-propelled vacuum cleaner 30 that constitutes the second configuration example of the electric cleaning device. Figure 6(A) is a view of the self-propelled vacuum cleaner 30 of this example, showing the bottom surface 30a side with its suction port 31, and Figure 6(B) is a view of the self-propelled vacuum cleaner 30 of this example, viewed from the direction of travel during cleaning operation. In Figure 6(A), arrow AR indicates the direction of travel (forward direction) of the self-propelled vacuum cleaner 30 during cleaning operation.

[0099] As shown in Figure 6(A), the self-propelled electric vacuum cleaner 30 has a nearly disc-shaped housing, with a suction port 31 provided on its bottom surface 30a. Inside the housing of the electric vacuum cleaner 30, as shown by the dotted line in Figure 6(B), there is a dust collection chamber 32 that communicates with the suction port 31 and collects the object to be cleaned and any foreign matter other than the object to be cleaned. Although not shown in Figure 6, the housing of the electric vacuum cleaner 30 is also equipped with a suction drive unit that sucks the object to be cleaned and any foreign matter other than the object to be cleaned into the dust collection chamber 32.

[0100] As shown in Figure 6(A), the vacuum cleaner 30 in this configuration example is equipped with a flexible brush 33 and a rolling brush 34 inside the suction port 31. A roller 35 is provided approximately in the center in front of the suction port 31 on the bottom surface 30a of the vacuum cleaner 30's housing in the direction of travel during cleaning. This roller 35 is mounted so as to be rotatable in a plane in which the axial direction of its rotation axis is parallel to the bottom surface 30a. Therefore, the axial direction of the rotation axis of the roller 35 can be directed not only in the direction perpendicular to arrow AR, but also in any direction. On both sides of the suction port 31 on the bottom surface 30a, there are wheels 36 and 37 that are rotationally driven by a drive unit (not shown in Figure 6).

[0101] The vacuum cleaner 30 moves under its own power as the wheels 36 and 37 are driven and rotated by the drive unit. As mentioned above, the roller 35 can rotate in any direction as its axis of rotation, so the vacuum cleaner 30 can not only move in a straight line in the direction of arrow AR, but also rotate and move in any direction intersecting arrow AR as its direction of travel. Furthermore, by controlling the rotation direction of the wheels by the drive unit, the vacuum cleaner 30 can also move backward in the opposite direction to arrow AR.

[0102] In this configuration example, cameras 381 and 382 are provided near the suction port 31 on the bottom surface 30a, with the direction perpendicular to the bottom surface 30a being the shooting direction (optical axis direction). These cameras 381 and 382 correspond to cameras 171 and 172 in the electric cleaning device 1 of the first configuration example.

[0103] Furthermore, a camera 383 is provided on the side of the housing in front of the roller 35 of the vacuum cleaner 30, such that the shooting direction (optical axis direction) is approximately parallel to the bottom surface 30a. In addition, a camera 384 is provided on the side of the housing behind the roller 35 of the vacuum cleaner 30, such that the shooting direction (optical axis direction) is approximately parallel to the bottom surface 30a. Camera 383 photographs the area in front of the self-propelled vacuum cleaner 30 in the direction of travel at a predetermined angle of view. Camera 384 photographs the area behind the self-propelled vacuum cleaner 30 in the direction of travel at a predetermined angle of view. These cameras 383 and 384 correspond to camera 173 in the vacuum cleaner device 1 of the first configuration example.

[0104] In other words, in the second configuration example, the surrounding images of the location where the foreign object was recognized are not only the images captured in front of the vacuum cleaner 30 in the direction of travel, but also the images captured behind it, which are stored in association with the recognized foreign object (foreign object ID).

[0105] Furthermore, since cameras 383 and 384 have a predetermined field of view, as long as they can capture images of the floor surface to be cleaned in the direction of travel of the vacuum cleaner 30, their shooting direction (optical axis direction) does not need to be parallel to the bottom surface 30a.

[0106] In this second configuration example, the vacuum cleaner 30 does not have an image recognition function or a display unit. These image recognition functions and the display unit are provided by a mobile phone terminal connected to the vacuum cleaner 30 via a wireless communication path.

[0107] The four cameras 381 to 384 described above are configured, similarly to the first configuration example, to include, for example, a CCD image sensor or a CMOS image sensor and an imaging lens.

[0108] [Circuit configuration of the electric vacuum cleaner in the second example configuration] Figure 7 shows the electrical circuit configuration of the vacuum cleaner 30 in this second configuration example and the mobile phone terminal 40 that is wirelessly connected to the vacuum cleaner 30.

[0109] In other words, as shown in Figure 7, the vacuum cleaner 30 has an electrical circuit configuration in which the drive control unit 302, the transmitted image information generation unit 303, the operation unit 304, the wireless communication unit 305, and the cameras 381, 382, ​​383, and 384 are connected via the system bus 300 to the control unit 301 which is equipped with a microcomputer. In Figure 7, the camera interface is omitted from the illustration for the sake of simplicity.

[0110] The drive control unit 302 is connected to a suction drive unit 306 and a travel drive unit 307. The suction drive unit 306 is for sucking up the object to be cleaned and any foreign objects other than the object that can be sucked up from the suction port 31. The travel drive unit 307 rotates the wheels 36 and 37 to allow the vacuum cleaner 30 to move on its own (self-propelled).

[0111] When the control unit 301 receives input from pressing the cleaning start button on the operation unit 304, it supplies a drive start control signal to the suction drive unit 306 and the travel drive unit 307 via the drive control unit 302 to start driving, thereby initiating the automatic cleaning operation of the vacuum cleaner 30 while it is moving on its own. Furthermore, when the control unit 301 receives input from pressing the cleaning stop button on the operation unit 304, or when it determines that cleaning is complete based on the cleaning support program stored in the memory built into the control unit 301, it supplies a drive stop control signal to the suction drive unit 306 and the travel drive unit 307 via the drive control unit 302 to stop driving the suction drive unit 306 and the travel drive unit 307, thereby stopping the cleaning operation of the vacuum cleaner 30.

[0112] Furthermore, in this second configuration example, when the control unit 301 receives a forced stop control signal from the mobile phone terminal 40 via the wireless communication unit 305 based on a cleaning support program stored in its built-in memory, it supplies the forced stop control signal to the suction drive unit 306 and the travel drive unit 307 via the drive control unit 302, thereby forcibly stopping their drive and forcibly stopping the cleaning operation of the vacuum cleaner 30.

[0113] Each of the cameras 381 to 384 is controlled by the control unit 301 to perform imaging operations during cleaning, in the same manner as in the first configuration example. In this second configuration example, each of the cameras 381, 382, ​​383, and 384 is also equipped with an illumination unit 381L, 382L, 383L, and 384L, for example, consisting of LEDs, and is controlled to light up by the control unit 301, in the same manner as in the first configuration example.

[0114] The transmitted image information generation unit 303 generates transmitted image information from each of the cameras 381 to 384 while the vacuum cleaner 30 is in cleaning operation. In this case, the transmitted image information generation unit 303 adds camera identification information (camera ID) to the transmitted image information from each of the cameras 381 to 384 to indicate which camera the image information is from. The transmitted image information generation unit 303 also performs encoding processing, such as data compression, on the transmitted image information from each of the cameras 381 to 384 as appropriate to generate transmitted image information, and then transfers it to the wireless communication unit 305.

[0115] As mentioned above, in this configuration example, the wireless communication unit 305 has the functionality of short-range wireless communication according to the Bluetooth® standard. The wireless communication unit 305 wirelessly transmits the transmitted image information from the transmitted image information generation unit 303 to the mobile phone terminal 40. The wireless communication unit 305 also receives a forced stop control signal based on the image recognition result sent from the mobile phone terminal 40 and passes it to the control unit 301 via the system bus 300.

[0116] As described above, when the control unit 301 receives a forced stop control signal via the wireless communication unit 305, it supplies the forced stop control signal to the suction drive unit 306 and the travel drive unit 307 via the drive control unit 302, thereby forcibly stopping their drive and forcibly stopping the cleaning operation of the vacuum cleaner 30.

[0117] In the configuration of the vacuum cleaner 30 shown in Figure 7, the functions of the drive control unit 302 and the transmitted image information generation unit 303 can be configured by the control unit 301 as software functions.

[0118] Next, we will describe the configuration example of the mobile phone terminal 40 used in this second configuration example. The mobile phone terminal 40 in this configuration example is a high-function mobile phone terminal known as a smartphone (abbreviated as "smartphone" in Figure 8). In other words, the mobile phone terminal 40 in this example has a configuration in which each part, which will be described later, is controlled by a control unit 401 equipped with a microcomputer.

[0119] The control unit 401 is connected via the system bus 400 to the wireless communication unit 402, the display control unit 403, the touch panel interface 404, the captured image storage unit 405, the image recognition unit 406, the image registration memory 407, the external interface 408, the display image generation unit 409, the mobile phone function unit 410, and the camera unit 411.

[0120] As described above, in this configuration example, the wireless communication unit 402 has a short-range wireless communication function conforming to the Bluetooth® standard and communicates wirelessly with the vacuum cleaner 30. This wireless communication unit 402 performs wireless communication completely independently of the mobile phone communication that the mobile phone terminal 40 performs via a mobile phone base station.

[0121] A display unit 412, which is, for example, an LCD, is connected to the display control unit 403. A touch panel 413 is connected to the touch panel interface 404. The touch panel 413 accepts touch input from a finger or a pen-type position indicator and outputs position information of the touch input. The touch panel 413 is transparent and is superimposed on the display screen of the display unit 412, which in this example is an LCD screen. Through the touch panel 413, the user can view the display image displayed on the display screen of the display unit 412 and can input instructions to the touch panel 413 as if they were inputting touch input to the display image. The position information of the touch input from the touch panel 413 corresponds to the position information on the display image displayed on the display screen.

[0122] The control unit 401 receives position information of touch operation input from the touch panel 413 via the touch panel interface 404, determines the control process set according to the user's instruction operation input on the touch panel 413 and the position of that operation, and executes the control process according to the determination result. The processing program for the control unit 401 to execute the processing function according to the position information of the touch operation input from the touch panel 413 is stored in the memory built into the control unit 401.

[0123] The display control unit 403 has the same functions as the display control unit 203 in the first configuration example, and has video RAM for storing display image data to be displayed on the display screen of the display unit 412. Based on the control of the control unit 401, it displays images as a mobile phone terminal 40, and when the cleaning support program described later is started, it displays the aforementioned thumbnail image, etc.

[0124] The image storage unit 405 decodes and separates the image information captured by cameras 381 to 384 from the transmitted image information sent from the vacuum cleaner 30 via the wireless communication unit 402, and temporarily stores it for image recognition by the image recognition unit 406. The image storage unit 406 has the same functions as the image memory 209 in the first configuration example, except for the function of decoding and separating the image information captured by cameras 381 to 384 from the transmitted image information.

[0125] The image recognition unit 406 is activated when a program supporting cleaning operations in the vacuum cleaner 30 is running, and performs the same processing operations as the image recognition unit 210 in the first configuration example described above. However, in this second configuration example, when a foreign object is recognized by the image recognition unit 406, the control unit 301 refers to the suction drive control mode stored in the image registration memory 407 corresponding to the foreign object ID of the recognized foreign object. If the suction drive control mode is stopped (forced stop), it wirelessly transmits a forced stop control signal to the vacuum cleaner 30 via the wireless communication unit 402.

[0126] Furthermore, the image registration memory 407 has the same function as the image registration memory 211 in the first configuration example described above. That is, in this second configuration example, by using the display unit 412 of the mobile phone terminal 40 instead of the display unit 16 in the first configuration example, images of foreign objects captured by cameras 381-384 can be registered and stored in the image registration memory 407 in the same way as described in the first configuration example. Also, in this second configuration example, images of foreign objects can be registered and stored in the image registration memory 407 from a USB memory or SD card via the external interface 408, similar to the first configuration example. In addition, images of foreign objects can be captured with the camera unit 411 of the mobile phone terminal 40 and registered and stored in the image registration memory 407. Furthermore, the mobile phone function unit 410 can access a website server that holds images of foreign objects via the internet, obtain the necessary images of foreign objects from the server, and register and store them in the image registration memory 407.

[0127] The program that assists in the process of registering images of foreign objects to the image registration memory 407 is downloaded in advance via the Internet from a server of a website that supports the functions of the vacuum cleaner 30 and stored in the memory built into the control unit 401 of the mobile phone terminal 40. Similarly, the cleaning support program that performs foreign object recognition during the cleaning operation of the vacuum cleaner 30, which will be described later, is also downloaded from the same server and stored in the memory built into the control unit 401 of the mobile phone terminal 40.

[0128] The display image generation unit 409 is activated when a program that supports cleaning operations in the vacuum cleaner 30 is running, and performs the same processing operations as the display image generation unit 213 in the first configuration example.

[0129] The mobile phone function unit 410 includes a transmitter unit and a receiver unit, and is a processing function unit that performs mobile phone functions such as making calls and sending emails via the mobile phone network. In this configuration example, the functions of the mobile phone are not directly related to this invention, so a detailed explanation thereof is omitted in this specification.

[0130] In the configuration of the mobile phone terminal 40 shown in Figure 7, the processing function portion of the image recognition unit 406 and the captured image storage unit 405 (excluding the memory), and the functions of the display image generation unit 409 can be configured as software functions by the control unit 401.

[0131] [Explanation of the process during cleaning in the second configuration example] In this second configuration example, the user activates a program on the mobile phone terminal 40 to assist with the cleaning operation of the vacuum cleaner 30 before pressing the start cleaning button on the control panel 304 of the vacuum cleaner 30. Then, when the user presses the start cleaning button on the control panel 304 of the vacuum cleaner 30, the following processing operations are executed on both the vacuum cleaner 30 and the mobile phone terminal 40.

[0132] Figure 8 is a flowchart illustrating an example of the processing flow in the vacuum cleaner 30 at this time. Figure 9 is a flowchart illustrating an example of the processing flow in the mobile phone terminal 40 when the cleaning support program is activated.

[0133] First, the processing operation of the vacuum cleaner 30 will be explained with reference to Figure 8. Note that the following explanation assumes that the control unit 301 performs the functions of the drive control unit 302 and the transmitted image generation unit 303 as software functions.

[0134] The control unit 301 determines whether the cleaning start button has been pressed and waits for cleaning to start (step S301). If it is determined in step S301 that the cleaning start button has been pressed, the control unit 301 starts driving the suction drive unit 306 and the travel drive unit 307, and instructs each of the cameras 381 to 384 to start capturing images. Furthermore, the wireless communication unit 305 generates a wireless communication path to the mobile phone terminal 40 and establishes a wireless connection with the mobile phone terminal 40 (step S302).

[0135] Then, the control unit 301 collects images captured from each of the cameras 381 to 384 during the cleaning operation, generates transmission image information (step S303), and transmits it to the mobile phone terminal 40 via the wireless communication unit 305 (step S304).

[0136] Then, the control unit 301 monitors the received signal from the wireless communication unit 305 to determine whether or not it has received a forced stop control signal from the mobile phone terminal 40 (step S305). If it is determined in step S305 that a forced stop control signal has not been received, the control unit 301 determines whether or not it has detected the end of cleaning based on whether or not it has completed cleaning all of the cleaning area (step S306). If it has not detected the end of cleaning, it returns to step S303, generates the transmission image information, and continues the cleaning operation while wirelessly transmitting it to the mobile phone terminal 40.

[0137] Then, in step S305, when the control unit 301 determines that it has received a forced stop control signal from the mobile phone terminal 40, it stops the operation of the suction drive unit 306 and the travel drive unit 307, and instructs each of the cameras 381 to 384 to stop taking images. Furthermore, the wireless communication unit 305 disconnects the wireless communication path with the mobile phone terminal 40 (step S307).

[0138] In step S306, when the end of cleaning is detected, the control unit 301 proceeds to step S307, stopping the operation of the suction drive unit 306 and the travel drive unit 307, instructing each of the cameras 381 to 384 to stop taking images, and further disconnecting the wireless communication path with the mobile phone terminal 40 via the wireless communication unit 305.

[0139] This concludes the processing for the cleaning operation of the vacuum cleaner 30.

[0140] Next, with reference to Figure 9, the processing operation of the mobile phone terminal 40 will be explained. Note that the following explanation assumes that the control unit 401 executes the processing functions of the image recognition unit 406, the captured image storage unit 405 (excluding memory), and the display image generation unit 409 as software functions.

[0141] First, the control unit 401 determines whether or not the cleaning support program has been started (step S401). If, in step S401, it determines that the cleaning support program has not been started, the control unit 401 executes other processing (step S402), and after completing the other processing, returns to step S401.

[0142] Then, in step S401, when it is determined that the cleaning support program has been activated, the control unit 401 generates a wireless communication channel with the vacuum cleaner 30 using the wireless communication unit 402 (step S403). Subsequently, the control unit 401 receives transmitted image information sent from the vacuum cleaner 30 through the generated wireless communication channel, separates and acquires the captured image information from each of the cameras 381 to 384, and performs image recognition processing by comparing the captured images from each of the cameras 381 to 384 with the images of foreign objects stored in the image registration memory 407 using pattern matching, as in the first configuration example (step S404).

[0143] Then, the control unit 401 determines whether or not a foreign object was detected based on the image recognition results in step S404 (step S405). If it is determined in step S405 that no foreign object was detected, the control unit 401 determines whether or not the wireless communication path was cut off by the vacuum cleaner 30 (step S409). If it is determined in step S409 that the wireless communication path was cut off, the control unit 401 determines that the cleaning operation of the vacuum cleaner 30 has stopped and terminates this processing routine.

[0144] Furthermore, if step S409 determines that the wireless communication path has not been disconnected, the control unit 401 determines that the cleaning operation of the vacuum cleaner 30 is continuing, returns the process to step S404, and repeats the process from step S404 onward.

[0145] Then, in step S405, if the image recognition determines that a foreign object has been detected, the control unit 401 sounds an alarm and displays a thumbnail image of the recognized foreign object on the display screen of the display unit 412. The alarm can be activated using one of the ringtones provided by the mobile phone function unit 410, or by using vibration from the vibrator. At this time, as described in the first configuration example, the control unit 401 determines whether the foreign object is inside the dust collection chamber 32 or outside without being sucked in, and displays the result of this determination on the thumbnail image as shown in Figure 4(A) (step S406).

[0146] Next, the control unit 401 refers to the contents of the image registration memory 407 to obtain information on the suction drive control mode stored in correspondence with the foreign object ID of the recognized foreign object, and determines whether or not it is necessary to forcibly stop the cleaning operation (step S407).

[0147] If, in step S407, it is determined that a forced stop of the cleaning operation is not necessary, the control unit 401 does not generate a forced stop control signal, allows the vacuum cleaner 30 to continue cleaning, returns to step S404, and repeats the process from step S404 onward.

[0148] Next, in step S407, if it is determined that the cleaning operation needs to be forcibly stopped, the control unit 401 generates a forced stop control signal to stop the suction drive unit 306 and the travel drive unit, and transmits it to the vacuum cleaner 30 via the wireless communication unit 402 (step S408). After that, the control unit 401 proceeds to step S409, and in step S409, waits for it to determine that the wireless communication path has been disconnected, and then terminates this processing routine.

[0149] [Effects of the second configuration example] As described above, with this second configuration example of an electric vacuum cleaner, during the cleaning operation of the self-propelled electric vacuum cleaner 30, foreign objects other than the intended cleaning target can be detected by image recognition, and a thumbnail image of the detected foreign object can be displayed on the display screen of the display unit 412 of the mobile phone terminal 40, and the detection of the foreign object can be notified by an alarm.

[0150] Therefore, the user is notified by the alarm that the self-propelled vacuum cleaner 30 has detected a foreign object, and can immediately identify what that foreign object is by viewing a thumbnail image on the display screen of the mobile phone terminal 40. In this case, the user does not need to perform any playback operations as described in the prior art document at the beginning, which is extremely convenient.

[0151] Furthermore, in this second configuration example, the display unit is not located on the self-propelled vacuum cleaner 30, but rather on the display unit 412 of a mobile phone terminal 40 located at a distance from the vacuum cleaner 30 via wireless connection. This eliminates the need to go to the self-propelled vacuum cleaner 30 while it is cleaning to view the display screen, making it convenient.

[0152] Furthermore, in this second configuration example, foreign objects that should be forcibly stopped from being suctioned can be registered in advance, allowing the suction operation to be automatically stopped. This prevents damage or malfunction of the vacuum cleaner 30, and also prevents foreign objects that should be kept clean from being sucked into the dust collection chamber 32.

[0153] Furthermore, the thumbnail images of recognized foreign objects include additional indications showing whether the foreign object has been sucked into the dust collection chamber 32 or is located outside without being sucked in, allowing the user to easily determine the location of the foreign object.

[0154] In this second configuration example, when a user clicks on the thumbnail image displayed on the display screen of the display unit 412, the actual image of the detected foreign object is displayed, along with surrounding image captures of the area in front of and behind the suction section at the cleaning location where the foreign object was image-recognized. This allows the user to accurately identify what the detected foreign object is and easily find out where it was located, which is convenient.

[0155] [Variation of the second configuration example] In the above description of the second configuration example, the cleaning of the vacuum cleaner 30 is started and stopped by operating the operation buttons provided on the operation unit 304 of the vacuum cleaner 30. However, when the cleaning support program is running on the mobile phone terminal 40, the cleaning of the vacuum cleaner 30 can also be started and stopped by instructing via the touch panel 413 of the mobile phone terminal 40.

[0156] Furthermore, although the vacuum cleaner 30 did not have an image recognition unit in the second configuration example described above, an image recognition unit may also be provided in the vacuum cleaner 30. In that case, the user can register images of foreign objects in the image registration memory of the vacuum cleaner 30, and the mobile phone terminal 40's image registration memory 407 can acquire images of foreign objects held on a website server via the internet and register and store them there. In this way, the user only needs to register images of objects they specifically want to be recognized as foreign objects in the vacuum cleaner 30, and other general foreign objects can be recognized by the mobile phone terminal 40 without the user having to register them.

[0157] Furthermore, if the vacuum cleaner 30 is equipped with an image recognition unit, the mobile phone terminal 40 may be provided without an image recognition unit, and the mobile phone terminal 40 may only use the display function.

[0158] In the description of the second configuration example above, the portable device used was a mobile phone terminal 40, but any portable device equipped with a display unit may be used, such as a pad-type portable terminal, tablet terminal, wearable device, or personal computer. Furthermore, it is not limited to portable devices, but can be applied to any device equipped with a display unit, such as desktop devices or stationary devices.

[0159] Furthermore, in the first configuration example, as in the second configuration example, a mobile phone terminal can be used as a separate display unit. On the other hand, in the second configuration example, the display unit can be provided in the vacuum cleaner 30, and a separate mobile phone terminal can not be used, resulting in a configuration similar to the first configuration example.

[0160] In the second configuration example, the suction drive control mode of the suction drive unit 306 is shown as a stop (forced stop) example. However, as with the first configuration example, various modes are possible, such as "suction turbo drive," "strong suction drive," "weak suction drive," and "weakest suction drive."

[0161] Furthermore, in the second configuration example, not only the suction drive control mode but also the travel drive control mode of the travel drive unit 307 can be considered independently or in parallel with the suction drive control, and control modes such as "high-speed travel drive," "low-speed travel drive," and "ultra-low-speed travel drive" can be used in addition to stopping (forced stop). In addition, as the travel drive control mode of the travel drive unit 307, depending on the foreign object detected by image recognition from the image captured by the camera 383, the travel drive control may be performed in a variety of modes such as "turn right while traveling," "turn left while traveling," "move forward while traveling," "move backward while traveling," and even "stop travel drive and turn right" and "stop travel drive and turn left" to avoid the foreign object that the vacuum cleaner has recognized. When the travel control is performed to avoid the foreign object, the suction drive control of the suction drive unit 306 may be performed in conjunction with it or not.

[0162] [Embodiments of this invention] In the second configuration example, since the image recognition processing is performed on a portable device connected to the vacuum cleaner 30 via a wireless communication path, it is conceivable that advanced image recognition processing may not be possible depending on the capabilities of the connected portable device. This embodiment of the invention is an example configured to improve this problem by using a server (cloud) on the internet.

[0163] Figure 10 shows an example of the electrical configuration of an electric vacuum cleaner according to an embodiment of the present invention. The electric vacuum cleaner according to this embodiment of the present invention consists of a self-propelled electric vacuum cleaner 30A and a server device 60 connected to the electric vacuum cleaner 30A via a communication network 50. In this example, the communication network 50 includes the Internet and a home wireless LAN that can connect to the Internet. The server device 60 is, for example, a server device operated by the sales company of the electric vacuum cleaner 30A that provides a cleaning support program to the electric vacuum cleaner 30A. In this embodiment of the present invention, as will be described later, it has a function to perform image recognition processing of foreign objects in the image information captured from each of the cameras 381 to 384 sent from the electric vacuum cleaner 30A.

[0164] The self-propelled vacuum cleaner 30A has the same configuration as the self-propelled vacuum cleaner 30 of the second configuration example in terms of suction cleaning function and driving function. However, the vacuum cleaner 30A of this embodiment of the invention differs from the vacuum cleaner 30 of the second configuration example in that it has a display unit for displaying foreign objects that have been recognized by image, and that it is configured to perform suction control and driving control after waiting for the image recognition result from the server device 60.

[0165] In the configuration of the vacuum cleaner 30A shown in Figure 10, the same reference numerals are used for parts identical to those of the vacuum cleaner 30 in the second configuration example, and their descriptions are omitted. Specifically, in the vacuum cleaner 30A, the drive control unit 302, the transmitted image generation unit 303, and the operation unit 304 are connected to the control unit 301A via the system bus 300, similar to the second configuration example. In addition, the display control unit 311 to which the display unit 312 is connected, the wireless communication unit 313, the non-cleaning object detection unit 314, and the current position detection unit 315 are also connected.

[0166] The display unit 312 is, for example, an LCD, and although not shown in the figure, it is provided on the upper surface of the housing of the vacuum cleaner 30A, opposite to the bottom surface 30a (see Figure 6), so that its display screen is exposed.

[0167] The wireless communication unit 313 is connected to a wireless LAN installed in a home or other location where the vacuum cleaner 30A is used, which is included in the communication network 50, and has a communication function for exchanging image data and image recognition result data with the server device 60 via the internet through the communication network 50. In this embodiment of the invention, when the vacuum cleaner 30A starts cleaning, it holds the address information of the server device 60 and other information stored in it in order to connect to the communication path with the server device 60.

[0168] The non-cleaning object detection unit 314 does not have the function of performing image recognition by comparing it with registered images, but it has the function of recognizing whether an object captured from each of the cameras 381 to 384 is a non-cleaning object, i.e., a foreign object, by determining whether its size is greater than or equal to a preset size, and whether the luminosity of the reflected light when light is shone on it is greater than or equal to a predetermined level, such as with precious metals. In other words, the non-cleaning object detection unit 314 constitutes a foreign object detection unit. In the non-cleaning object detection unit 314, the images captured from cameras 383 and 384 are configured to detect only objects that are present on the floor surface to be cleaned. This is because objects present elsewhere do not affect the suction drive and travel drive during the cleaning operation.

[0169] In this embodiment, the current location detection unit 315 includes a GPS (Global Positioning System) positioning function to detect the current location of the vacuum cleaner 30A. In this embodiment, the detected current location information is added to the image information captured by cameras 381 to 384, which is transmitted from the vacuum cleaner 30A to the server device 60, as will be described later. This current location information is used in image recognition by the server device 60 to take into account the presence of country- or region-specific items such as banknotes and personal belongings.

[0170] As mentioned above, the server device 60 is equipped with a function to perform image recognition processing for foreign objects in the image information captured from each of the cameras 381 to 384 sent from the vacuum cleaner 30A. For convenience, the configuration of the server device 60 shown in Figure 10 shows only the configuration of this foreign object image recognition processing function unit.

[0171] In other words, the server device 60 is connected to the communication network 50 and includes a communication interface 601 for communicating with the vacuum cleaner 30A, an image capture storage unit 602, an image recognition unit 603, an image registration memory 604, and a recognized foreign object information generation unit 605.

[0172] The image storage unit 602 has the same functions as the image storage unit 405 in the mobile phone terminal 40 of the second configuration example described above. It decodes and separates the image information from cameras 381 to 384 from the transmitted image information sent from the vacuum cleaner 30A received via the communication interface 601, and temporarily stores it for image recognition by the image recognition unit 603. Except for the function of decoding and separating the image information from cameras 381 to 384 from the transmitted image information, the image storage unit 602 has the same functions as the image memory 209 of the first configuration example.

[0173] The image recognition unit 603 has the same functions as the image recognition unit 406 in the mobile phone terminal 40 of the second configuration example described above. The image registration memory 604 also has the same functions as the image registration memory 211 in the first configuration example described above, but the image registration memory 604 in this embodiment of the invention registers and stores images of a wide variety of foreign objects. In addition, for foreign objects that differ by country or region, such as banknotes and coins, the image registration memory 604 also stores information such as the country in which they were issued.

[0174] The image information sent from the vacuum cleaner 30A to the server device 60 includes information about the vacuum cleaner 30A's current location. The server device 60 uses this current location information to recognize the country or region in which the vacuum cleaner 30A is cleaning, and reads out the banknotes and coins of the country identified by the current location from the image registration memory 604 for comparison in image recognition, as these differ from country or region to country or region. For example, in Japan, it uses 10,000 yen, 5,000 yen, 2,000 yen, and 1,000 yen bills, 500 yen, 100 yen, 50 yen, 10 yen, 5 yen, and 1 yen coins; in the United States, it uses dollar bills, dollar coins, and cent coins; in the EU, it uses euros; and in China, it uses yuan. Of course, it is also possible to read out banknotes and coins of all countries and all eras, including old bills and coins, for comparison in image recognition.

[0175] The recognized foreign object information generation unit 605, based on the image recognition results from the image recognition unit 603, obtains the foreign object ID, thumbnail image, and suction drive control mode information of the recognized foreign object from the image registration memory 604 when the image recognition unit 603 recognizes a foreign object, and transmits this information as information about the recognized foreign object to the vacuum cleaner 30A via the communication interface 601. In addition, the recognized foreign object information generation unit 605 also includes location information of whether the foreign object recognized by the image recognition unit 603 is inside the dust collection chamber of the vacuum cleaner 30A or outside without being sucked in, and sends this information to the vacuum cleaner 30A.

[0176] Then, when the recognized foreign object information generation unit 605 receives a request for detailed image information from the vacuum cleaner 30A, resulting from the user clicking on a thumbnail image displayed on the display unit 312, it transmits the captured image information stored in the captured image storage unit 602 to the vacuum cleaner 30A via the communication interface 601, as information corresponding to the request.

[0177] In this embodiment of the invention, the image registration memory 604 also stores premium information about the registered foreign object, such as whether it is a valuable antique, a valuable old coin, or a valuable stamp with a premium value, although this information is not shown in the illustration. The server device 60 transmits the recognized foreign object information including this premium information.

[0178] [Description of the process during cleaning operation in the embodiment of this invention] In this embodiment of the invention, the user presses the cleaning start button on the control panel 304 of the vacuum cleaner 30A. Then, the following processing operations are performed in the vacuum cleaner 30A and the server device 60.

[0179] Figures 11 and 12 are flowcharts illustrating an example of the processing flow in the vacuum cleaner 30A at this time. Figure 13 is a flowchart illustrating an example of the processing flow in the server device 60.

[0180] First, the processing operation of the vacuum cleaner 30A will be explained with reference to Figures 11 and 12. The following explanation assumes that the control unit 301A executes the functions of the drive control unit 302, the transmitted image generation unit 303, and the non-cleaning object detection unit 314 as software functions.

[0181] The control unit 301A determines whether the cleaning start button has been pressed and waits for cleaning to start (step S311). If it is determined in step S311 that the cleaning start button has been pressed, the control unit 301A starts driving the suction drive unit 306 and the travel drive unit 307, and instructs each of the cameras 381 to 384 to start capturing images. Furthermore, the wireless communication unit 313 creates a communication channel with the server device 60 (step S312).

[0182] Then, the control unit 301A collects images captured from each of the cameras 381 to 384 during the cleaning operation, generates transmission image information, and transmits it to the server device 60 via the wireless communication unit 313 along with the current location information detected by the current location detection unit 315 (step S313).

[0183] Next, the control unit 301A performs a process to detect foreign objects other than the object to be cleaned based on their size and brightness, as described above, for each of the images captured from cameras 381 to 384 (step S314). Then, the control unit 301A determines whether or not it has detected the presence of foreign objects other than the object to be cleaned (step S315). If it determines that it has detected the presence of foreign objects, it supplies a pause instruction signal to the suction drive unit 306 and the travel drive unit 307 via the drive control unit 302 to temporarily pause the cleaning operation (step S316). Then, the control unit 301A determines whether or not it has received recognition foreign object information from the server device 60 (step S317).

[0184] Even if it is determined in step S315 that no foreign object has been detected, the control unit 301A jumps to step S317 to determine whether or not it has received recognition information about the foreign object from the server device 60.

[0185] Then, in step S317, if it is determined that no recognition information for foreign objects has been received from the server device 60, the control unit 301A determines whether a certain amount of time has elapsed since the cleaning operation was temporarily suspended (step S318). If it determines that a certain amount of time has not elapsed, it returns to step S317 and repeats the process from step S317 onward.

[0186] Furthermore, if the control unit 301A determines in step S318 that a certain amount of time has elapsed since the cleaning operation was temporarily suspended, the control unit 301A supplies a drive start signal to the suction drive unit 306 and the travel drive unit 307 via the drive control unit 302 to release the paused state and resume the cleaning operation (step S319). Then, the control unit 301A returns to step S313 and repeats the process from step S313 onward.

[0187] Furthermore, in step S317, when the control unit 301A determines that it has received recognition information for a foreign object from the server device 60, it displays the received thumbnail image on the display screen of the display unit 312, along with its location information (whether inside or outside the dust collection chamber) (step S320). In this embodiment of the invention, the vacuum cleaner 30A does not provide an alarm notification when it displays a thumbnail image of a foreign object on the display screen of the display unit 312, but it is also possible to provide an alarm notification, as in the first and second configuration examples described above.

[0188] Next, the control unit 301A determines whether or not to stop the cleaning operation based on whether or not the recognized foreign object information includes a forced stop control signal (step S321 in Figure 12). If it is determined in step S321 that the forced stop control signal is not included, the control unit 301A determines whether or not it has detected the end of cleaning based on whether or not it has completed cleaning all of the cleaning range (step S322). If it has not detected the end of cleaning, it returns to step S313, generates the transmission image information, and continues the cleaning operation while wirelessly transmitting it to the server device 60.

[0189] Then, in step S321, if it is determined that a forced stop control signal is included, the control unit 301A stops the operation of the suction drive unit 306 and the travel drive unit 307, and instructs each of the cameras 381 to 384 to stop capturing images. Furthermore, it disconnects the communication path with the server device 60 (step S323).

[0190] Even when the end of cleaning is detected in step S322, the control unit 301A proceeds to step S323, stopping the operation of the suction drive unit 306 and the travel drive unit 307, instructing each of the cameras 381 to 384 to stop capturing images, and further disconnecting the communication path with the server device 60.

[0191] This concludes the process for using the 30A vacuum cleaner during cleaning.

[0192] Next, the processing operation of the server device 60 will be explained with reference to Figure 13. First, the server device 60 determines whether or not it has received a request to create a communication channel from the vacuum cleaner 30A (step S601). If it determines that it has received a request to create a communication channel, it creates a communication channel with the vacuum cleaner 30A through the communication network 50 (step S602).

[0193] Next, the server device 60 receives transmitted image information sent from the vacuum cleaner 30A through the generated wireless communication channel, separates and acquires the captured image information from each of the cameras 381 to 384, and performs image recognition processing by comparing the captured images from each of the cameras 381 to 384 with the images of foreign objects stored in the image registration memory 407 using pattern matching, as in the first configuration example (step S603).

[0194] Then, the server device 60 determines whether or not it has detected a foreign object based on the image recognition results in step S603 (step S604). If it determines in step S604 that no foreign object has been detected, the server device 60 determines whether or not the communication path has been cut by the vacuum cleaner 30A (step S608). If it determines in step S608 that the communication path has been cut, the server device 60 determines that the cleaning operation of the vacuum cleaner 30A has stopped and terminates this processing routine.

[0195] Furthermore, if it is determined in step S608 that the communication channel has not been disconnected, the server device 60 determines that the cleaning operation of the vacuum cleaner 30A is continuing, returns to step S603, and repeats the processing from step S603 onward.

[0196] Then, in step S604, if the image recognition determines that a foreign object has been recognized, the system generates recognized foreign object information including a thumbnail image of the recognized foreign object, a foreign object ID, location information, and premium information, and sends it to the vacuum cleaner 30A (step S605).

[0197] Furthermore, the server device 60 refers to the contents of the image registration memory 604 to obtain information on the suction drive control mode stored in correspondence with the foreign object ID of the recognized foreign object, and determines whether or not it is necessary to forcibly stop the cleaning operation (step S606).

[0198] If, in step S606, it is determined that a forced stop of the cleaning operation is not necessary, the server device 60 continues the cleaning operation of the vacuum cleaner 30A without generating a forced stop control signal, returns to step S603, and repeats the process from step S603 onward.

[0199] Next, in step S606, if it is determined that the cleaning operation needs to be forcibly stopped, the server device 60 generates a forced stop control signal to stop the suction drive unit 306 and the travel drive unit 307 and transmits it to the vacuum cleaner 30A (step S607). After that, the server device 60 proceeds to step S608, and in step S608, waits for it to determine that the communication path has been disconnected, and then terminates this processing routine.

[0200] In this embodiment of the invention, when a user clicks on a thumbnail image displayed on the display screen of the display unit 312, the vacuum cleaner 30A sends a request to the server device 60 to acquire detailed image information corresponding to the thumbnail image, along with the foreign object ID of the thumbnail image. In response to this acquisition request, the server device 60 reads the actual image information and surrounding image information of the foreign object identified by the foreign object ID from the image storage unit 602 and transmits the image information to the vacuum cleaner 30A. The vacuum cleaner 30A displays the received image information on the display screen of the display unit 312, and also displays the surrounding image images in front of and behind the suction part at the cleaning location where the foreign object was image-recognized. Therefore, the user can accurately recognize what the detected foreign object is and easily find out where it was located, which is convenient. Furthermore, if the recognized foreign object information received in step S605 includes premium information, the vacuum cleaner 30A can change the background color of the thumbnail image of the foreign object, flash the display screen, or sound an alarm. This helps users avoid overlooking valuable or expensive foreign objects.

[0201] [Effects of the embodiments of this invention] As described above, in the electric vacuum cleaner according to this embodiment of the invention, when the self-propelled electric vacuum cleaner 30A recognizes the presence of a foreign object, it temporarily stops the cleaning operation and waits for the arrival of recognized foreign object information from the server device 60. If the electric vacuum cleaner 30A does not receive recognized foreign object information from the server device 60 even after waiting for a certain period of time, the server device 60 determines that it has not recognized a foreign object and that there is no problem in continuing the cleaning operation, and continues the cleaning operation. When the server device 60 recognizes a foreign object, the recognized foreign object information is sent within a certain period of time, so the electric vacuum cleaner 30A waits for its reception, and if the recognized foreign object information includes a forced stop control signal, it can stop the cleaning operation in accordance with the forced stop control signal.

[0202] As described above, the self-propelled vacuum cleaner 30A temporarily suspends its cleaning operation to account for the delay in image recognition processing in the server device 60, so that it can perform the intended control regardless of the delay in image recognition processing in the server device 60. Therefore, it is possible to prevent malfunctions caused by the self-propelled vacuum cleaner 30A sucking up foreign objects that it should not be sucking up.

[0203] Furthermore, after pausing, the vacuum cleaner 30A may, based on its own judgment or instructions from the server device 60, zoom in on the foreign object or move to photograph the foreign object from a different angle. Alternatively, the vacuum cleaner 30A may be configured to automatically zoom in on the foreign object after pausing. By transmitting the image captured through this zoom imaging back to the server device 60 from the vacuum cleaner 30A, the server device 60 can perform image recognition with higher accuracy.

[0204] In addition to the effects described above, it goes without saying that the same effects as those in the first and second configuration examples described above can be obtained in this embodiment of the invention.

[0205] [Modified embodiments of this invention] In the above-described embodiment, a current location detection unit 315 equipped with a GPS positioning function is provided. However, the main point is that the country or region in which the vacuum cleaner 30A is used can be transmitted to the server device 60. For example, the user may be asked to input the address of the place where the vacuum cleaner 30A will be used, and this input address information may be transmitted to the server device 60.

[0206] Furthermore, although the vacuum cleaner 30A did not have an image recognition unit in the above embodiment, an image recognition unit may also be provided in the vacuum cleaner 30A. In that case, the user registers images of foreign objects in the image registration memory of the vacuum cleaner 30A, and the image recognition unit of the vacuum cleaner 30A recognizes the foreign objects registered in this image registration memory, while the server device 60 recognizes foreign objects that are not registered in the image registration memory of the vacuum cleaner 30A. Note that the server device is not limited to one unit, and multiple server devices can be used in conjunction with each other.

[0207] Furthermore, in this embodiment of the invention, the display unit 312 may be provided on a portable device such as a mobile phone terminal that is wirelessly connected to the vacuum cleaner 30A, or on a tablet terminal, wearable terminal, or personal computer. It may also be provided on a device equipped with various display units, such as a desktop device or a stationary device.

[0208] [Other embodiments or modifications] While the above explanation primarily focuses on household vacuum cleaners, it goes without saying that it can also be applied to commercial vacuum cleaners used in factories, shops, restaurants, and other commercial establishments. Furthermore, it can be applied to various types of vacuum cleaners, including tabletop vacuums designed for use on desks and cleaners specifically for mattresses.

[0209] In the embodiments and configuration examples described above, camera shooting is performed in continuous shooting mode (video recording), but intermittent shooting of still images may also be used. Furthermore, a combination of both methods may be used.

[0210] Furthermore, the camera shooting method and resolution do not have to be the same and may vary depending on the mode and situation. For example, a lost item mode (search mode) can be set up, and when a foreign object other than the object being cleaned (potential lost item, potential lost item) is found, either continuously from the start of camera shooting or after camera shooting has started, the camera may perform different shooting actions than normal shooting, such as taking a flash photograph, taking a zoom photograph, taking a photograph from multiple angles, increasing the resolution, or using multiple cameras, making it easier to find the lost item (search item). In addition, the image recognition processing method and accuracy of the captured images may be changed depending on the mode and situation.

[0211] For specific foreign objects designated to be searched in lost-item mode from images registered in the image registration memory, multi-stage image recognition may be performed. For example, initially, to minimize the probability of overlooking an object, a low-resolution image may be used to broaden the range of similarity during the search. Then, for the found object, high-resolution images may be used to perform image recognition from multiple angles in order to narrow down the search. Special processing for image recognition may also be added. For example, various feature extractions and filters such as contour (edge) extraction and enhancement, brightness extraction and enhancement, and hue extraction and enhancement may be added, or various enhancement processes may be applied and used for image recognition. Furthermore, image recognition may be performed on high-resolution images without compression, and the image recognition processing time may be extended beyond the normal time, for example, several times longer.

[0212] If the lost item is a ring, for example, a platinum ring with a ruby, the system's ability to selectively identify color and luster (light reflection) can be enhanced to detect the ruby's red color and the platinum ring's silvery (lustrous grayish-white) color. Furthermore, the system's ability to selectively identify shape can be enhanced to improve detection accuracy based on the shape of the ruby ​​stone and the ring's size, as defined by its inner diameter.

[0213] Furthermore, lost items may be products with standardized sizes and dimensions. For example, an SD card is 24mm wide, 32mm long, and 2.1mm thick, while a microSD card is 11mm wide, 15mm long, and 1.0mm thick; their shapes are precisely defined by the SD standard. Therefore, if the lost item is an SD card or microSD card, one can first search for a rectangular object based on its shape, then filter based on whether the aspect ratio (length to width ratio) matches, and finally perform precise image recognition. Of course, using a rangefinder allows for the measurement of absolute values ​​such as the inner diameter and length of the object, which can then be used for identification. Even without a rangefinder, if there is a reference object for length and size, the absolute values ​​of length and size can be determined by relative comparison. In addition, the logo or text on the surface of the SD card or microSD card can be used for identification through character recognition.

[0214] Furthermore, if the lost item is a transparent contact lens, when photographed, the background color will show through, so detection accuracy can be improved by precisely detecting differences in brightness and color, or by detecting reflected light that sparkles when a flash is shone on it, and then identifying the shape of the thin, aspherical lens that is unique to contact lenses. Typically, hard contact lenses have a diameter of around 9 mm, and soft contact lenses have a diameter of around 13 mm, so specifying whether the lost contact lens is hard or soft will improve the ability to identify it by size and make it easier to find.

[0215] Furthermore, even if the image of a foreign object has a low similarity to a lost item and is unlikely to be a lost item, the similarity threshold can be made variable to detect the foreign object as a lost item, thereby reducing the risk of missing the lost item.

[0216] Needless to say, various methods such as template matching, cluster analysis, and neural networks can be applied to image recognition and similarity level calculation.

[0217] Furthermore, in lost item mode, the suction drive control can be set to "weak suction drive" or "weakest suction drive" until the lost item is found. It can also be set to "stop suction drive". In the case of a self-propelled vacuum cleaner, not only the suction drive control but also the driving drive control can be set to "low driving speed" or "very low driving speed" until the lost item is found. It can also be set to "stop driving drive (pause)" at predetermined intervals or after traveling a predetermined distance. In this case, while paused, the surroundings can be carefully photographed not only with normal shooting but also with flash shooting and zoom shooting. Furthermore, panoramic shooting or 360-degree shooting can be performed while rotating to search for the lost item.

[0218] Furthermore, in lost item mode, images of foreign objects to be searched, such as rings, SD cards, microSD cards, and contact lenses, can be read from the image registration memory and displayed on the screen for selection.

[0219] Furthermore, in the above embodiments and configuration examples, the recognition (detection) of foreign objects during cleaning is performed by image recognition of image information captured from the camera. However, other sensors such as infrared sensors, microphones, ultrasonic sensors, tactile sensors, biosensors, odor sensors, and metal detection sensors may also be used to recognize (detect) foreign objects. Of course, multiple types of these sensors may be used in combination.

[0220] If a microphone is used as a sensor, for example, the microphone can collect sound or voice, and if an object emitting sound or voice is detected, it can be recognized as a foreign object other than the object being cleaned. In this case, if "foreign object detection based on recognition results of images captured by a camera" in the flowcharts of Figures 5, 8, 9, 11, 12, or 13 is replaced with "foreign object detection based on recognition results of sound or voice collected by a microphone," a vacuum cleaner capable of foreign object detection by voice recognition can be realized. Possible forms of foreign object detection by voice recognition include collecting sound near the suction part with a sound collection unit during cleaning operation to detect foreign objects before they are sucked into the vacuum cleaner, or collecting abnormal sounds emitted after foreign objects such as metal or bags have been sucked into the vacuum cleaner to detect them. When foreign object detection is performed by voice recognition, similar to foreign object detection by image recognition, forms of suction drive control such as stopping the suction drive can be considered. Furthermore, forms of alarm notification can be performed by display control, voice control, vibration control, etc. Furthermore, the noise emitted by the vacuum cleaner itself can be removed through sound source separation processing.

[0221] Furthermore, infrared cameras or video cameras may be used as cameras 171-174 and 381-384. They may also be used in combination. While the above embodiment describes an example with four cameras, the number of cameras is not limited. Moreover, the camera installation locations are not limited to the above-described embodiment and configuration example; various configurations are possible.

[0222] In the embodiments and configuration examples described above, stationary objects were used as examples of objects to be cleaned and foreign objects other than objects to be cleaned. However, it is of course possible to use moving objects such as insects or other living things, or motor-driven toys.

[0223] Although preferred embodiments of this invention have been described in detail above with reference to the drawings, this invention is not limited to such examples. It is clear to any person with ordinary skill in the art of this invention that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these are also understood to fall within the technical scope of this invention. [Explanation of symbols]

[0224] 1: Electric vacuum cleaner 10: Vacuum cleaner body 11: Suction hose 12: Handle 13: Fittings 14: Suction section 141: Inlet 15:Operation unit 16: Display section 16D:Display screen 101: Dust collection room 102: Suction drive unit 171~174: Camera 20: Control circuit section 201: Control Unit 202: Drive Control Unit 210: Image Recognition Unit 211: Image registration memory 213: Display Image Generation Unit 214: Alarm generation unit 30: Self-propelled electric vacuum cleaner 31: Suction part 32: Dust collection room 381~384: Camera 406: Image Recognition Unit 407: Image registration memory 409: Display Image Generation Unit 412:Display section

Claims

1. An electric vacuum cleaner comprising: a suction unit having a suction port for sucking up objects to be cleaned; a suction drive unit for sucking up and taking in the objects to be cleaned from the suction port; a drive unit for moving the unit itself; and a separate device connected to the electric vacuum cleaner via a communication network, An image recognition unit that recognizes foreign objects other than the object to be cleaned, When the image recognition unit recognizes the foreign object, a display image generation unit generates a display image to inform the user what the recognized foreign object is. A display control unit that displays the display image generated by the display image generation unit on the display screen of a display unit provided in the separate device, Equipped with, Each time the image recognition unit recognizes a new foreign object, the display image generation unit generates a new display image to inform the user of what the recognized foreign object is. Each time a new display image is generated, the new display image is added to the display screen of the display unit. An electric cleaning device characterized by the following features.

2. A determination unit determines whether the foreign object recognized by the image recognition unit is located inside the vacuum cleaner after being sucked in through the suction port, or outside the vacuum cleaner. The display image generation unit generates the display image, including additional information indicating whether the foreign object is located inside or outside the vacuum cleaner, according to the determination result from the determination unit. The electric cleaning device according to feature 1.

3. A storage unit that stores an image of the foreign object and the name of the foreign object in association with each other, An imaging unit for imaging the object to be cleaned and foreign objects other than the object to be cleaned, Equipped with, The image recognition unit performs image recognition by comparing the captured image captured by the imaging unit with the image of the foreign object stored in the storage unit, and recognizes the registered foreign object stored in the storage unit. The display image generation unit retrieves the name of the foreign object recognized in the image from the storage unit based on the recognition result of the image recognition unit, and generates the display image including the name of the foreign object. The electric cleaning device according to claim 1 or 2.

4. When the image recognition unit recognizes the foreign object, the control unit determines whether to perform suction of the recognized foreign object by the suction drive unit or to restrict suction, including stopping suction, and performs suction drive control on the suction drive unit according to the result of the determination, The display image generation unit generates a display image that includes the text "Suction stopped" for foreign objects whose suction has been stopped by the control unit. The electric cleaning device according to any one of claims 1 to 3.

5. The display image generation unit generates the display image which includes a thumbnail image of the foreign object recognized by the image recognition unit. The electric cleaning device according to any one of claims 1 to 4.

6. The aforementioned vacuum cleaner is equipped with one or more imaging units, The aforementioned vacuum cleaner or the aforementioned separate device is An image storage unit that stores the image of the foreign object captured by the imaging unit, An instruction input receiving unit that receives instruction input for the thumbnail image of the foreign object displayed on the aforementioned display screen, Equipped with, When the instruction input receiving unit receives the instruction input, it retrieves the image of the foreign object corresponding to the thumbnail image received in the instruction input from the image storage unit and displays it on the display screen. The electric cleaning device according to feature 5.

7. The aforementioned vacuum cleaner is equipped with a peripheral image capturing unit that captures images of the area around the unit, The aforementioned vacuum cleaner or the aforementioned separate device is The system includes means for storing a peripheral image captured by the peripheral image capturing unit in response to the foreign object detected by the foreign object detection unit, The stored surrounding image is displayed on the display screen along with the captured image of the foreign object corresponding to the thumbnail image of the foreign object that was instructed to be displayed. The electric cleaning device according to feature 6.

8. The system includes an imaging unit for imaging the object to be cleaned and foreign objects other than those to be cleaned. The display image generation unit generates the display image which includes the captured image of the foreign object captured by the imaging unit or a reduced image of the captured image of the foreign object. The electric cleaning device according to any one of claims 1 to 7.

9. A storage unit for storing an image of the foreign object, An imaging unit for imaging the object to be cleaned and foreign objects other than the object to be cleaned, Equipped with, The image recognition unit performs image recognition by comparing the captured image captured by the imaging unit with the image of the foreign object stored in the storage unit, and recognizes the registered foreign object stored in the storage unit. The display image generation unit generates the display image by obtaining the image of the foreign object recognized by the image recognition unit from the storage unit based on the recognition result of the image recognition unit. The electric cleaning device according to any one of claims 1 to 8.

10. The image of the foreign object is obtained from a website and stored in the storage unit. The electric cleaning device according to feature 9.

11. The aforementioned separate device is a portable device including a tablet terminal, a mobile phone terminal, and a wearable terminal. The electric cleaning device according to any one of claims 1 to 10.

12. The aforementioned vacuum cleaner is equipped with one or more imaging units, and transmits the images captured by the imaging units to the separate device via a first wireless communication unit. The separate device receives the captured image transmitted from the vacuum cleaner via the second wireless communication unit, and the image recognition unit recognizes the foreign object based on the received captured image. The electric cleaning device according to any one of claims 1 to 11.

13. The aforementioned vacuum cleaner includes the image recognition unit. The electric cleaning device according to any one of claims 1 to 12.

14. Each time the image recognition unit recognizes a foreign object, the display image is shown on the display screen, and an alarm indicating that a foreign object has been recognized is output from the vacuum cleaner and / or the separate device. The electric cleaning device according to any one of claims 1 to 13.

15. The aforementioned foreign object includes string-like objects such as cords and cables. The electric cleaning device according to any one of claims 1 to 14.

16. The system includes a voice recognition unit that recognizes the foreign object based on the voice recognition results of sounds and voices collected by a microphone, The microphone picks up sound near the suction section, and the voice recognition unit recognizes the foreign object by voice before it is sucked into the vacuum cleaner. The electric cleaning device according to any one of claims 1 to 15.

17. The system includes a voice recognition unit that recognizes the foreign object based on the voice recognition results of sounds and voices collected by a microphone, The microphone collects the abnormal sound emitted after the foreign object is sucked into the vacuum cleaner, and the voice recognition unit recognizes the foreign object sucked into the vacuum cleaner by voice. The electric cleaning device according to any one of claims 1 to 16.

18. If the voice recognition unit recognizes the foreign object by voice, the suction by the suction drive unit will stop. The electric cleaning device according to claim 16 or 17.

19. The noise emitted by the vacuum cleaner itself is removed through sound source separation processing. The electric cleaning device according to any one of claims 16 to 18.

20. A vacuum cleaner comprising a suction unit having a suction port for sucking up objects to be cleaned, a suction drive unit for sucking up and taking in the objects to be cleaned from the suction port, and a drive unit for moving the unit itself, and a separate device connected to the vacuum cleaner via a communication network, wherein the computer of the vacuum cleaner and / or the separate device is provided for An image recognition unit that recognizes foreign objects other than the object to be cleaned, When the image recognition unit recognizes the foreign object, a display image generation unit generates a display image to inform the user what the recognized foreign object is. A display control unit that displays the display image generated by the display image generation unit on the display screen of a display unit provided in the separate device. A program for an electric cleaning device to function as such, Each time the image recognition unit recognizes a new foreign object, the display image generation unit generates a new display image to inform the user of what the recognized foreign object is. Each time a new display image is generated, the new display image is added to the display screen of the display unit. A program for an electric cleaning device characterized by the following features.

Images