A remote control system that can operate various electrical appliances, a remote control and robot equipped with the system, and an application for setting up the system

Abstract

Conventional remote controls require separate remote controls for different electrical appliances, such as a television remote control, a lighting remote control, and an air conditioner remote control. However, we provide a remote control that can automatically identify and operate various electrical appliances with one remote control, a remote control system for the remote control, and an application that can customize the system. [Solution] For example, when the power ON button (6) on a remote control (1) is operated, the camera (2) on the remote control takes a picture of the electrical appliance the remote control (1) is pointed at, and the electrical appliance is instantly identified using machine learning learning data, and a wireless communication signal to turn the power ON for that electrical appliance is transmitted from the remote control. Depending on what the user points the remote control (1) at, various electrical appliances can be turned ON with the same button (6). This remote control system can also be installed on smartphones and robots that help with housework and work.

Description

[Technical Field]

[0001] The present invention relates to a system that can identify various electrical appliances that can be operated via wireless communication using machine learning and operate each electrical appliance, a remote control and robot equipped with the system, and an application for setting the system. [Background technology]

[0002] Conventional known remote controls require multiple remote controls for each electrical appliance, such as a television remote control for a television, a lighting remote control for a light, and an air conditioner remote control for an air conditioner.

[0003] In recent years, it has become possible to control multiple home appliances from a single device, such as a smartphone or a voice recognition engine like Amazon Alexa. However, with a voice recognition engine, you have to speak to activate it, and there are times when you don't want to speak or make much noise, and when operating from a smartphone, you have to switch screens to operate the control app, which can be inconvenient, so it cannot be said that the demand for convenience like a remote control has completely disappeared.

[0004] While there is another remote control system equipped with machine learning that checks whether an electrical appliance is operating normally, such as in Patent Document 1 (see Solution, Claim 1, and paragraph 0011), the present invention is equipped with machine learning that distinguishes the electrical appliance or the object to be operated, and the application and program are different. The present invention is an application and program in which machine learning distinguishes "this" or "that" that a user points to, such as "turn that on" or "turn that off," from sensor information such as a camera module, and changes the command depending on the identified object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-170744 Summary of the Invention [Problem to be solved by the invention]

[0006] To provide a remote control that can automatically identify and operate various electrical appliances with one remote control, eliminating the need for separate remote controls for each electrical appliance, such as a television remote control, a lighting remote control, and an air conditioner remote control, and to provide a remote control system and an application that can customize the system. [Means for solving the problem]

[0007] The remote control system has a camera that captures the object being operated, and when the remote control is operated, the camera captures the object, and the captured image is used as learning data through machine learning to instantly identify the electrical appliance in question, and the remote control transmits a wireless communication signal equipped with that electrical appliance from the remote control.

[0008] We also provide an application that can be launched on a PC or other device, which can take pictures of the electrical appliances the user owns using the remote control camera, and can perform additional learning using the images taken, as well as customize the association between the registration of electrical appliances and the button operations on the remote control. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram of a remote control comprising the remote control system of the present invention; [Figure 2] 1 is a schematic diagram of a computer frame that supports the mouse operation mode of the remote control of the present invention; [Figure 3] 4 is a flowchart of each mode of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] In the present invention, a remote control system refers to a method or mechanism, and a remote control refers to a device operated by human hands.

[0011] A device such as a remote control (1) has one or more camera modules (hereinafter referred to as cameras) (2) for photographing the object to be operated, and when the remote control (1) is operated, the camera (2) first takes a photo of the object, stores the captured image in the remote control's internal memory, and then uses pre-machine-trained learning data from the stored image stored in a processing device such as a microcomputer, CPU, or logic chip also inside the remote control to determine what the image is. A command corresponding to the identified object is then sent via wireless communication. There are remote control systems that require three steps: 1. data acquisition by a camera module, 2. image analysis and discrimination using machine learning, and 3. wireless transmission of a command corresponding to the discrimination.

[0012] With this remote control system, even if the user presses the power ON (6) button on the same remote control (1), the remote control will determine the next command depending on whether the remote control is being pointed at a TV, a light, or an air conditioner, and will automatically send the correct command to the target.

[0013] This is due to a program built into the processing unit of the remote control system, and the programming of this program is object-oriented. The outline of the programming is that the target electrical appliances etc. are held as instances of the operation target class, and the operation target class has methods for when the power ON button is pressed, the OFF button is pressed, the up button is pressed, and the down button is pressed, and is a class that has captured images, a list of operations for the electrical appliance, standard actions, etc. as member variables. This makes it possible to change the behavior of the buttons polymorphically, as is typical of object-oriented programming, and instances of the operation target class can be freely registered, edited, and deleted later by the application.

[0014] For example, when you are sleeping and want to operate an air conditioner using a remote control (1) placed next to your pillow even when the room is dark, it is better to provide an infrared light (3) and an infrared camera (2) so that the object to be operated can be recognized, and when taking a photograph to identify the object, the infrared light (3) can be used to flash and the infrared camera (2) can be used to take a photograph so that the object can be identified by the human eye even in the dark.

[0015] Since most of the electrical appliances with wireless communication use infrared communication, it is better for the remote control to have an infrared transmitter (4) for sending commands to the electrical appliance and an infrared receiver (4) for learning the infrared pattern from the original remote control of the electrical appliance.

[0016] It is also a good idea for the remote control to have wireless communication such as Wi-Fi for controlling smart home appliances, as well as Bluetooth, which is used for other wireless communication.

[0017] Furthermore, with smart home appliances and other appliances that do not use infrared communication, for example, if the same type of lighting is used in different rooms, it will be impossible to determine which room's lighting is in by analyzing only the learning data from the image data of the photo taken by the remote control (1).To solve this problem, it would be even better to equip the remote control with sensors such as a GPS sensor, gyro sensor, direction sensor, and acceleration sensor, and supplement machine learning with this sensor data to make it possible to distinguish between the lighting in each room even if the lighting is the same shape.

[0018] It has an application that can be launched on a PC or smartphone to register new information on the remote control (1), learn additional images captured by the remote control (1), and assign commands to the buttons (6) on the remote control, thereby making it possible to change the settings of the remote control (1) and the programs and class instances of the processing unit in the remote control.

[0019] It would be even better if this application had preset learning data that could identify major products from major electronics manufacturers, etc. Furthermore, it would be good if the presets were constantly updated and added with application updates, so that the application could constantly evolve and encourage users to sign up for subscriptions, etc.

[0020] The remote control (1) can be linked to the application via Wi-Fi, Bluetooth, or physical cables such as USB, making it easy for users to operate.

[0021] When using the remote control (1), the object pointed at the camera (2) does not have to be an electrical appliance. For example, if you point it at a bed, it can switch on the lights for sleeping, or if you point it at the front door before leaving, it can turn off the air conditioner and lights all at once. This allows for flexible customization by combining what objects to learn and what commands to issue depending on what it identifies. Even an illustration of a bed can be registered, and if the system is equipped with a direction sensor and machine learning is performed to increase the influence of the sensor, it can register, for example, the lighting to be white like a morning sun in an east-facing ceiling corner and the lighting to be warm like a sunset in a west-facing ceiling corner.

[0022] It is expected that smart home appliances and those with wireless communication capabilities will continue to increase in the future, allowing us to open car doors, lock the front door, and close the curtains. This remote control has the potential to be pointed at any object, like a magic wand. Furthermore, not only can we turn the power on and off, but we can also use the same up and down buttons (7) to control things like air conditioning temperature, TV volume, curtain opening, and lighting brightness, all with polymorphic programming.

[0023] By constantly recording with the camera (2) on the remote control (1), it is possible to capture images of a computer monitor, etc., and operate it like a computer mouse.

[0024] The area of ​​the captured image that is the monitor (12) and the coordinates of the center of the camera within that monitor area (13) are transmitted to a computer via Bluetooth or other means, and mouse operation can be performed by moving the pointer. It is also possible to send the image to a computer and have the coordinates calculated on the computer side. At this time, it is even better if an application that lights up the monitor frame (14) or similar is running on the computer to provide hints as to which area of ​​the image is the monitor, making it easier to see.

[0025] For applications that show this monitor frame, it is better to use red dotted lines (14E) on the left and right sides of the monitor, green dotted lines (14C) at the top, and blue dotted lines (14D) at the bottom, for example, as this makes image recognition easier.

[0026] This frame must be displayed in front of any applications, but it does not have to be displayed all the time, and even if it is not displayed, it can be hidden if the monitor can be identified by image recognition. If the image is not recognized even when a hint frame is displayed on the monitor (14A), it may be that the remote control is too close to the monitor, so in that case, you can try displaying a hint frame in the center of the screen (14B) and scanning it to enable image recognition.

[0027] The hint frame (14) uses lens distortion, perspective, dotted line visibility, color recognition, and 3D camera tracking technology used in CG video production to calculate not only where the remote control is pointing on the monitor, but also its relative position and angle from the monitor. This means that it can be used not only for mouse operations, but also as a device for moving your hand, such as a game controller.

[0028] Since it is difficult to perform precise mouse operations by simply changing the direction of the remote control (1), it would be better if this could be supplemented by a joystick (7), a cross key, a trackball, etc.

[0029] It would also be better if the mouse click button (8) was located in a position that was easy to click with the index finger or middle finger when holding the remote control (1).

[0030] It is better for the camera (2) of the remote control (1) used to operate electrical appliances to be directional, but it is better for the camera (2) used to operate the mouse to have a wide angle and a wide range, like a fisheye lens. This can be done by using multiple cameras (2) with different lenses, or by using a wide-angle lens to operate electrical appliances, and the image can be cropped.

[0031] In the appliance operation mode (A), a photo of the target is taken after pressing a button on the remote control (1), but in the mouse mode (D), the camera (1) must constantly take pictures and calculate coordinates to operate the pointer, so the standby state behavior is different. Therefore, a toggle switch (9) is attached to switch between the appliance operation mode (A) and the mouse operation mode (D).

[0032] It is also better to be able to switch between the operation mode (A), operation target registration mode (B), and infrared learning mode (C) in the electrical appliance operation mode (A).

[0033] It would be even better if the remote control (1) had a microphone (11) so that the user could input voice commands. This would allow commands to be issued to electrical appliances by voice input, but it could also be used to register and learn the objects to be controlled. For example, after voice input of "television" (B04), if you take several photos of the television for machine learning (B07), it could automatically create a folder called "television" in memory (in the programming process, it creates an instance of the "television" class for the object to be controlled) and store the captured images in it (B08).

[0034] The remote control (1) should have a light such as an LED (5) to indicate to the user what operation the remote control is performing, whether the battery is sufficient, etc. It would be better if this indication to the user was a simple display rather than an LED (5).

[0035] The remote control (1) has a laser pointer (10) that is as gentle on the eyes as possible, which helps the user to determine where the remote control (1) is pointing.

[0036] The remote control system of the present invention, which has a program that photographs and identifies the object to be operated and transmits the corresponding command, and the remote controller that issues commands to the system with simplified buttons (6, 7), may be remotely controlled. For example, a drone may be used to patrol a building, and a security guard at a distance may monitor the video captured by the drone. If an electrical appliance is left on, the security guard can turn it off with the remote controller in hand, thereby turning off the electrical appliance in front of the drone.

[0037] Furthermore, even if the robot is fully autonomously operated by AI or program control, the remote control system of the present invention can allow a patrol robot to take a photo of an electrical appliance using a camera module, as described above, identify the electrical appliance using machine learning learning data, and control it using wireless communication appropriate for that electrical appliance.

[0038] In addition, as a future application of the remote control system of this invention, a commanding robot could take pictures of work robots with a camera module, identify the work robot using learning data from machine learning, and control the work robot via wireless communication appropriate to the robot. In this case, wireless communication would be primarily radio wave-based, such as Wi-Fi or Bluetooth, but technologies such as sonic communication or infrared communication could also be used to send commands only to robots within a certain range, or only to robots facing each other. [Example]

[0039] The remote control equipped with the above remote control system is placed in the living room and used by the whole family. [Example]

[0040] A small remote controller equipped with the remote control system is carried like a key holder and can be used anywhere, including at home, in a car, or at work. [Example]

[0041] The above remote control system is provided when remotely operating radio-controlled vehicles or drones for building security, etc. [Example]

[0042] The above remote control system is provided when remotely operating a nursing care robot or a robot that supports daily life or work. [Industrial Applicability]

[0043] The remote control system of the present invention is sold. The remote control of the present invention is sold. The remote control system and application of the present invention are maintained and subscriptions are sold. [Explanation of symbols]

[0044] 1 Remote Control 2 Camera Module 3. Infrared light 4 Infrared transmitter / receiver 5 LED lights that display signs to users 6 Remote control operation buttons 7 Joystick for fine-tuning the up / down / left / right inputs on the remote control and the pointer for mouse functions 8 Click button for mouse operation mode 9 Mode toggle switch 10. Laser Pointer 11 Audio input microphone 12 Computer monitor 13 Computer screen 14 Hint Frame 14A Hint box for scanning the entire screen 14B Hint box for partial scanning when full-screen scanning fails 14C Hint frame for the top of the screen, with different colors such as green to let you know it is the top 14D Hint frame for the bottom of the screen, with different colors such as blue to help users recognize that it is the bottom 14E Hint frames for the left and right sides of the screen, with different colors such as red to help users identify the left and right sides

Claims

1. A remote control system is provided which includes one or more camera modules, means for discriminating images captured by the camera modules using machine learning data learned from images used as training data for determining commands, and for determining commands based on the discriminated images, and a wireless communication device which can send the commands to electrical appliances and execute pre-set routine actions.The remote control system is capable of executing a control flow in which, each time a user performs an operation such as "power on," the target at which the camera module is pointing is automatically photographed immediately after the operation, the captured image is analyzed by the machine learning means to determine the command, and the command is sent from the wireless communication means to the target.

2. A remote control system according to claim 1, comprising a memory for storing captured images, programs, and learning data, and a CPU or microcomputer for analyzing images from the learning data, thereby enabling the machine learning of claim 1 to be performed and additional learning of images for determining commands.

3. A remote control system according to claim 2, characterized by comprising an infrared communication transmitter, an infrared receiver for learning infrared patterns from the original remote control of the electrical appliance, and a transmitter such as Wi-Fi or Bluetooth that is generally used for wireless communication to send commands to the electrical appliance.

4. An application that can be run on a PC or smartphone and can perform additional learning of image data captured by the remote control system described in claim 1, add image teacher data for determining commands, update learning data, set commands and standard actions to be sent to the object to be operated, and set associations with image data.

5. The remote control system of claim 1, having the function of communicating with the application of claim 4 via Wi-Fi, Bluetooth, physical cables such as USB, etc., and sharing settings, image data, and learning data.

6. A remote control system characterized in that one or more of the one or more cameras of the remote control system described in claim 1 is an infrared camera, and is equipped with an infrared light for illuminating an object in the dark, so that the infrared light illuminates the object even in the dark, and the system can identify electrical appliances from images taken by the infrared camera and send commands according to the object.

7. The remote control system according to claim 1 is characterized in that it is equipped with sensors such as a GPS sensor, a gyro sensor, and a direction sensor so that even if the products are the same, they can be distinguished from one another, and the sensor information is used to assist in image analysis and learning data.

8. A remote controller comprising the remote control system according to any one of claims 1 to 3 or 5 to 7.

9. A remote control system comprising the remote control system according to any one of claims 1 to 3 or any one of claims 5 to 7, wherein user operation of the remote control system is a command operation by the robot, and a robot comprising the remote control system.

10. A smartphone equipped with the remote control system according to any one of claims 1 to 3 or any one of claims 5 to 7.

11. The remote controller according to claim 8, further comprising one or more buttons for operating an object.

12. 9. The remote control according to claim 8, further comprising a microphone for voice input.

13. 9. The remote control of claim 8, further comprising a laser pointer for making it easier to see what the remote control is pointing at.

Images