A remote control system capable of operating various electrical appliances, a remote control equipped with the system, a robot, and an application for configuring the system.

Abstract

Traditional remote controls required separate remotes for different electrical appliances such as TVs, lights, and air conditioners. This product provides a remote control that can automatically identify and operate various electrical appliances with a single remote, along with a remote control system and an application to customize that system. [Solution] For example, when the power ON button (6) of the remote control (1) is operated, the camera (2) attached to the remote control (1) takes a picture of the electrical appliance that the remote control (1) is pointed at, and the machine learning training data instantly identifies the electrical appliance, and the remote control transmits a wireless communication signal to turn on the power of that electrical appliance. Depending on what the user points the remote control (1) at, they can turn on various electrical appliances with the same button (6). This remote control system can also be installed on smartphones and robots that help with housework and work.

Description

Technical Field

[0001] The present invention relates to a system capable of discriminating various electrical appliances that can be operated by wireless communication by machine learning and operating each electrical appliance, a remote control and a robot provided with the same, and an application for setting the same.

Background Art

[0002] For known conventional remote controls, multiple remote controls corresponding to individual electrical appliances are required for a TV, a lighting device, and an air conditioner, respectively.

[0003] In recent years, for example, it is possible to operate multiple home appliances from one device such as a voice recognition engine such as Amazon Alexa or a smartphone. However, with a voice recognition engine, it is necessary to utter a wake-up call to activate the voice recognition engine, and there are times when you do not want to make much noise. With operations from a smartphone, there are inconveniences such as switching the screen to operate the operation application, and it cannot be said that the convenience like a remote control is no longer needed.

[0004] In addition, there is a remote control system equipped with machine learning for checking whether an electrical appliance is operating normally, such as in Patent Document 1 (see the solution, claim 1, and paragraph 0011). However, the present invention is equipped with machine learning for discriminating electrical appliances or operation targets, and the uses and programs are different. The present invention is, for example, a use and program in which machine learning discriminates "that" or "that" pointed by a user such as "Turn on that" or "Turn off that" from sensor information such as a camera module, and changes commands according to the discriminated target.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] The present invention provides a remote control system and a remote control application that can automatically identify and operate various electrical appliances, such as TV remotes, lighting remotes, and air conditioner remotes, using a single remote control, eliminating the need for separate remotes for each appliance. [Means for solving the problem]

[0007] A remote control system is provided that includes a camera on the remote control that photographs the object being controlled. When the remote control is operated, the camera photographs the object, and the captured image is instantly identified using machine learning-based training data. The remote control then transmits a wireless communication signal from the remote control to the electrical appliance.

[0008] Furthermore, the system provides an application that can be run on a PC or other device, allowing users to photograph their electrical appliances with the remote control's camera, learn from the captured images, and customize the association between registered appliances and remote control button operations. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of a remote control equipped with the remote control system of the present invention. [Figure 2] This is a schematic diagram of a PC frame that assists the mouse operation mode of the remote control of the present invention. [Figure 3] This is a flowchart of each mode of the present invention. [Modes for carrying out the invention]

[0010] In this invention, "remote control system" refers to a method or mechanism, while "remote control" refers to a device operated by human hands.

[0011] A remote control system is required to have three steps: 1) data acquisition by the camera module, 2) image analysis and identification using machine learning, and 3) wireless transmission of a command corresponding to the identified object. The system includes a device such as a remote control (1) equipped with one or more camera modules (hereinafter referred to as cameras) (2) for photographing the object to be controlled. When the remote control (1) is operated, the camera (2) first takes a picture of the object, saves the captured image to memory inside the remote control, and then uses a processing unit such as a microcontroller, CPU, or logic chip, also inside the remote control, to determine what the image depicts using pre-trained machine learning data. Subsequently, a command corresponding to the identified object is sent via wireless communication.

[0012] This remote control system allows the user to automatically send the correct command to the target device by determining whether the remote control is pointed at the TV, the lights, or the air conditioner, even when pressing the same power ON (6) button on the remote control (1).

[0013] This is due to a program embedded in the processing unit of the remote control system. This program is written using object-oriented programming. The program's structure is such that it holds instances of the target electrical appliance as an instance of the "Control Target" class. The "Control Target" class has methods for actions such as when the power ON button is pressed, the OFF button is pressed, the up button is pressed, and the down button is pressed. It also has member variables such as captured images, a list of electrical appliance operations, and standard actions. This allows for polymorphic button behavior, as is typical of object-oriented programming, and instances of the "Control Target" class can be freely registered, edited, and deleted by the application later on.

[0014] For example, when you want to operate the air conditioner with a remote control (1) placed by your bedside while you are sleeping and the room is dark, it would be even better if the device were equipped with an infrared light (3) and an infrared camera (2) so that the target can be recognized. When taking a photograph to identify the target, the infrared light (3) would flash and the infrared camera (2) would take the picture, allowing the human eye to identify the target even in complete darkness.

[0015] Since most home appliances with wireless communication use infrared communication, it is preferable for the remote control to be equipped with an infrared transmitter (4) for sending commands to the appliance and an infrared receiver (4) for learning infrared patterns from the appliance's original remote control.

[0016] It would also be good if the remote control had wireless communication capabilities such as Wi-Fi for controlling smart home appliances, as well as Bluetooth for other wireless communication purposes.

[0017] Furthermore, with smart home appliances and other appliances that do not use infrared communication, for example, if the same type of light fixture is used in different rooms, the remote control (1) cannot determine which room the light fixture belongs to based solely on analysis of the image data from the photos it takes. To solve this, it would be even better to equip the remote control with sensors such as a GPS sensor, gyroscope, compass sensor, and accelerometer, and supplement machine learning with the data from these sensors so that even if the lights are the same shape, the remote control can distinguish which light fixture belongs to which room.

[0018] The system includes an application that can be launched on a PC or smartphone to register new information to the remote control (1), to learn additional images taken by the remote control (1), and to assign commands to the buttons (6) of the remote control. This allows for changes to the settings of the remote control (1) and the program and class instance of the processing unit within the remote control.

[0019] It is even better if this application has learning data that can identify major products of major consumer electronics companies, etc., prepared in advance as presets. Furthermore, with application version updates, presets are constantly updated and added, and the application is constantly evolving, and it would be good to have a mechanism that allows users to register for subscriptions, etc.

[0020] The remote control (1) can be linked with the above-mentioned application that is easy for the user to operate through physical cables such as Wi-Fi, Bluetooth, USB, etc.

[0021] The object to be aimed at by the camera (2) when operating with the remote control (1) does not have to be a consumer electronics product. For example, if you operate while aiming at the bed, the lighting for sleeping can be switched, or if you operate while aiming at the entrance door before going out, you can turn off the power of the air conditioner, lighting, etc. all at once. This allows for flexible customization by combining what objects to add for learning and what commands to issue based on the identified objects. Furthermore, even an illustration of a bed can be registered, and for example, if the influence degree of sensors such as a direction sensor is greatly learned by machine learning, the lighting at the east-facing corner of the ceiling can be made white like the morning sun, and the lighting at the west-facing corner of the ceiling can be made warm-colored like the evening sun, etc. can also be registered.

[0022] Also, in the future, it is considered that the number of smart home appliances and those equipped with wireless communication will increase, such as opening the car door, locking the key at the entrance, closing the curtain. This remote control may be able to be operated towards any object like a magic wand. Also, not only turning on and off the power, but raising and lowering the temperature of the air conditioner, raising and lowering the volume of the TV, the degree of opening of the curtain, the brightness of the lighting, etc. can be operated with the same up and down buttons (7) by a polymorphic program.

[0023] By constantly taking pictures with the camera (2) on the remote control (1), it is possible to take pictures of a monitor such as a computer and thereby perform operations like a computer mouse.

[0024] The camera transmits information about which part of the captured image is the monitor (12) and the coordinates of the camera's center within that monitor area (13) to a computer via Bluetooth or other means, allowing mouse operation by moving the pointer. Alternatively, the image can be sent to the computer and the computer can calculate the coordinates. In this case, it would be even better if an application is running on the computer that provides hints, such as lighting up the monitor frame (14) to make it easier to see which part of the image is the monitor.

[0025] For applications that indicate the monitor frame, it would be even better if the left and right sides of the monitor were marked with red dotted lines (14E), the top with a green dotted line (14C), and the bottom with a blue dotted line (14D), as this would make image recognition easier.

[0026] This frame also needs to be displayed in front of all other applications, but it doesn't need to be displayed all the time. If the monitor can be identified through image recognition, it can be hidden even if it's not displayed. If the monitor is not recognized even when a hint frame is displayed on the monitor (14A), it's possible that the remote control is too close to the monitor. In that case, you can try displaying a hint frame in the center of the screen (14B) or scanning the screen to enable image recognition.

[0027] The hint frame (14) can calculate not only where the remote control is pointing on the monitor, but also its relative position and angle from the monitor, based on lens distortion, perspective, how dotted lines appear, color recognition, and 3D camera tracking technology used in CG video production. Therefore, it can be used not only for simple mouse operation, but also as a controller for games that involve hand movements.

[0028] Since precise mouse movements are difficult to perform using only the orientation of the remote control (1), it would be even better if a joystick (7), directional pad, or trackball could be used to supplement this.

[0029] Furthermore, it would be even better if the mouse click buttons (8) were positioned in a location that makes them easy to click with the index or middle finger when holding the remote control (1).

[0030] The camera (2) for the remote control (1) used to operate electrical appliances should be directional, but the camera (2) used to operate the mouse should be wide-angle and have a wide field of view, like a fisheye lens. This means that there can be multiple cameras (2) with different lenses, or the image can be cropped when using a wide-angle lens to operate home appliances.

[0031] In the mode for operating electrical appliances (A), a photo of the target is taken after pressing a button on the remote control (1), but in mouse mode (D), the camera (1) needs to constantly take pictures and calculate coordinates in order to operate the pointer, so the standby behavior is different. Therefore, a toggle switch (9) is provided to switch between the electrical appliance operation mode (A) and the mouse operation mode (D).

[0032] Furthermore, it would be better if the appliance operation mode (A) also allowed switching between operation mode (A), target registration mode (B), and infrared learning mode (C).

[0033] The remote control (1) has a microphone (11), and it would be even better if the user could use voice input. This would allow commands to be issued to electrical appliances via voice input, but it could be used more for registering and learning which devices to control. For example, after voice-inputting "television" (B04), taking several photos of a television for machine learning (B07) would automatically create a folder called "television" in memory (in programming terms, this creates an instance of the control target class called "television"), and then place the captured images into it (B08).

[0034] The remote control (1) should have lights such as LEDs (5) that send signals to the user, allowing them to determine what operation the remote control is performing, whether the battery is sufficient, etc. A simple display would be even better than the LEDs (5) for these user signals.

[0035] The remote control (1) should have a laser pointer (10) that is as gentle on the eyes as possible, which would be helpful in helping the user determine which direction the remote control (1) is pointing.

[0036] Furthermore, the relationship between the remote control system of the present invention, which has a program that photographs and identifies the target of this operation and sends corresponding commands, and the remote control that issues commands to the system using simplified buttons (6, 7), may be remotely controlled. For example, a drone can be used for security patrols of a building, and a security guard at a distance can monitor the video footage taken by the drone. If an electrical appliance is left on, the security guard can turn it off with the remote control in front of the drone.

[0037] Furthermore, even if the robot is fully automated and controlled by AI or programmed systems, the remote control system of the present invention allows, for example, a patrol robot to photograph electrical appliances with a camera module, identify the appliances using machine learning training data, and control them using wireless communication appropriate for those appliances.

[0038] Furthermore, as a future application of the remote control system of the present invention, a commander robot can photograph work robots with a camera module, identify the work robots using machine learning training data, and control them via wireless communication appropriate to each work robot. In this case, wireless communication is thought to be primarily radio wave-based, such as Wi-Fi or Bluetooth, but technologies such as sound wave communication and infrared communication can also be used to send commands only to robots within a certain range, or only to robots facing each other. [Examples]

[0039] The remote control equipped with the above remote control system is placed in the living room for the whole family to use. [Examples]

[0040] The small remote control equipped with the above remote control system can be carried like a keychain and used anywhere, such as at home, in the car, or at work. [Examples]

[0041] The above remote control system is used when remotely operating radio-controlled devices or drones for building security, etc. [Examples]

[0042] The above remote control system is used when remotely operating caregiving robots or robots that support daily life and work. [Industrial applicability]

[0043] We will sell the remote control system of the present invention. We will sell the remote control of this invention. We maintain the remote control system and applications of the present invention and sell subscriptions for them. [Explanation of symbols]

[0044] 1 Remote control 2 Camera Modules 3. Infrared light 4. Infrared Transmitter / Receiver 5. LED lights that display signs for the user 6 Remote control buttons 7. Joystick for fine-tuning the up, down, left, and right inputs of the remote control and the pointer of the mouse function. 8. Click buttons for mouse operation mode 9. Toggle switch to switch modes 10 Laser Pointers 11. Microphone for voice input 12 PC monitors 13. Computer screen 14 Hint Box 14A Hint box for scanning the entire screen 14B Hint box for partial scanning when full-screen scanning fails. 14C Hint frame for the top of the screen; use a different color, such as green, to indicate that it is the top of the screen. 14D Hint box for the bottom of the screen; use a different color, such as blue, to indicate that it is the bottom. 14E Hint frames for the left and right sides of the screen; different colors, such as red, are used to indicate which side is which.

Claims

1. A remote control system characterized by having one or more camera modules, which analyze images captured by these modules using machine learning data obtained from household electrical appliances as training data, and can send commands according to the type of wireless communication that the identified electrical appliance can receive, or execute pre-set routine actions, thereby enabling commands to be sent from a single device to various electrical appliances equipped with wireless communication.

2. A remote control system as described in claim 2, characterized in that it includes a memory for storing images, programs, and training data captured by the remote control system, and a processing unit such as a CPU or microcontroller for analyzing images from the training data.

3. A remote control system characterized by having an object-oriented, polymorphic program that has a method for turning on the same power supply to different electrical appliances, as described in claim 3.

4. A remote control system as described in claim 1, characterized in that it comprises an infrared communication transmitter and a transmitter such as Wi-Fi or Bluetooth, which is generally used for wireless communication to send commands to electrical appliances, and an infrared receiver for learning infrared signals.

5. An application that can be launched on a personal computer or the like to update the learning data by adding image data captured by the remote control system described in claim 4, adding training data such as electrical appliances, and registering and editing electrical appliances, setting routine actions, and associating them with photographs for the program described in claim 1.

6. The remote control system according to claim 1, further comprising the function to communicate with the application described in claim 5 via Wi-Fi, Bluetooth, USB or other physical cables, and to share settings, photos, and learning data.

7. A remote control system characterized in that one or more of the cameras in the remote control system described in claim 6 are infrared cameras, and it is equipped with an infrared light for illuminating a target in darkness, and the infrared light illuminates a target even in darkness that is invisible to the human eye, and it can identify an electrical appliance from the image captured by the infrared camera and transmit a command corresponding to the target.

8. A remote control system according to claim 7, characterized in that it is equipped with sensors such as a GPS sensor, a gyro sensor, and a compass sensor so that it can distinguish between identical products, for example, home lighting and office lighting, even if they are the same product, and the sensor information is used to assist image analysis and learning data.

9. A remote control equipped with the remote control system described in claims 1 to 4 or claims 6 to 8.

10. A robot including a radio-controlled car or drone equipped with the remote control system described in claims 1 to 4 or claims 6 to 8.

11. A smartphone equipped with the remote control system described in claims 1 to 4 or claims 6 to 8.

12. A remote control according to claim 9, further comprising one or more buttons for operating an object.

13. A remote control equipped with a microphone capable of voice input, according to claim 9.

14. A remote control according to claim 9, further comprising a laser pointer for making it easier to understand what the remote control is pointing to.

15. This remote control features one or more camera modules and one or more click buttons. The camera modules capture images of a computer monitor, determine which pixels on the monitor the camera is pointing at, and transmit the coordinate information of those pixels or the captured image to the computer. It also functions as a mouse, allowing for pointer control and clicking.

16. A remote control according to claim 15, further comprising a joystick or directional pad, or trackball, for fine-tuning the pointer.

17. An application that communicates with the remote control of claim 16, illuminates the pixels of the frame of the PC screen, and gives a hint of the PC screen position to the camera provided in the remote control.

18. A remote control characterized by the ability to switch between a remote control for operating an electrical appliance as described in claim 9 and a remote control having the mouse function of a personal computer as described in claim 16.

Images