A remote control system capable of operating various electrical appliances, a remote control equipped with the system, a robot, and an application for configuring the system.

Abstract

Traditional remote controls required separate remotes for different electrical appliances such as TVs, lights, and air conditioners. This product provides a remote control that can automatically identify and operate various electrical appliances with a single remote, along with a remote control system and an application to customize that system. [Solution] For example, when the power ON button (6) of the remote control (1) is operated, the camera (2) attached to the remote control (1) takes a picture of the electrical appliance that the remote control (1) is pointed at, and the machine learning training data instantly identifies the electrical appliance, and the remote control transmits a wireless communication signal to turn on the power of that electrical appliance. Depending on what the user points the remote control (1) at, they can turn on various electrical appliances with the same button (6). This remote control system can also be installed on smartphones and robots that help with housework and work.

Description

Technical Field

[0001] The present invention relates to a system capable of discriminating various electrical appliances that can be operated by wireless communication through machine learning and operating each electrical appliance, a remote control and a robot equipped with the same, and an application for setting the same.

Background Art

[0002] For known conventional remote controls, a plurality of remote controls corresponding to individual electrical appliances were required for a TV, a lighting remote control for lighting, and an air conditioner remote control for an air conditioner.

[0003] In recent years, for example, it is possible to operate a plurality of home appliances from one device such as a voice recognition engine such as Amazon Alexa or a smartphone. However, with a voice recognition engine, it is necessary to utter a wake-up call to activate the voice recognition engine, and there are times when you don't want to make much noise. When operating from a smartphone, there are nuisances such as switching the screen to operate the operation application, and it cannot be said that the convenience like a remote control is completely unnecessary.

[0004] In addition, there is a remote control system equipped with machine learning for checking whether an electrical appliance is operating normally, such as in Patent Document 1 (see the solution, claim 1, and paragraph 0011). However, the present invention is equipped with machine learning for discriminating an electrical appliance or an operation target, and the uses and programs are different. The present invention is a use and program in which machine learning discriminates "that" or "that" pointed by a user such as "Turn on that" or "Turn off that" from sensor information such as a camera module, and changes a command according to the discriminated target.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] The present invention provides a remote control system and a remote control application that can automatically identify and operate various electrical appliances, such as TV remotes, lighting remotes, and air conditioner remotes, using a single remote control, eliminating the need for separate remotes for each appliance. [Means for solving the problem]

[0007] A remote control system is provided that includes a camera on the remote control that photographs the object being controlled. When the remote control is operated, the camera photographs the object, and the captured image is instantly identified using machine learning-based training data. The remote control then transmits a wireless communication signal from the remote control to the electrical appliance.

[0008] Furthermore, the system provides an application that can be run on a PC or other device, allowing users to photograph their electrical appliances with the remote control's camera, learn from the captured images, and customize the association between registered appliances and remote control button operations. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of a remote control equipped with the remote control system of the present invention. [Figure 2] This is a schematic diagram of a PC frame that assists the mouse operation mode of the remote control of the present invention. [Figure 3] This is a flowchart of each mode of the present invention. [Modes for carrying out the invention]

[0010] In this invention, "remote control system" refers to a method or mechanism, while "remote control" refers to a device operated by human hands.

[0011] A remote control system is required to have three steps: 1) data acquisition by the camera module, 2) image analysis and identification using machine learning, and 3) wireless transmission of a command corresponding to the identified object. The system includes a device such as a remote control (1) equipped with one or more camera modules (hereinafter referred to as cameras) (2) for photographing the object to be controlled. When the remote control (1) is operated, the camera (2) first takes a picture of the object, saves the captured image to memory inside the remote control, and then uses a processing unit such as a microcontroller, CPU, or logic chip, also inside the remote control, to determine what the image depicts using pre-trained machine learning data. Subsequently, a command corresponding to the identified object is sent via wireless communication.

[0012] This remote control system allows the user to automatically send the correct command to the target device by determining whether the remote control is pointed at the TV, the lights, or the air conditioner, even when pressing the same power ON (6) button on the remote control (1).

[0013] This is due to a program embedded in the processing unit of the remote control system. This program is written using object-oriented programming. The program's structure is such that it holds instances of the target electrical appliance as an instance of the "Control Target" class. The "Control Target" class has methods for actions such as when the power ON button is pressed, the OFF button is pressed, the up button is pressed, and the down button is pressed. It also has member variables such as captured images, a list of electrical appliance operations, and standard actions. This allows for polymorphic button behavior, as is typical of object-oriented programming, and instances of the "Control Target" class can be freely registered, edited, and deleted by the application later on.

[0014] For example, when you want to operate the air conditioner with a remote control (1) placed by your bedside while you are sleeping and the room is dark, it would be even better if the device were equipped with an infrared light (3) and an infrared camera (2) so that the target can be recognized. When taking a photograph to identify the target, the infrared light (3) would flash and the infrared camera (2) would take the picture, allowing the human eye to identify the target even in complete darkness.

[0015] Since most home appliances with wireless communication use infrared communication, it is preferable for the remote control to be equipped with an infrared transmitter (4) for sending commands to the appliance and an infrared receiver (4) for learning infrared patterns from the appliance's original remote control.

[0016] It would also be good if the remote control had wireless communication capabilities such as Wi-Fi for controlling smart home appliances, as well as Bluetooth for other wireless communication purposes.

[0017] Furthermore, with smart home appliances and other appliances that do not use infrared communication, for example, if the same type of light fixture is used in different rooms, the remote control (1) cannot determine which room the light fixture belongs to based solely on analysis of the image data from the photos it takes. To solve this, it would be even better to equip the remote control with sensors such as a GPS sensor, gyroscope, compass sensor, and accelerometer, and supplement machine learning with the data from these sensors so that even if the lights are the same shape, the remote control can distinguish which light fixture belongs to which room.

[0018] The system includes an application that can be launched on a PC or smartphone to register new information to the remote control (1), to learn additional images taken by the remote control (1), and to assign commands to the buttons (6) of the remote control. This allows for changes to the settings of the remote control (1) and the program and class instance of the processing unit within the remote control.

[0019] It is even better if this application has learning data that can identify major products of major appliance companies, etc., prepared in advance as presets. Furthermore, with software updates, the presets are constantly updated and added, enabling the application to constantly evolve. It is also beneficial to have a mechanism that allows users to register for subscriptions, etc.

[0020] The remote control (1) can be linked with the above-mentioned application for easy operation by the user through physical cables such as Wi-Fi, Bluetooth, USB, etc.

[0021] The object to be aimed at by the camera (2) when operating with the remote control (1) does not have to be an electrical appliance. For example, if you operate it while aiming at the bed, it can switch to the lighting for sleeping, or if you operate it while aiming at the front door before going out, you can turn off the power of the air conditioner, lighting, etc. all at once. This allows for flexible customization by combining what objects to add to the learning and what commands to issue based on the identified objects. Furthermore, even an illustration of a bed can be registered, and for example, if the influence degree of sensors such as a direction sensor is greatly learned by machine learning, it is possible to register to make the lighting white like the morning sun for the east-facing corner of the ceiling and make the lighting warm-colored like the evening sun for the west-facing corner of the ceiling.

[0022] [[ID=,12]]Also, in the future, it is considered that the number of smart home appliances and those equipped with wireless communication will increase, such as opening the car door, locking the front door, closing the curtains. This remote control may be able to operate on any object like a magic wand. Also, not only turning on and off the power, but raising and lowering the temperature of the air conditioner, raising and lowering the volume of the TV, opening degree of the curtains, brightness of the lighting, etc. can be operated with the same up and down buttons (7) by a polymorphic program.

[0023] By constantly taking pictures with the camera (2) on the remote control (1), it is possible to take pictures of a monitor such as a computer and perform operations like a computer mouse.

[0024] The camera transmits information about which part of the captured image is the monitor (12) and the coordinates of the camera's center within that monitor area (13) to a computer via Bluetooth or other means, allowing mouse operation by moving the pointer. Alternatively, the image can be sent to the computer and the computer can calculate the coordinates. In this case, it would be even better if an application is running on the computer that provides hints, such as lighting up the monitor frame (14) to make it easier to see which part of the image is the monitor.

[0025] For applications that indicate the monitor frame, it would be even better if the left and right sides of the monitor were marked with red dotted lines (14E), the top with a green dotted line (14C), and the bottom with a blue dotted line (14D), as this would make image recognition easier.

[0026] This frame also needs to be displayed in front of all other applications, but it doesn't need to be displayed all the time. If the monitor can be identified through image recognition, it can be hidden even if it's not displayed. If the monitor is not recognized even when a hint frame is displayed on the monitor (14A), it's possible that the remote control is too close to the monitor. In that case, you can try displaying a hint frame in the center of the screen (14B) or scanning the screen to enable image recognition.

[0027] The hint frame (14) can calculate not only where the remote control is pointing on the monitor, but also its relative position and angle from the monitor, based on lens distortion, perspective, how dotted lines appear, color recognition, and 3D camera tracking technology used in CG video production. Therefore, it can be used not only for simple mouse operation, but also as a controller for games that involve hand movements.

[0028] Since precise mouse movements are difficult to perform using only the orientation of the remote control (1), it would be even better if a joystick (7), directional pad, or trackball could be used to supplement this.

[0029] Furthermore, it would be even better if the mouse click buttons (8) were positioned in a location that makes them easy to click with the index or middle finger when holding the remote control (1).

[0030] The camera (2) for the remote control (1) used to operate electrical appliances should be directional, but the camera (2) used to operate the mouse should be wide-angle and have a wide field of view, like a fisheye lens. This means that there can be multiple cameras (2) with different lenses, or the image can be cropped when using a wide-angle lens to operate home appliances.

[0031] In the mode for operating electrical appliances (A), a photo of the target is taken after pressing a button on the remote control (1), but in mouse mode (D), the camera (1) needs to constantly take pictures and calculate coordinates in order to operate the pointer, so the standby behavior is different. Therefore, a toggle switch (9) is provided to switch between the electrical appliance operation mode (A) and the mouse operation mode (D).

[0032] Furthermore, it would be better if the appliance operation mode (A) also allowed switching between operation mode (A), target registration mode (B), and infrared learning mode (C).

[0033] The remote control (1) has a microphone (11), and it would be even better if the user could use voice input. This would allow commands to be issued to electrical appliances via voice input, but it could be used more for registering and learning which devices to control. For example, after voice-inputting "television" (B04), taking several photos of a television for machine learning (B07) would automatically create a folder called "television" in memory (in programming terms, this creates an instance of the control target class called "television"), and then place the captured images into it (B08).

[0034] The remote control (1) should have lights such as LEDs (5) that send signals to the user, allowing them to determine what operation the remote control is performing, whether the battery is sufficient, etc. A simple display would be even better than the LEDs (5) for these user signals.

[0035] The remote control (1) should have a laser pointer (10) that is as gentle on the eyes as possible, which would be helpful in helping the user determine which direction the remote control (1) is pointing.

[0036] Furthermore, the relationship between the remote control system of the present invention, which has a program that photographs and identifies the target of this operation and sends corresponding commands, and the remote control that issues commands to the system using simplified buttons (6, 7), may be remotely controlled. For example, a drone can be used for security patrols of a building, and a security guard at a distance can monitor the video footage taken by the drone. If an electrical appliance is left on, the security guard can turn it off with the remote control in front of the drone.

[0037] Furthermore, even if the robot is fully automated and controlled by AI or programmed systems, the remote control system of the present invention allows, for example, a patrol robot to photograph electrical appliances with a camera module, identify the appliances using machine learning training data, and control them using wireless communication appropriate for those appliances.

[0038] Furthermore, as a future application of the remote control system of the present invention, a commander robot can photograph work robots with a camera module, identify the work robots using machine learning training data, and control them via wireless communication appropriate to each work robot. In this case, wireless communication is thought to be primarily radio wave-based, such as Wi-Fi or Bluetooth, but technologies such as sound wave communication and infrared communication can also be used to send commands only to robots within a certain range, or only to robots facing each other. [Examples]

[0039] The remote control equipped with the above remote control system is placed in the living room for the whole family to use. [Examples]

[0040] The small remote control equipped with the above remote control system can be carried like a keychain and used anywhere, such as at home, in the car, or at work. [Examples]

[0041] The above remote control system is used when remotely operating radio-controlled devices or drones for building security, etc. [Examples]

[0042] The above remote control system is used when remotely operating caregiving robots or robots that support daily life and work. [Industrial applicability]

[0043] We will sell the remote control system of the present invention. We will sell the remote control of this invention. We maintain the remote control system and applications of the present invention and sell subscriptions for them. [Explanation of Symbols]

[0044] 1 Remote control 2 Camera Modules 3. Infrared light 4. Infrared Transmitter / Receiver 5. LED lights that display signs for the user 6 Remote control buttons 7. Joystick for fine-tuning the up, down, left, and right inputs of the remote control and the pointer of the mouse function. 8. Click buttons for mouse operation mode 9. Toggle switch to switch modes 10 Laser Pointers 11. Microphone for voice input 12 PC monitors 13. Computer screen 14 Hint Box 14A Hint box for scanning the entire screen 14B Hint box for partial scanning when full-screen scanning fails. 14C Hint frame for the top of the screen; use a different color, such as green, to indicate that it is the top of the screen. 14D Hint box for the bottom of the screen; use a different color, such as blue, to indicate that it is the bottom. 14E Hint frames for the left and right sides of the screen; different colors, such as red, are used to indicate which side is which.

Claims

[Claim 1] The invention described in the specification.

Images