Remote control system and method based on event camera, electronic equipment and storage medium

Abstract

The invention relates to a remote control system and method based on an event camera, electronic equipment and a computer readable storage medium. The system comprises display equipment and remote control equipment, the display equipment comprises a display screen and a matched light source arranged on the display screen; the remote control equipment comprises a transmitting module, an event camera and a calculation control module; wherein the transmitting module is used for transmitting invisible light to the display screen; the event camera is used for monitoring the relative position of the invisible light relative to the matched light source to obtain an event image; and the calculation control module is used for determining a relative position according to the event image, determining a position or a movement track of the invisible light relative to the display screen according to the relative position, and performing interaction control on the display screen according to the position or the movement track. The interaction function can be expanded, and the interaction efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of interactive control technology, and in particular to a remote control system, method, electronic device, and storage medium (computer-readable storage medium) based on an event camera. Background Technology

[0002] In daily life and work, display devices such as televisions, computers, and projectors are widely used, and usually require the use of matching interactive devices such as remote controls, mice, or laser pointers for instruction or operation.

[0003] Regardless of the interaction method, the supported interactive functions are very limited.

[0004] For example, laser pointers only support click and drag operations, and the interactive functions they can support are very limited, far from meeting the user's interactive needs. Summary of the Invention

[0005] This application provides a remote control system based on an event camera, a control method based on an event camera, an electronic device, and a computer-readable storage medium, which aim to expand interactive functions and improve interactive efficiency.

[0006] To achieve the above objectives, according to a first aspect of this application, a remote control system based on an event camera is provided, the system comprising:

[0007] Display devices, and remote control devices;

[0008] The display device includes a display screen and a matching light source disposed on the display screen;

[0009] The remote control device includes a transmitting module, an event camera, and a computing control module; wherein,

[0010] The transmitting module is used to emit invisible light to the display screen;

[0011] The event camera is used to monitor the relative position of the invisible light with respect to the matching light source and to obtain event images;

[0012] The calculation and control module is used to determine the relative position based on the event image, and also to determine the position or motion trajectory of the invisible light relative to the display screen based on the relative position, and to perform interactive control of the display screen based on the position or the motion trajectory.

[0013] Optionally, the light emitted by the matching light source and the invisible light are both light whose intensity varies with time. The light emitted by the matching light source and the invisible light reflected by the display screen are both incident on the event camera to form the event image. The emitting module is a freely movable emitting module so that the invisible light can move freely relative to the display screen, and the event camera can obtain multiple consecutive frames of the event image.

[0014] Optionally, the step of interactively controlling the display screen based on the position or the motion trajectory includes:

[0015] The invisible light incident point is mapped onto the display screen as a display cursor according to the position, the display cursor is moved according to the motion trajectory, and the display screen is interactively controlled based on the display cursor.

[0016] Optionally, the step of interactively controlling the display screen based on the position or the motion trajectory includes:

[0017] Based on the location or the movement trajectory, obtain the operation command;

[0018] The display screen is interactively controlled according to the operation instructions.

[0019] Optionally, the remote control device is provided with multiple function buttons, including a control mode switching button. The control mode switching button is used to trigger a control mode selection event to interactively control the display screen according to the control mode corresponding to the control mode selection event.

[0020] According to a second aspect of this application, a control method based on an event camera is provided, the method being applied to a remote control device, the remote control device including a transmitting module and the event camera, the remote control device being communicatively connected to a display device, the display device including a display screen and a matching light source disposed on the display screen, the method comprising:

[0021] The emission module emits invisible light whose intensity varies over time into the display screen;

[0022] The event camera monitors the relative position of the invisible light with respect to the matching light source to obtain an event image, wherein the matching light source emits light with intensity varying over time.

[0023] The relative position is determined based on the event image, and the position or trajectory of the invisible light relative to the display screen is determined based on the relative position.

[0024] The display screen is interactively controlled based on the location or the movement trajectory.

[0025] Optionally, the step of interactively controlling the display screen based on the position or the motion trajectory includes:

[0026] The invisible light incident point is mapped onto the display screen as a display cursor according to the position, the display cursor is moved according to the motion trajectory, and the display screen is interactively controlled based on the display cursor.

[0027] Optionally, the step of interactively controlling the display screen based on the position or the motion trajectory includes:

[0028] Based on the location or the movement trajectory, obtain the operation command;

[0029] The display screen is interactively controlled according to the operation instructions.

[0030] Optionally, before emitting invisible light of varying intensity over time to the display screen via the emitting module, the method further includes:

[0031] Trigger the control mode selection event;

[0032] In response to the control mode selection event, the remote control device enters the target control mode and performs corresponding interactive control.

[0033] If the target control mode is the first control mode, then during interactive control, a function control signal is first triggered, and then the function control signal is responded to to generate a corresponding operation instruction and send the operation instruction to the display device to control the display device.

[0034] If the target control mode is the second control mode, then during interactive control, the transmitting module and the matching light source are activated, so that the transmitting module emits invisible light with intensity varying over time to the display screen, and the matching light source emits light with intensity varying over time. The event camera is also activated to monitor the relative position of the invisible light with respect to the matching light source and obtain the event image.

[0035] According to a third aspect of this application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the electronic device to implement the steps of the above-described event camera-based control method.

[0036] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when the computer program is run, implements the steps of the above-described event-based camera control method.

[0037] According to a fifth aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to implement the steps of the above-described event-based camera control method.

[0038] The remote control system, method, electronic device, computer-readable storage medium, and computer program product based on an event camera in this application embodiment allow users to monitor the relative position of the invisible light emitted by the invisible light emitting module relative to the matching light source by moving the remote control device. This results in an event image, which the computing control module can then use to interactively control the display screen based on the event image. Since the movement trajectory or actions of the remote control device relative to the display screen vary, this application can combine multiple different trajectories or actions to define multiple different control commands, enabling various complex operations on the display screen and greatly expanding the interactive functions.

[0039] Based on this, since the incident point of the invisible light emitted by the invisible light emitting module is very small, while the size of the display screen is relatively large, even if the remote control device moves (or rotates) only slightly relative to the display device, it can be effectively magnified and clearly displayed on the display screen and collected and identified by the event camera and computing control module, which effectively improves the sensitivity and accuracy of the sensing remote control. In addition, since the event camera has a small data volume, low power consumption, and low risk of leaking user privacy, this application can also ensure the security of data interaction.

[0040] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] To gain a more complete understanding of this application and its beneficial effects, the following description will be provided in conjunction with the accompanying drawings, wherein the same reference numerals in the following description denote the same parts.

[0043] Figure 1 This is a schematic diagram of a remote control system structure provided in an exemplary embodiment of this application;

[0044] Figure 2This is a schematic diagram of display interaction provided in one exemplary embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the emission frequency of the matching light source provided in an exemplary embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the first process of the control method provided in an exemplary embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the second process of the control method provided in an exemplary embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0051] In this embodiment, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following associated objects have an "or" relationship.

[0052] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. In the following description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0053] In conjunction with the background description of this application, display devices such as televisions, computers, and projectors are widely used in daily life and work, and usually require the use of matching interactive devices such as remote controls, mice, or laser pointers for instruction or operation.

[0054] However, existing interactive devices have at least the following drawbacks:

[0055] (1) Remote control is easy to operate and suitable for long-distance control. It generally has basic functions such as channel switching, volume adjustment and program selection. However, due to the limited number of buttons and function design, it is often relatively simple in function and has low flexibility. It cannot support complex display screen operations, such as text input and fine graphic editing.

[0056] (2) The mouse is highly accurate and flexible, and can achieve precise positioning and complex operations on the display screen, such as clicking, dragging, scrolling, and selecting. However, the interaction between the mouse and the display device (such as a computer) requires a flat operating environment (such as a flat desktop or other operating platform) for stable mouse control, which makes the mouse user easily confined to the limited space next to the desktop.

[0057] (3) Laser pointers use laser beams to indicate specific areas on the display screen to attract the audience’s attention. Some laser pointers also integrate remote control functions, which can be used for simple display screen operations, such as page scrolling. However, the display screen operations that laser pointers can support are limited, such as only simple operations like clicking and dragging. Moreover, the laser beam may cause damage to the eyes.

[0058] (4) In order to improve the user experience, a technical solution can also be considered to use a CMOS image sensor to collect the motion trajectory of the remote control or laser pointer, locate the light spot on the display screen, and use specific software to accurately realize remote tapping, dragging, and selection functions on the display screen. However, traditional image sensor positioning has disadvantages such as high power consumption, large data volume, and risk of user privacy leakage.

[0059] With the development of image sensing technology, event-based vision sensors (EVS) have emerged. Event-based vision sensors (DVS), also known as dynamic vision sensors, are a type of biomimetic vision sensor. Unlike traditional CMOS image sensors that operate on a frame-by-frame basis, event-based vision sensors mimic the structure of the human eye. They asynchronously output event information, including location, time, and polarity, only when the logarithmic change in light intensity between frames exceeds a certain threshold. Compared to traditional CMOS image sensors, event-based vision sensors have the advantages of low power consumption, small data volume, and high dynamic range.

[0060] This application takes into account the aforementioned characteristics of event cameras and applies them to display device interaction. By combining an invisible light emitting module, an event camera, and a matching light source to design a remote control system, the invisible light emitted by the invisible light emitting module can be displayed on the screen of the display device, and the movement of the invisible light emitting module can instruct the display screen. Furthermore, by combining the various trajectories or movements of the invisible light emitted by the invisible light emitting module relative to the display device, various different control commands can be defined to enrich interactive functions, improve interactive efficiency, and achieve low-cost, feature-rich, and flexible interactive instruction and remote control. At the same time, because the event camera has small data volume, low power consumption, and low risk of user privacy leakage, this application can effectively improve data security during the interaction process.

[0061] Specifically, this application proposes a remote control system based on an event camera, a control method based on an event camera, an electronic device, and a computer-readable storage medium, aiming to reduce interaction costs, expand interaction functions, and improve interaction flexibility.

[0062] In one embodiment, the remote control system based on an event camera in this application may include:

[0063] Display devices, and remote control devices;

[0064] Display devices, including a display screen and a matching light source disposed on the display screen;

[0065] The remote control device includes a transmitting module, an event camera, and a computing control module; among which,

[0066] The transmitting module is used to emit invisible light to the display screen;

[0067] An event camera is used to monitor the relative position of invisible light with respect to a matching light source and to obtain event images.

[0068] The calculation and control module is used to determine the relative position based on the event image, and also to determine the position or motion trajectory of invisible light relative to the display screen based on the relative position, and to perform interactive control of the display screen based on the position or motion trajectory.

[0069] In this embodiment, the remote control system may include a display device and a remote control device. The remote control device and the display device can be connected wirelessly, such as via Bluetooth, infrared, mobile network, or WLAN.

[0070] The display device may include a display screen and a matching light source mounted on the display screen. In addition, the display device may include a communication module, which may be a two-way communication module or a one-way receiving module, for communicating with a remote control device.

[0071] Specifically, such as Figure 2 As shown, the supporting light source in this embodiment may include multiple independent point light sources. These multiple independent point light sources can be fixedly arranged around the display screen. For example, four point light sources can be placed outside the four corners of the display screen to form a regular rectangle. Furthermore, two point light sources can be placed to the left of the two left corners of the display screen, and the other two point light sources can be placed to the right of the two right corners of the display screen, so that the height of the supporting light source is the same as the height of the display screen, while the width of the supporting light source is greater than the width of the display screen. Alternatively, a larger number of point light sources can be arranged at intervals around the edge of the display screen.

[0072] It is understandable that, in theory, the number and distribution of point light sources can be flexibly set and are not restricted. In order to effectively improve the positioning accuracy of the cursor and reduce the difficulty of coordinate mapping in subsequent embodiments, the matching light source can generally be set to a shape similar to or the same as the display screen of the display device.

[0073] It should be noted that the light emitted by the matching light source can be visible light, such as white light, red light, green light, or blue light, and the corresponding point light source can be a laser or a light-emitting diode. However, considering that the visible light emitted by the matching light source may interfere with the user of the display device, the light emitted by the matching light source can also be invisible light, such as infrared light of 850nm or 940nm, and the corresponding point light source can be an infrared light-emitting diode (i.e., an infrared lamp bead), or other invisible light of other wavelengths such as ultraviolet light, which will not be elaborated here.

[0074] The human eye is less sensitive to light with wavelengths above 700nm, and cannot see infrared light sources of normal intensity. Therefore, using infrared LEDs will not affect the user's screen viewing experience. Furthermore, considering that the EVS camera module (i.e., the event camera) only responds to changes in light intensity, in this optional embodiment, the infrared LEDs in the matching light source can be controlled to operate in a flickering mode, for example, as shown below. Figure 3 As shown, the emission frequency of the infrared LED is f = 300Hz.

[0075] Based on this, in more detail, such as Figure 2 As shown, once the matching light source is fixedly installed on the display device, the relative positions of each point light source and the display screen of the display device are fixed. By calibrating and determining the relationship between each point light source and the corner point of the display screen, and then combining the actual size of the display screen, the actual size of the pixels and the actual size of the point light source, the mapping from the coordinate system of the matching light source to the coordinate system of the display screen can be realized, and the calibration of the relative position between the matching light source and the display screen can be completed.

[0076] In this embodiment, the remote control device may include a transmitting module, an event camera, and a computing control module. Figure 2 (not shown in the image), wherein the emitting module in this embodiment can specifically be an invisible light emitting module, which can emit invisible light and project it onto the display screen. It is understood that the invisible light emitting module can be a VCSEL infrared laser emitting module, which is not limited here.

[0077] Remote control devices may also include a communication module. Figure 2 (Not shown in the image), the communication module can be a two-way communication module or a one-way transmission module, and the communication module (first communication module) in the remote control device is communicatively connected to the communication module (second communication module) in the display device.

[0078] In detail, such as Figure 2 As shown, the remote control device also includes function buttons and a sampling module. Figure 2 (Not shown in the image) and the casing.

[0079] Specifically, the function button can be a physical button or a virtual button. Multiple function buttons (physical buttons) are located on the front of the casing for function selection and triggering. The function buttons can include a five-way navigation key, power button, number keys, arrow keys, annotation keys, delete keys, paste keys, and capture keys, etc. The five-way navigation key can further include up keys, down keys, "+" keys, "-" keys, and OK keys, which are not limited here.

[0080] The sampling module, the first communication module, and the calculation control module are located inside the housing. The sampling module is connected to the function button, and the calculation control module is connected to both the sampling module and the first communication module. When a function button is pressed, it triggers the sampling module to generate a corresponding function control signal. The calculation control module generates a control command based on the function control signal and sends the control command to the display device through the first communication module. The display device can receive the control command through the second communication module and respond to various control commands through its internal processing module to execute corresponding control actions, such as display mode switching, channel switching, volume adjustment, and program selection.

[0081] In detail, such as Figure 2 As shown, the invisible light emitting module and the EVS camera module (i.e., the event camera in this embodiment) are respectively disposed at the front end of the housing. The light-emitting surface of the invisible light emitting module and the photosensitive surface of the EVS camera module are exposed from the side of the front end of the housing. The light-emitting surface of the invisible light emitting module and the photosensitive surface of the EVS camera module are symmetrical about the central axis of the housing, so as to facilitate the acquisition of the brightness change of the invisible light emitted by the invisible light emitting module on the display screen through the EVS camera module.

[0082] The invisible light emitting module and the EVS camera module are respectively connected to the computing control module. The computing control module enables and triggers the invisible light emitting module and the EVS camera module respectively. The event images collected by the EVS camera module are sent to the computing control module for calculation and processing to solve the position and trajectory of the invisible light emitted by the invisible light emitting module relative to the display screen.

[0083] Understandably, the EVS camera module can also be installed separately, rather than on the housing, as long as it can capture the light emitted by the light source on the display screen and the invisible light reflected from the display screen.

[0084] In this way, when the user moves the remote control device to drive the invisible light emitting module, the EVS camera module monitors the relative position of the invisible light with respect to the matching light source and obtains the corresponding event image. Then, the calculation control module can determine the relative position of the invisible light with respect to the matching light source based on at least two consecutive event images, and calculate the position or motion trajectory of the invisible light with respect to the display screen based on the relative position, and perform interactive control of the display screen based on the position or motion trajectory.

[0085] Optionally, the light emitted by the matching light source and the invisible light are both light whose intensity changes with time. The light emitted by the matching light source and the invisible light reflected by the display screen are both incident on the event camera to form an event image. The transmitting module is a freely movable transmitting module so that the invisible light can move freely relative to the display screen, and the event camera can obtain multiple consecutive event images.

[0086] In this embodiment, based on the above description, considering that the visible light emitted by the supporting light source may interfere with the user of the display device, the light emitted by the supporting light source can be invisible light, such as 850nm or 940nm infrared light. The corresponding point light source can be an infrared light-emitting diode (i.e., an infrared LED), which operates in a flickering mode. Furthermore, the invisible light emitting module in the remote control device in this embodiment can be a VCSEL infrared laser emitting module. Thus, in this embodiment, both the light emitted by the supporting light source and the invisible light emitted by the invisible light emitting module are light whose intensity changes over time, so that both the light emitted by the supporting light source and the invisible light reflected by the display screen are incident on the event camera to form an event image.

[0087] Understandably, when a user controls a display device using a remote control, the user can move the remote control so that the invisible light emitted by the invisible light emitting module can move freely relative to the display screen. The event camera can obtain corresponding multi-frame continuous event images, and the remote control can determine the position or trajectory of the invisible light relative to the display screen based on the multi-frame continuous event images, and interactively control the display screen based on the position or trajectory.

[0088] Optionally, in an optional embodiment of this application, the aforementioned "interactive control of the display screen based on position or motion trajectory" may include:

[0089] The invisible light incident point is mapped onto the display screen as a display cursor based on its position. The display cursor is moved according to the motion trajectory, and interactive control of the display screen is performed based on the display cursor.

[0090] In this embodiment, when the remote control device moves relative to the display device, after the event camera in the remote control device captures event images of the matching light source at at least two locations, for each frame of the event image, the calculation and control module in the remote control device can calculate the relative position and trajectory of the invisible light relative to the matching light source based on the event image. Then, combined with the relative position between the matching light source and the display screen, the position of the invisible light relative to the display screen can be calculated. Based on the above position, the incident point of the invisible light can be mapped onto the display screen as a display cursor, and the display screen can be interactively controlled based on the display cursor.

[0091] Specifically, for example, when the remote control device moves relative to the display device, for a certain frame of event image, the position of invisible light relative to the display screen is determined based on the position of the light source in the event image. Then, based on this position, the incident point of the invisible light is projected onto the display screen as a display cursor. When the remote control device moves relative to the display device, for multiple consecutive frames of event images, the position of invisible light relative to the display screen is determined based on the position of the light source in the event image. Based on the corresponding multiple positions of the invisible light relative to the display screen, the motion trajectory of the invisible light relative to the display device is determined, and the display cursor is moved in real time based on the motion trajectory of the invisible light relative to the display screen. In addition, when mapping or moving the display cursor, the display screen can be manipulated based on the display cursor. For example, the moving display cursor can be used to move the contents of documents, pictures, etc. on the display screen. Alternatively, function keys and the moving display cursor can be combined to select a file in a folder and perform operations such as copying or deleting. Another example is to combine function keys and the moving display cursor to select part of the contents of an expanded document and perform operations such as copying or deleting. These will not be elaborated further here.

[0092] It should be noted that, in this embodiment, as described above, the remote control device may be equipped with function keys for function selection and triggering. These function keys may include a five-way navigation key, power key, number keys, movement keys, annotation keys, delete keys, paste keys, and capture keys, among others. Based on this, in this embodiment, the user can combine the function keys and the display cursor to interactively control the display device, performing operations such as dragging, selecting, scrolling, annotation, deleting (erasing), inserting, or zooming.

[0093] Optionally, the remote control device is equipped with multiple function buttons, including a control mode switching button. The control mode switching button is used to trigger a control mode selection event, so as to interactively control the display screen according to the control mode corresponding to the control mode selection event.

[0094] In this embodiment, according to the above description, the remote control device further includes function buttons ( Figure 2 (Not shown in the image), the function key can be a physical key or a virtual key. In this embodiment, the function key may include a control mode switching key, which the user can operate to switch the control mode of the remote control device. For example, the control mode may include remote control mode, indicator mode, and combined mode, so that the remote control device can enter one of the three modes.

[0095] Among them, the remote control mode can be a normal button remote control mode, which can realize operation actions such as display mode switching, channel switching, volume adjustment, and program selection; the indicator mode refers to the cursor indicator mode, which can be used for positioning and is particularly suitable for displaying the content of documents without damaging the eyes; however, for documents, it is not enough to just use the cursor to display the content. Generally, it is also necessary to perform display content switching operations such as selection and page scrolling, as well as display content editing operations such as annotation, deletion (erasure), insertion, or scaling. Therefore, a comprehensive mode that combines function keys and cursor indicators can be implemented to achieve a variety of complex functions.

[0096] In detail, the working principle of the remote control device after entering remote control mode is as follows:

[0097] The computing control module generates control commands based on the function control signals triggered by function keys other than the control mode switching button, and sends them to the display device through the first communication module. The display device receives the control commands through the second communication module and responds to the control commands through the internal processing module, performing control actions such as display mode switching, channel switching, volume adjustment, and program selection. It is worth noting that at this time, the invisible light emitting module, EVS camera module, and matching light source are not activated.

[0098] In detail, the working principle of the remote control device after entering the indication mode is as follows:

[0099] (1) The computing control module can send a first enable signal, a second enable signal and a third enable signal. The first enable signal enables the invisible light emitting module to start, the second enable signal enables the EVS camera module to start, and the display device receives and responds to the third enable signal to enable the matching light source to start. The matching light source emits light with intensity that changes over time.

[0100] (2) The user moves the remote control device to a suitable position and controls the remote control device to project the invisible light emitted by the invisible light emission module onto the display screen. The user also identifies and locates the matching light source and invisible light by collecting event images through the EVS camera module. At the same time, the calculation control module issues control commands to map the invisible light onto the display screen as a display cursor.

[0101] (3) Then, the remote control device is moved to drive the invisible light emitting module to move. Under the premise that the invisible light emitted by the display screen and the invisible light emitting module is always within the field of view of the EVS camera module, the EVS camera module collects multiple frames of continuous event images over a period of time. The position and trajectory of the invisible light relative to the matching light source are calculated by using the multiple frames of continuous event images. The position and trajectory of the invisible light relative to the display screen are then calculated by combining the relative positional relationship between the matching light source and the display screen.

[0102] (4) Finally, the calculation control module issues control commands to move the display cursor according to the motion trajectory of the invisible light relative to the display screen.

[0103] In detail, the working principle of the remote control device after entering the integrated mode is as follows:

[0104] (1) First, similar to the working principle of the above indication mode, the calculation control module sends an enable signal to enable the invisible light emitting module, EVS camera module and matching light source to start. The remote control device is moved to drive the invisible light emitted by the invisible light emitting module to move. The EVS camera module collects the continuous position change of the invisible light relative to the matching light source over a period of time and calculates the position and trajectory of the invisible light relative to the display screen. The calculation control module sends a control command to map the invisible light onto the display device as a display cursor and moves the display cursor according to the trajectory of the invisible light relative to the display screen.

[0105] (2) Secondly, more complex operations can be achieved by combining the display cursor and function keys, such as dragging, selecting, scrolling, marking, deleting (erasing), inserting or scaling.

[0106] Furthermore, in the comprehensive mode, the following operations can be performed on the document: the user can drag the document by pressing and holding the movement key on the remote control while moving the display cursor; the user can scroll up the document page by pressing and holding the up key, scroll down the document page by pressing and holding the down key, select a portion of the text content by pressing and holding the OK key while moving the display cursor, enlarge or bold the selected text content by pressing and holding the "+" key, shrink the selected text content by pressing and holding the "-" key, highlight the selected text content with background or text color using the annotation key, delete the content at the cursor position using the delete key, and enter content at the cursor position using the number keys.

[0107] Furthermore, in the comprehensive mode, the following operations can be performed on images: press and hold the movement key while moving the display cursor to drag the image; press and hold the up key to flip the image upwards, press and hold the down key to flip the image downwards; press and hold the OK key while moving the display cursor to select a portion of the image with a rectangle or any shape and crop it; move the display cursor and use the paste key to paste the cropped portion of the image; use the "+" key to zoom in on the cropped portion of the image, use the "-" key to zoom out on the cropped portion of the image; press and hold the annotation key while moving the display cursor to select a portion of the image with a rectangle or any shape and annotate it; use the delete key and move the display cursor to erase the image content; and use the number keys and move the display cursor to input text or numbers.

[0108] Furthermore, in the comprehensive mode, the following operations can be performed on the video: press and hold the movement key while moving the display cursor to drag the video; use the up key to rewind the video, and the down key to fast forward the video; press and hold the OK key to play or pause the video; while the video is playing, move the display cursor to select a video segment while pressing and holding the trim key to trim the video content; while the video is paused, move the display cursor relative to the display screen while pressing and holding the trim key to trim the current video frame; move the display cursor and use the paste key to paste and save the trimmed portion; press and hold the "+" key to increase the video volume, and press and hold the "-" key to decrease the video volume; use the delete key to delete the video at the cursor's indicated position.

[0109] Furthermore, in the comprehensive mode, the following audio controls can be performed: press and hold the movement key while moving the display cursor to drag the audio; use the up key to rewind the audio, and the down key to fast forward the audio; press and hold the OK key to play or pause the audio; while the audio is playing, move the display cursor to select a time segment while pressing and holding the trim key to trim the audio content; move the display cursor and use the paste key to paste and save the trimmed portion; use the "+" key to increase the audio volume, and the "-" key to decrease the audio volume; use the delete key to delete the audio at the cursor's indicated position.

[0110] It should be noted that in this embodiment, the operation actions in the comprehensive mode are flexible and diverse, and are not limited to the above-mentioned drag, select, scroll, mark, delete, insert or zoom operations. They can be flexibly selected and designed according to actual needs. The above are just examples of possible operations and are not limited here.

[0111] It is worth noting that in this embodiment, since the calibration parameters of the EVS camera module and the relative positions of the matching light source and the display screen are known, based on the acquired multi-frame continuous event images, computer vision algorithms and related positioning algorithms can be combined to analyze and determine the position and trajectory of the invisible light emitted by the invisible light emitting module relative to the display screen. Therefore, when moving the display cursor according to the trajectory of the invisible light relative to the display screen, only the projection of the three-dimensional spatial trajectory of the invisible light emitting module on the display screen is considered, representing its up, down, left, and right movement relative to the display screen (the XY plane where the display screen is located), without considering the forward and backward movement (Z-axis direction) or the change in distance between the invisible light emitting module and the display screen. In other words, if the projection of the three-dimensional spatial trajectory of the invisible light emitting module on the display screen remains unchanged and only the distance between the invisible light emitting module and the display screen changes, the corresponding display cursor will not move, but it can still be used for selection or zooming operations.

[0112] Optionally, in another alternative embodiment of this application, the above-mentioned "interactive control of the display screen based on position or motion trajectory" may further include:

[0113] Obtain operation instructions based on location or movement trajectory;

[0114] Interact with the display screen according to the operation instructions.

[0115] In this embodiment, when the remote control device moves relative to the display device, the remote control device controls the EVS camera module to capture event images at at least two locations. After calculating the position and trajectory of the invisible light relative to the display device based on the event images, the remote control device can obtain operation instructions based on the position or trajectory, and then perform interactive control of the display device based on the operation instructions.

[0116] It is worth noting that, as described in the above embodiments, various complex operations of the display device in the integrated mode require the combination of function keys and a display cursor. The corresponding remote control operations are relatively complex, requiring the pressing of various function keys while moving the remote control device. Therefore, this embodiment simplifies the remote control actions for various complex operations of the display device in the integrated mode, relying primarily or entirely on the movement of the display cursor to achieve complex control of the display device. That is, this embodiment can directly obtain the corresponding operation commands based on the aforementioned position or movement trajectory to operate the display device, achieving control without the need for function keys.

[0117] Specifically, for example, the working principle of a remote-controlled device after entering integrated mode can also be as follows:

[0118] (1) First, similar to the working principle of the indicator mode, the calculation control module sends an enable signal to enable the invisible light emitting module, the EVS camera module and the matching light source to start. The remote control device is moved to drive the invisible light emitted by the invisible light emitting module to move. The EVS camera module collects the continuous position change of the invisible light relative to the matching light source over a period of time and calculates the position and trajectory of the invisible light relative to the display screen. The calculation control module sends a control command to map the invisible light onto the display device as a display cursor and moves the display cursor according to the trajectory of the invisible light relative to the display screen.

[0119] (2) Secondly, based on the display cursor for indication, more complex operations can be achieved by special actions of remote control devices or invisible light emission modules, that is, more complex operations can be achieved by special actions of the display cursor, such as dragging, selecting, page scrolling, marking, deleting (erasing), inserting or scaling.

[0120] Thus, in comprehensive mode, the following complex operations can be performed on the document:

[0121] (2.1) First, move the remote control device to control the display cursor corresponding to the invisible light emitted by the invisible light emitting module to move to the document's location. While keeping the display cursor stationary, move the remote control device again to shorten the distance between the invisible light emitting module and the display screen twice in succession, thereby selecting the document.

[0122] (2.2) After selecting the document in (2.1) above, move the remote control device to make the display cursor move on the display screen, and then drag the selected document;

[0123] (2.3) After selecting the document in (2.1) above, move the remote control device to slide the display cursor from left to right to the right edge of the display screen to open the document;

[0124] (2.4) After opening the document in (2.3) above, move the remote control device so that the display cursor moves to the left side of the screen and slides from bottom to top on the left side of the screen, thereby flipping the document page upwards;

[0125] (2.5) After opening the document in (2.3) above, move the remote control device so that the display cursor moves to the left side of the screen and slides from top to bottom on the left side of the screen, thereby flipping the document page down;

[0126] (2.6) After opening the document in (2.3) above, move the remote control device so that the display cursor slides down from the top edge of the display screen twice in succession, thereby triggering the text selection function, and then move the remote control device again to select part of the text content of the document by displaying the cursor.

[0127] (2.7) After selecting part of the text content in (2.6) above, while keeping the display cursor stationary, move the remote control device to shorten the distance between the invisible light emitting module and the display screen, thereby enlarging the font of the selected part of the text content;

[0128] (2.8) After selecting part of the text content in (2.6) above, while keeping the display cursor stationary, move the remote control device to increase the distance between the invisible light emitting module and the display screen, thereby reducing the font size of the selected part of the text content;

[0129] (2.9) After selecting part of the text content in (2.6) above, move the remote control device to make the display cursor slide from the right edge of the display screen to the left, thereby deleting the selected part of the text content;

[0130] (2.10) First, move the remote control device to move the display cursor to the document's location. While keeping the display cursor stationary, move the remote control device again to increase the distance between the invisible light emitting module and the display screen twice in succession for document input. At this time, the display cursor becomes an electronic pen, and the remote control device writes characters on the display screen.

[0131] It is understandable that complex manipulations of files such as images, videos, and audio can be designed similarly, and will not be elaborated upon here; at the same time, the cursor display actions and corresponding complex manipulations can have more optional forms, which can be flexibly designed according to actual needs, and are not limited here.

[0132] Thus, this embodiment combines an invisible light emitting module, an EVS camera module, and a matching light source to design a remote control system based on an event camera. The system moves the remote control device and its invisible light emitting module while keeping the display device and its matching light source stationary. At this time, the EVS camera module can capture multiple consecutive event images over a period of time reflecting the relative position change of the invisible light emitted by the emitting module relative to the matching light source. Based on the known calibration parameters of the EVS camera module and the relative position of the matching light source and the display screen, the position and trajectory of the invisible light relative to the display screen can be calculated from the multiple consecutive event images. The incident point of the invisible light can then be mapped onto the display device as a display cursor, and the display cursor can be moved according to the position and trajectory of the invisible light relative to the display screen. Alternatively, the display screen can be manipulated based on the position and trajectory of the invisible light relative to the display screen.

[0133] The calibration parameters of the EVS camera module can be pre-calibrated before leaving the factory or calibrated temporarily, and there is no limitation here.

[0134] In detail, in an optional embodiment of this application, the EVS camera module can be calibrated in advance: the EVS camera module is calibrated using a black and white checkerboard pattern to obtain parameters such as the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the EVS camera module, and the parameters are saved for subsequent use.

[0135] Therefore, the overall workflow of the remote control system in this embodiment can also be as follows:

[0136] (1) The display device enters the working mode. After the display device is turned on, the internal control module of the display device controls the matching light source at the four corners of the screen to enter the strobe working mode, and at the same time controls the EVS camera module to enter the monitoring mode from idle.

[0137] (2) The invisible light emitting module interacts with the display screen. The EVS camera module detects the matching light sources at the four corners of the display screen and the incident point of the invisible light projected onto the display screen by the invisible light emitting module. It transmits the relevant image data to the computing control module. The computing control module converts and calculates the coordinates of the incident point of the invisible light in the display screen coordinate system based on the calibration parameters of the EVS camera module, the pixel coordinates of the matching light sources, and the pixel coordinates of the incident point of the invisible light projected onto the display screen by the invisible light emitting module. Based on the current content of the display screen, the position and movement trajectory of the invisible light relative to the display screen, the computing control module realizes the interactive operation between the user and the display screen, and performs cursor display and various complex control actions.

[0138] In summary, this application utilizes an invisible light emitting module and an EVS camera module on the remote control device, along with a matching light source and computer vision algorithms on the display device, to calculate the position and motion trajectory of the remote control device or the invisible light emitting module relative to the display screen. This achieves precise sensing and acquisition capabilities. This sensing method can be used for remote control interaction, and is particularly suitable for positioning indication or complex motion control. It can combine various different trajectories or actions to define multiple different control commands, enriching the interactive functions and improving the interactive efficiency, eliminating the need for numerous function buttons or sensing sampling devices. Furthermore, the incident point of the invisible light emitted by the invisible light emitting module is very small, while the display screen is relatively large. Based on the projection of invisible light, even the slightest movement (or rotation) of the remote control device or the invisible light emitting module can be effectively magnified and clearly presented on the display screen, and collected and recognized by the EVS camera module and the computing control module, effectively improving the sensitivity and accuracy of the sensing remote control. The EVS camera module has a small data volume, low power consumption, and low risk of leaking user privacy, ensuring the security of interactive data.

[0139] In one embodiment, this application also provides a control method based on an event camera. This method can be applied to a remote control device, which includes a transmitting module and an event camera. The remote control device is communicatively connected to a display device, which includes a display screen and a matching light source mounted on the display screen. The control method based on the event camera in this embodiment is as follows: Figure 4 As shown, it may include the following steps:

[0140] S10 emits invisible light whose intensity varies over time to the display screen via the emitting module;

[0141] S20, through an event camera, monitors the relative position of invisible light with respect to the matching light source and obtains an event image, wherein the matching light source emits light whose intensity changes over time;

[0142] S30, determine the relative position based on the event image, and determine the position or motion trajectory of the invisible light relative to the display screen based on the relative position;

[0143] S40 allows for interactive control of the display screen based on position or movement trajectory.

[0144] In this embodiment, the transmitting module in the remote control device can transmit invisible light with intensity varying over time to the display device to be controlled.

[0145] It should be noted that, in this embodiment, the remote control device may include a transmitting module, an event camera, and a computing control module. Figure 2 (not shown in the image), wherein the emitting module in this embodiment can specifically be an invisible light emitting module, which can emit invisible light and project it onto the display screen. It is understood that the invisible light emitting module can be a VCSEL infrared laser emitting module, which is not limited here.

[0146] Furthermore, the event camera in the remote control device can be controlled to monitor the position information of the invisible light relative to the matching light source on the display screen, obtain event images, analyze the light emitted by the matching light source and the invisible light reflected by the display screen in the event image, determine the relative position of the invisible light relative to the matching light source, and then determine the position or motion trajectory of the invisible light relative to the display screen based on the relative position, and interactively control the display screen based on the position or motion trajectory.

[0147] Optionally, in the above S40, "interactively controlling the display screen based on position or motion trajectory" may include:

[0148] S401 maps the incident point of invisible light onto the display screen as a display cursor based on its position, moves the display cursor according to its motion trajectory, and performs interactive control of the display screen based on the display cursor.

[0149] In this embodiment, after the remote control device controls the event camera to capture event images of the matching light source at at least two locations, for each frame of the event image, the calculation and control module in the remote control device can calculate the relative position and motion trajectory of the invisible light relative to the matching light source based on the event image. Then, combined with the relative position between the matching light source and the display screen, the position and motion trajectory of the invisible light relative to the display device can be calculated. Thus, based on the above position, the incident point of the invisible light can be mapped onto the display device as a display cursor. The display cursor can be moved according to the above position and motion trajectory, and the display device can be interactively controlled based on the display cursor.

[0150] It should be noted that, in this embodiment, as described above, the remote control device is equipped with function buttons for function selection and triggering, allowing the user to operate the function buttons and move the transmitting module to perform more complex control operations on the display device. Thus, the remote control device can perform more complex interactive control of the display device based on the function buttons and the display cursor, as described in the above embodiment, and will not be repeated here.

[0151] Optionally, in the above S40, "interactively controlling the display screen based on position or motion trajectory" may further include:

[0152] S402, obtain operation instructions based on position or movement trajectory;

[0153] S403 allows for interactive control of the display screen based on operating instructions.

[0154] In this embodiment, the remote control device controls the event camera to capture event images at at least two locations. Based on the light emitted by the matching light source in the event image and the invisible light reflected by the display screen, the position and trajectory of the invisible light relative to the display device are calculated. Based on the position or trajectory, an operation command is obtained, and then the display device is interactively controlled according to the operation command.

[0155] It is worth noting that in this embodiment, the remote control device can directly obtain the corresponding operation instructions to operate the display device based on the above position or movement trajectory. Complex operation of the display device can be achieved without the need for function buttons. Referring to the description of the above embodiment, it will not be repeated here.

[0156] Optionally, before the above-described S10 step of "emitting invisible light whose intensity varies with time to the display screen via the emission module", the following may also be included:

[0157] S50 triggers the control mode selection event;

[0158] S60, responds to the control mode selection event to cause the remote control device to enter the target control mode and perform corresponding interactive control;

[0159] If the target control mode is the first control mode, then during interactive control, the function control signal is triggered first, then the function control signal is responded to, the corresponding operation command is generated and sent to the display device to control the display device;

[0160] If the target control mode is the second control mode, then during interactive control, the transmitting module and the matching light source are activated, so that the transmitting module emits invisible light with intensity varying over time to the display screen, and the matching light source emits light with intensity varying over time. The event camera is also activated to monitor the relative position of the invisible light with respect to the matching light source and obtain event images.

[0161] It should be noted that, in this embodiment, according to the above description, the function keys of the remote control device may include a control mode switching key. In this way, the user can operate the function key to switch the control mode of the remote control device. For example, the control mode may include remote control mode, indicator mode and comprehensive mode, so that the remote control device can enter one of the remote control mode, indicator mode and comprehensive mode.

[0162] Based on this, the remote control device can respond to the control mode selection event triggered by the user's operation of the remote control device, so that the remote control device enters the target control mode and performs corresponding interactive control. The target control mode can be one of the following: remote control mode, instruction mode, and integrated mode.

[0163] Furthermore, if it is determined that the target control mode is the first control mode (i.e., the remote control mode in this embodiment), the function control signal triggered by the user based on the remote control device can be obtained, and the operation command corresponding to the function control signal can be sent to the display device to control the display device. For example, the video content, audio content, and document content of the display device can be controlled. It is worth noting that at this time, the invisible light emitting module, the EVS camera module, and the matching light source are not started.

[0164] If the target control mode is determined to be the second control mode (i.e., the indication mode or the comprehensive mode in this embodiment), the event camera in the remote control device can be activated, and the matching light source on the display device can be activated, so that the matching light source outputs invisible light with varying intensity. The matching light source can emit invisible light with varying intensity over time to the display device. In this way, the remote control device can control the activated event camera to monitor the relative position of the invisible light with respect to the matching light source and obtain event images.

[0165] In general, such as Figure 5 As shown, the control method based on the event camera in this embodiment may include:

[0166] Step 1: Trigger the matching light source on the display device to emit light with intensity varying over time, and trigger the invisible light emitting module and EVS camera module on the remote control device.

[0167] Step 2: Move the remote control device to move the invisible light emitted by the invisible light emitting module, and monitor the relative position change of the invisible light emitted by the invisible light emitting module relative to the matching light source over a period of time through the EVS camera module on the remote control device to obtain multiple frames of continuous event images.

[0168] Step 3: Based on multiple consecutive event images, the calibration parameters of the EVS camera module, the relative positions of the matching light source and the display screen, computer vision algorithms are used to calculate the position and motion trajectory of invisible light relative to the display screen.

[0169] Step 4: Map the incident point of the invisible light onto the display screen as a display cursor, and move the display cursor according to the position and trajectory of the invisible light relative to the display screen;

[0170] Step 5: Control the display screen based on the position and trajectory of the invisible light relative to the display screen.

[0171] Since the principle of the above-mentioned event camera-based control method is similar to that of the aforementioned event camera-based remote control system, the same technical details will not be repeated. The detailed steps of the above method can be analyzed with reference to the aforementioned event camera-based remote control system.

[0172] This embodiment implements precise sensing and data acquisition based on an event camera, applying it to remote control interaction. Changes in position and motion trajectory are particularly suitable for positioning indication or complex action control. Multiple different trajectories or actions can be combined to define various control commands, enriching interactive functions and improving interaction efficiency, eliminating the need for numerous function buttons or sensing sampling devices. Simultaneously, the incident point of the invisible light emitted by the invisible light emitting module is very small, while the display screen size is relatively large. Based on the projection of invisible light, even slight movements (or rotations) of the remote control device or the invisible light emitting module can be effectively magnified and clearly presented on the display screen and collected and recognized by the EVS camera module and the computing control module, effectively improving the sensitivity and accuracy of sensor-based remote control. The EVS camera module also features small data volume, low power consumption, and a low risk of user privacy leakage.

[0173] Accordingly, embodiments of this application also provide an electronic device, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of an electronic device 1100 provided in an embodiment of this application. The electronic device 1100 includes a processor 1101 with one or more processing cores, a memory 1102 with one or more computer-readable storage media, and a computer program stored on the memory 1102 and executable on the processor. The processor 1101 and the memory 1102 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0174] The processor 1101 is the control center of the electronic device 1100. It connects various parts of the electronic device 1100 via various interfaces and lines. By running or loading software programs and / or units stored in the memory 1102, and by calling data stored in the memory 1102, it executes various functions and processes data of the electronic device 1100, thereby providing overall monitoring of the electronic device 1100. The processor 1101 can be a processor (Central Processing Unit, CPU), graphics processing unit (GPU), network processor (NP), etc., and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0175] In this embodiment, the processor 1101 in the electronic device 1100 loads the instructions corresponding to the processes of one or more applications into the memory 1102 according to the following steps, and the processor 1101 runs the applications stored in the memory 1102 to realize various functions, such as:

[0176] The emission module emits invisible light whose intensity varies over time onto the display screen;

[0177] An event camera monitors the relative position of invisible light with respect to a matching light source to obtain event images, in which the matching light source emits light with intensity varying over time.

[0178] The relative position is determined based on the event image, and the position or trajectory of the invisible light relative to the display screen is determined based on the relative position.

[0179] Interactive control of the display screen based on location or movement trajectory.

[0180] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0181] Optional, such as Figure 6 As shown, the electronic device 1100 also includes: a touch display screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch display screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107. Those skilled in the art will understand that... Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0182] The touch display screen 1103 can be used to display a graphical user interface and receive operation commands generated by the user interacting with the graphical user interface. The touch display screen 1103 may include a display panel and a touch panel.

[0183] The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces (GUIs) of electronic devices. These GUIs can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using liquid crystal displays (LCDs), organic light-emitting diodes (OLEDs), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands.

[0184] Optionally, the touch panel may include two parts: a touch display system and a touch controller. The touch display system detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch display system, converts it into touch point coordinates, and sends it to the processor 1101. It can also receive and execute commands from the processor 1101. The touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 1101 to determine the type of touch event. Subsequently, the processor 1101 provides corresponding visual output on the display panel based on the type of touch event.

[0185] In this embodiment, the touch panel and display panel can be integrated into the touch display screen 1103 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 1103 can be used as two independent components to achieve input and output functions. That is, the touch display screen 1103 can also be used as part of the input unit 1106 to achieve input functions.

[0186] The radio frequency circuit 1104 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0187] Audio circuit 1105 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 1105 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 1105, converted back into audio data, and then processed by processor 1101 before being transmitted via radio frequency circuit 1104 to, for example, another electronic device, or output to memory 1102 for further processing. Audio circuit 1105 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0188] The input unit 1106 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0189] Power supply 1107 is used to supply power to various components of electronic device 1100. Optionally, power supply 1107 can be logically connected to processor 1101 through a power management device, thereby enabling functions such as charging, discharging, and power consumption management through the power management device. Power supply 1107 may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0190] although Figure 6 As not shown in the diagram, the electronic device 1100 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0191] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0192] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0193] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple computer programs. These computer programs can be loaded by a processor to implement any of the event-based camera control methods provided in this application. When the computer program is loaded and executed, it can implement the following steps of the event-based camera control method:

[0194] The emission module emits invisible light whose intensity varies over time onto the display screen;

[0195] An event camera monitors the relative position of invisible light with respect to a matching light source to obtain event images, in which the matching light source emits light with intensity varying over time.

[0196] The relative position is determined based on the event image, and the position or trajectory of the invisible light relative to the display screen is determined based on the relative position.

[0197] Interactive control of the display screen based on location or movement trajectory.

[0198] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0199] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0200] Since the computer program stored in the computer-readable storage medium can execute any of the event camera-based control methods provided in the embodiments of this application, the beneficial effects that any of the event camera-based control methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0201] In the above descriptions of the event camera-based control method, electronic device, computer-readable storage medium, and computer program product, each embodiment has its own emphasis. Parts not detailed in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the event camera-based control method, electronic device, computer-readable storage medium, computer program product, and their corresponding units described above can be referred to the description of the event camera-based remote control system in the above embodiments, and will not be repeated here.

[0202] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A remote control system based on an event camera, characterized in that, The system includes: Display devices, and remote control devices; The display device includes a display screen and a matching light source disposed on the display screen; The remote control device includes a transmitting module, an event camera, and a computing control module; wherein, The transmitting module is used to emit invisible light to the display screen; The event camera is used to monitor the relative position of the invisible light with respect to the matching light source and to obtain event images; The calculation and control module is used to determine the relative position based on the event image, and also to determine the position or motion trajectory of the invisible light relative to the display screen based on the relative position, and to perform interactive control of the display screen based on the position or the motion trajectory.

2. The system according to claim 1, characterized in that, The light emitted by the matching light source and the invisible light are both light whose intensity changes with time. The light emitted by the matching light source and the invisible light reflected by the display screen are both incident on the event camera to form the event image. The emitting module is a freely movable emitting module so that the invisible light can move freely relative to the display screen, and the event camera can obtain multiple consecutive frames of the event image.

3. The system according to claim 2, characterized in that, The step of interactively controlling the display screen based on the position or the motion trajectory includes: The invisible light incident point is mapped onto the display screen as a display cursor according to the position, the display cursor is moved according to the motion trajectory, and the display screen is interactively controlled based on the display cursor.

4. The system according to claim 3, characterized in that, The step of interactively controlling the display screen based on the position or the motion trajectory includes: Based on the location or the movement trajectory, obtain the operation command; The display screen is interactively controlled according to the operation instructions.

5. The system according to any one of claims 1-4, characterized in that, The remote control device is equipped with multiple function buttons, including a control mode switching button. The control mode switching button is used to trigger a control mode selection event, so as to interactively control the display screen according to the control mode corresponding to the control mode selection event.

6. A control method based on an event camera, characterized in that, The method is applied to a remote control device, the remote control device including a transmitting module and the event camera, the remote control device being communicatively connected to a display device, the display device including a display screen and a matching light source disposed on the display screen, the method comprising: The emission module emits invisible light whose intensity varies over time into the display screen; The event camera monitors the relative position of the invisible light with respect to the matching light source to obtain an event image, wherein the matching light source emits light with intensity varying over time. The relative position is determined based on the event image, and the position or trajectory of the invisible light relative to the display screen is determined based on the relative position. The display screen is interactively controlled based on the location or the movement trajectory.

7. The method according to claim 6, characterized in that, The step of interactively controlling the display screen based on the position or the motion trajectory includes: The invisible light incident point is mapped onto the display screen as a display cursor according to the position, the display cursor is moved according to the motion trajectory, and the display screen is interactively controlled based on the display cursor.

8. The method according to claim 6, characterized in that, The step of interactively controlling the display screen based on the position or the motion trajectory includes: Based on the location or the movement trajectory, obtain the operation command; The display screen is interactively controlled according to the operation instructions.

9. The method according to claim 6, characterized in that, Before emitting invisible light of varying intensity over time to the display screen via the emission module, the method further includes: Trigger the control mode selection event; In response to the control mode selection event, the remote control device enters the target control mode and performs corresponding interactive control. If the target control mode is the first control mode, then during interactive control, a function control signal is first triggered, and then the function control signal is responded to to generate a corresponding operation instruction and send the operation instruction to the display device to control the display device. If the target control mode is the second control mode, then during interactive control, the transmitting module and the matching light source are activated, so that the transmitting module emits invisible light with intensity varying over time to the display screen, and the matching light source emits light with intensity varying over time. The event camera is also activated to monitor the relative position of the invisible light with respect to the matching light source and obtain the event image.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the electronic device to perform the steps of the method as described in any one of claims 6 to 9.

11. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed, implements the steps of the method as described in any one of claims 6 to 9.

Images