Event-based sensor projection direction calibration method and system

Abstract

The invention discloses an event-based sensor projection pointing calibration method and system, and belongs to the technical field of human-computer interaction. In order to solve the problem that deviation exists between pointing point pixel coordinates and actual screen coordinates in an existing projection pointing scheme, the method comprises the following steps: playing a calibration video containing a calibration pattern, and shooting and acquiring an event stream by using an event-based sensor; processing the event stream, and determining the range of a projection picture in a sensor pixel coordinate system; determining calibration parameters according to the range and the resolution of the projection picture; and during pointing operation, calling the calibration parameters, and mapping the pixel coordinates of the pointing point into the actual pointing coordinates of the projection picture. The system comprises a projector, an event-based sensor and a processing unit, wherein the processing unit executes the method. According to the invention, accurate coordinate mapping is realized, the pointing deviation is effectively eliminated, and the projection pointing precision and the user experience are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically to a method and system for calibrating the projection direction of an event-based sensor. Background Technology

[0002] Projection pointing technology is an emerging human-computer interaction method. It uses visual sensors to capture the user's pointing motion in front of the projected image, calculates the coordinates of the pointing point on the screen, and thus achieves interactive control. Compared with traditional interaction methods such as touch screens and laser pointers, projection pointing technology has advantages such as being contactless, having a large screen, and being highly portable, and has broad application prospects in fields such as smart conferencing, education and training, and exhibitions.

[0003] Event-based sensors (EBS) are a novel type of biomimetic vision sensor. Their core consists of dynamic visual pixels that sense changes in light intensity and output event signals. Examples include dynamic vision sensors (DVS, also known as event cameras) and hybrid sensors (such as DAVIS) that integrate both DVS pixels and active pixel sensors (APS). These sensors monitor continuous changes in light intensity within their field of view, asynchronously outputting events when the change exceeds a preset threshold, forming an event stream. Compared to traditional frame cameras, event-based sensors offer significant advantages such as microsecond-level temporal resolution, high dynamic range, extremely low data redundancy, and low power consumption. Therefore, they have been introduced into projection pointing systems to capture the rapid movements of pointing devices, significantly improving pointing response speed and tracking accuracy.

[0004] In the prior art, there are already systems and devices that utilize event-based sensors to achieve projection interaction.

[0005] Prior art 1: CN120529053A, pointing method and interactive device, pointed-to device.

[0006] Prior art 1 discloses a novel projection pointing scheme, but this type of scheme does not consider or disclose how to correct the deviation (not exactly overlapping) between the field of view of the event-based sensor and the image (projected image) generated by the projection optical engine.

[0007] In event-based sensor-based projection pointing systems, the coordinate calculation of the pointing point relies on event stream information collected by the sensor. Due to the different physical installation positions of the sensor and the projected image, unless an absolutely fixed geometric relationship is used, differences inevitably exist between the sensor's field of view and the projected image. This leads to an unavoidable mapping deviation between the pixel coordinates of the pointing point generated based on the event stream and the actual pointing coordinates on the projected image. If this deviation is not corrected, it will directly affect the accuracy of pointing and the user experience, becoming a key issue restricting the practical application of this technology.

[0008] In existing technologies, common calibration methods are mostly based on capturing calibration patterns such as checkerboard patterns or dot arrays using traditional frame cameras, and calculating mapping parameters through corner detection and coordinate transformation. However, since event-based sensor output captures the calibration patterns in existing technologies, the intended calibration purpose cannot be achieved.

[0009] Therefore, there is an urgent need for a calibration method for projection pointing systems based on event-based sensors that can solve the problem of accurately mapping the pixel coordinates of the pointing point to the pointing coordinates of the projected image. Summary of the Invention

[0010] To alleviate or partially alleviate the above-mentioned technical problems, the solution of the present invention is as follows:

[0011] On one hand, this invention discloses a projection pointing calibration method for an event-based sensor, comprising the following steps:

[0012] Play a calibration video, which includes a calibration pattern with periodic brightness changes formed on a projected screen;

[0013] The calibration video is captured using an event-based sensor to obtain an event stream;

[0014] The event stream is processed to determine the range of the projected image in the pixel coordinate system of the event-based sensor;

[0015] The calibration parameters are determined based on the range of the projected image in the pixel coordinate system and the resolution of the projected image.

[0016] The calibration parameters are used to map the coordinates of the pointing point in the pixel coordinate system to the actual pointing coordinates on the projected screen during pointing operations.

[0017] In one embodiment, the calibration pattern is set in the boundary area of ​​the projected image and is presented by alternating black and white to indicate the boundary and / or center point of the projected image.

[0018] In one embodiment, processing the event stream includes:

[0019] Generate a perceptual image that reflects the boundary and / or center point of the calibration pattern to determine the range of the projected image in the pixel coordinate system.

[0020] In one embodiment, processing the event stream includes:

[0021] The event stream is compressed into frames to accumulate events within a time window and generate event frames. Image analysis is performed on the event frames to extract the boundaries of the calibration pattern, thereby determining the boundaries and / or center point of the projected image in the pixel coordinate system.

[0022] In one embodiment, the calibration parameters include mapping coefficients calculated based on the range of the projected image in the pixel coordinate system and the resolution of the projected image.

[0023] In one type of embodiment, it further includes:

[0024] Event data is continuously collected over multiple time windows, generating multiple event frames and calculating multiple sets of calibration parameters;

[0025] The average value of the multiple sets of calibration parameters is calculated or the average value is taken after removing outliers, and the obtained average value is used as the final determined calibration parameter.

[0026] In one embodiment, mapping the pixel coordinates of the pointing point to the actual pointing coordinates on the projected image includes:

[0027] A screen coordinate system is established with the predetermined corner point of the projected image as the origin. The actual pointing coordinates are calculated based on the calibration parameters and the offset of the pointing point in the pixel coordinate system relative to the boundary of the projected image in the pixel coordinate system.

[0028] In one embodiment, when processing the event stream, an edge detection algorithm combined with Hough transform or contour search algorithm is used to extract the boundary of the calibration pattern.

[0029] On the other hand, the present invention discloses an event-based sensor projection pointing system, comprising:

[0030] A projector used to play calibration videos containing calibration patterns on a projected screen;

[0031] An event-based sensor is used to capture the projected image and acquire an event stream;

[0032] A processing unit for performing the event-based sensor projection pointing calibration method as described in the preceding item.

[0033] In one embodiment, the optical path of the event-based sensor is configured with an infrared filter and an infrared switch;

[0034] Before playing the calibration video, the infrared filter is removed by the infrared switcher;

[0035] After calibration, the infrared filter is reset via the infrared switch.

[0036] The technical solution of the present invention has one or more of the following beneficial technical effects:

[0037] (1) It realizes the accurate mapping from the pixel coordinates of the pointing point to the actual pointing coordinates of the projected screen, effectively eliminating the pointing deviation caused by the difference between the sensor and the screen coordinate system, and significantly improving the positioning accuracy and user experience.

[0038] (2) The calibration process is simple and efficient, the calibration pattern is flexible and can be adapted to different hardware configurations and application scenarios. It supports recalibration at any time to maintain the long-term stability of the mapping accuracy.

[0039] (3) The system is highly robust. By combining the infrared switch design with the high dynamic range characteristics of the event sensor, it effectively suppresses ambient light interference and ensures the accuracy and reliability of the calibration results.

[0040] Furthermore, other beneficial effects of the present invention will be mentioned in the specific embodiments. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the infrared filter position when the projection pointing system is working normally in one embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the infrared filter position during the calibration operation of the projection pointing system in one embodiment of the present invention;

[0043] Figure 3 This is the calibration pattern used in one embodiment of the present invention;

[0044] Figure 4 This is an example of the imaging of a calibration pattern in a DVS according to one embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram illustrating the relationship between the sensor's field of view and the projected image range based on the event-based method of this invention;

[0046] Figure 6 This is a schematic diagram of the coordinate mapping of a pointing point in one embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order.

[0049] The term "pixel coordinate system" refers to a two-dimensional coordinate space defined by the pixel array of an event-based sensor, where each pixel has unique coordinates (usually represented by row and column indices). Each event output by an event-based sensor contains the pixel coordinates of its occurrence location, which belong to this pixel coordinate system.

[0050] "Projected screen coordinate system": This refers to the two-dimensional coordinate space defined by the resolution of the image signal output by the projector. For example, for a projected screen with a resolution of 1920x1080, its coordinate system ranges from x=0 to 1919, and y=0 to 1079. The core objective of calibration is to establish a precise mapping relationship between the pixel coordinate system of the event-based sensor and the projected screen coordinate system.

[0051] The term "perceptual image" refers to an image representation generated by processing an event stream that reflects the boundaries and / or center point of a calibration pattern. Specific generation methods include, but are not limited to, accumulating events within a time window to generate event frames; performing spatiotemporal filtering on events to generate an event density map to reflect the spatial distribution density of events; or directly extracting boundary features from the event stream using edge detection algorithms to reconstruct an edge feature map reflecting the contour of the calibration pattern. Regardless of the specific form used, the perceptual image should clearly reflect the boundary position and / or center point position of the projected image in the pixel coordinate system of the event-based sensor, thus providing an image analysis basis for subsequently determining the range of the projected image in that pixel coordinate system. Unlike images output by traditional frame cameras, perceptual images are not generated by direct sensor exposure but are image representations reconstructed through post-processing of asynchronous event streams.

[0052] This invention provides a calibration method for projection pointing based on an event-based sensor. This method aims to solve the problem of deviation between the pixel coordinates of the pointing point and the actual pointing coordinates on the projected screen due to the difference between the coordinate systems of the event-based sensor and the projected screen. Through a complete calibration process, a precise mapping relationship is established between the two, thereby improving the accuracy of projection pointing and the user experience.

[0053] The present invention will exemplify the use of DVS as an event-based sensor in the following description, and will use this example to illustrate the technical concept of the invention. Those skilled in the art will understand that the scheme described below is also applicable to hybrid sensors such as DAVIS that simultaneously include DVS pixels and APS pixels. Each event output by the DVS carries its coordinates in the pixel coordinate system.

[0054] Figure 1 This is a schematic diagram showing the position of the infrared filter when the projection pointing system is working normally in one embodiment of the present invention. As shown in the figure, in the normal pointing operation state, an infrared (bandpass) filter is installed in front of the lens of the event-based sensor. This filter is used to allow only infrared light or light in the nearby band of the incident light to pass through.

[0055] Figure 2 This diagram illustrates the position of the infrared filter during calibration of the projection pointing system in one embodiment of the present invention. As shown, when the system needs to be calibrated, the infrared filter is removed via an infrared switch (IR-cut), putting the event-based sensor in a "naked-eye" state. At this time, the event-based sensor exhibits high sensitivity to a wide spectrum of light, including the infrared band, to clearly capture the calibration pattern projected by the projected image during subsequent calibration. This design fully utilizes the high dynamic range of the event-based sensor while avoiding the adverse effects of the infrared filter on the imaging quality of the calibration pattern.

[0056] The core design concept of the calibration pattern used in this invention lies in utilizing the sensitivity of event-based sensors to brightness changes. Periodic brightness variations, stably perceptible by the event-based sensor, are generated by pixels on the projected image, thereby forming spatiotemporal features in the event data that characterize the screen boundaries and positions. Therefore, any pattern design capable of achieving the above function can be used as the calibration pattern of this invention, without being limited to a specific form. For example, it could be a uniform flashing pattern across the entire screen, directly extracting the screen outline through event density map segmentation; it could also be a scanning moving light strip, determining the boundary position by tracking the moment the light strip reaches the screen edge; or it could be setting markers with different flashing frequencies or encoded sequences at specific locations on the screen, identifying the coordinates of feature points through frequency analysis or decoding, and then establishing a mapping relationship through perspective transformation. These alternative patterns can all achieve the calibration purpose of determining the boundaries and center points of the projected image in the pixel coordinate system of the event-based sensor.

[0057] It is worth mentioning that the "periodic brightness change" mentioned in this invention has a specific technical meaning, which is different from the inherent refresh characteristics of projection devices: this change is a calibration signal actively designed for calibration purposes. For example, by using an alternating black and white switching method in the calibration pattern, a full-range jump from the lowest brightness to the highest brightness is generated at the pixel point, thereby deliberately constructing detectable spatiotemporal features in the event data to identify the boundaries of the projected image; while the refresh rate of the projected image is an inherent physical characteristic of the display device, which is used to realize the continuous display of image content, and the brightness change of the same pixel point during the refresh process is usually weak and gradual, far below the trigger threshold of the event-based sensor, so it cannot effectively trigger event output, let alone extract any meaningful boundary information from it.

[0058] Specifically, the so-called periodic pixel brightness variation in this invention does not originate from or differ from the brightness variation caused by the periodic refresh rate of the projected image (common FPS values ​​are 24Hz, 30Hz, 60Hz, etc.). For example, in a projection system with an FPS of 30Hz, for a pixel displaying a specific brightness value for 0.5 seconds, there is no periodic pixel brightness variation as described in this invention during those 0.5 seconds; instead, the pixel is continuously lit at the specific brightness value. After 0.5 seconds, the brightness value of the pixel may change to another brightness value (for example, 3% less than the aforementioned specific brightness value). The light emission process of this pixel does not conform to the periodic pixel brightness variation described in this invention.

[0059] Figure 3This is a calibration pattern used in one embodiment of the present invention. As shown in the figure, the calibration video includes a black-and-white alternating calibration pattern specifically designed for event-based sensor imaging. Specifically, the calibration pattern is distributed at the top, bottom, left, and right boundaries of the projected image, and the pattern alternates between black and white at a predetermined frequency. The purpose of this design is that event-based sensors are sensitive to changes in brightness, and the black-and-white alternating pattern can trigger stable and clear event signals on the pixel plane of the event-based sensor, thereby enabling the position of the screen boundary to be accurately identified in the event data.

[0060] Figure 4 This is an example of imaging a calibration pattern in a DVS (Digital Sensor Display) according to one embodiment of the present invention. After removing the infrared filter, the calibration video projected onto the screen is captured using the DVS, and the image is obtained after frame compression processing of the event data. As shown in the figure, a clear white rectangular frame is presented in the image. This rectangular frame is the imaging result of the calibration pattern in the DVS, and its boundary corresponds to the range of the projected image in the sensor pixel coordinate system. Through image processing algorithms, the four boundary positions and the coordinates of the center point of this white rectangular frame can be accurately extracted, thereby determining the coverage area of ​​the projected image in the sensor pixel coordinate system.

[0061] It should be noted that the "calibration video" referred to in this invention is a dynamic image sequence composed of a series of continuous video frames played on a projection screen, its function being to present the calibration pattern on the projection screen. In other words, the calibration video is the carrier of the calibration pattern, and the calibration pattern is the content of the calibration video. By playing the calibration video, the calibration pattern can be displayed on the screen according to a predetermined timing sequence (such as a specific frequency, order, or scanning path), thereby providing a stable and controllable source of brightness variation for event-based sensors.

[0062] The technical solution of this invention assumes that the system has been correctly deployed. That is, before implementing the calibration method, reasonable hardware selection and installation (such as choosing a suitable lens and installation position) ensures that the field of view of the event-based sensor can cover the entire projected image. Furthermore, as a more complete implementation, this invention can also introduce a field of view pre-check step into the calibration process. By analyzing a full-screen reference image covering the entire projected image, the field of view coverage is determined, and the user is guided to make adjustments. For special scenarios such as ultra-large screens, multiple event-based sensors can be deployed to work collaboratively to ensure full coverage. This invention provides measures to ensure that the field of view of the event-based sensor can completely capture the calibration pattern, thereby ensuring the reliable implementation of the calibration method.

[0063] Based on the above hardware configuration and calibration pattern, the specific implementation process of one embodiment of the present invention is as follows.

[0064] In one embodiment, the system controls an infrared switcher (IR-cut) to remove the infrared filter in front of the event-based sensor lens, allowing the event-based sensor to enter broadband imaging mode. Simultaneously, a control unit connected to a projector begins playing a preset calibration video. This calibration video contains alternating black-and-white calibration patterns designed for the event-based sensor, located in the four boundary regions of the projected image, and switching between black and white at a fixed frequency (e.g., 30Hz, 60Hz, or other suitable frequencies).

[0065] Event-based sensors, through their inherent working mechanism, continuously sense changes in brightness within their field of view. When a calibration pattern projected onto the screen switches between black and white, pixels at the edges of the pattern experience significant brightness changes. If this change exceeds a preset threshold for the event-based sensor, it triggers the sensor to asynchronously output an event. These events collectively form an event stream, which is transmitted in real-time to the processing unit connected to the event-based sensor.

[0066] It should be noted that the purpose of processing the event stream is to generate a perceptual image that reflects the boundaries and / or center point of the calibration pattern, in order to determine the range of the projected image in the pixel coordinate system of the event-based sensor. Specific implementation methods include, but are not limited to: accumulating events over a period of time into event frames; performing spatiotemporal filtering on the events to generate an event density map; or directly extracting boundary features from the event stream using an edge detection algorithm. Regardless of the method used, the final generated perceptual image should clearly reflect the precise position of the projected image in the sensor's pixel coordinate system (i.e., boundary coordinates and center point coordinates).

[0067] In one embodiment, after receiving the original event stream, the processing unit first performs frame compression processing. Frame compression processing refers to accumulating events within a time window (e.g., a complete black-and-white switching cycle) into a single two-dimensional image frame. Specifically, for each event in the event stream, its corresponding pixel coordinates are counted or weighted to generate an image reflecting the event distribution density and intensity within that time window, i.e., an event frame. Due to the periodic switching of the calibration pattern, events corresponding to the boundary areas of the projected image will appear as continuous high-brightness areas in the event frame, while background noise events, lacking spatiotemporal consistency, will appear as scattered points.

[0068] Subsequently, the processing unit performs image analysis on the generated event frames to determine the precise location of the projected image boundaries in the sensor pixel coordinate system. Edge detection algorithms, such as the Canny edge detection operator, combined with Hough transform or contour finding algorithms, can be used to extract the four boundaries of the high-brightness regions in the event frames. Since the calibration pattern is designed to be located at the four boundaries of the projected image, these four extracted boundaries represent the imaging boundaries of the projected image in the event-based sensor. Based on the obtained boundaries of the projected image in the sensor pixel coordinate system, the sensor pixel dimensions occupied by the projected image in the horizontal and vertical directions can be calculated for subsequent calibration parameter calculations.

[0069] Figure 5 This is a schematic diagram illustrating the relationship between the sensor's field of view and the projected image range based on the event-driven nature of this invention. As shown in the figure, since the calibration video is played by a projector, the resolution of the projected image (e.g., horizontal resolution W) is... p Vertical resolution H p The extent of the projected image in the sensor pixel coordinate system was obtained through image analysis; for example, its horizontal occupancy was W. s Each sensor pixel occupies H vertically. s Each sensor pixel. Based on these two sets of data, the mapping relationship from the event-based sensor's pixel coordinate system to the projected image coordinate system can be directly calculated. For example, the mapping coefficients in the horizontal and vertical directions can be calculated:

[0070] k x = W p / W s k y = H p / H s

[0071] This mapping coefficient is the calibration parameter we are looking for. It describes the distance a pixel moves in the sensor pixel coordinate system in the horizontal or vertical direction, corresponding to how many pixels it moves in the projection screen coordinate system. Using this parameter, the coordinates of any pointing point detected in the sensor pixel coordinate system can be accurately mapped to its target pixel coordinates on the projection screen.

[0072] To improve the robustness and accuracy of the calibration parameters, the above process can be repeated multiple times. For example, event data from multiple black-and-white switching cycles can be continuously collected to generate multiple event frames, and the calibration parameters can be calculated for each frame. Since occasional noise interference or ambient light fluctuations may occur during the calibration process, resulting in slight deviations in the calculation results of a single calculation, the results obtained from multiple calculations can be filtered and fused. A preferred implementation is to calculate the average of the multiple results, or to take the average after removing outliers with large deviations, and save the final average as the calibration parameter in the system's non-volatile memory for subsequent normal pointing operations.

[0073] After the calibration parameters are calculated and saved, the system controls the infrared switch (IR-cut) to move the infrared filter back in front of the event-based sensor lens, so that the event-based sensor returns to its normal pointing working state.

[0074] When a user uses the projection pointing function, the system retrieves the pre-saved calibration parameters. Figure 6 This is a schematic diagram of the pointing point coordinate mapping in one embodiment of the present invention. As shown in the figure, when the event-based sensor captures the pointing point of the laser emitter within the projected image area and outputs an event stream, the processing unit performs clustering and centroid calculation on the corresponding events generated by the pointing point to obtain the real-time pixel coordinates (x, y, x) of the pointing point in the pixel coordinate system of the event-based sensor. pixel ,y pixel Then, based on the mapping coefficients in the calibration parameters, the pixel coordinates are mapped to the actual pointing coordinates (x, y) on the projected image. pointer , y pointer Specifically, a projection screen coordinate system can be established with a predetermined corner point (e.g., the upper left corner) of the projection screen as the origin. By calculating the boundary offset of the pointing point pixel coordinates relative to the projection screen in the sensor pixel coordinate system and multiplying it by the mapping coefficients in the horizontal and vertical directions, the accurate actual pointing coordinates can be obtained.

[0075] Through the above process, the present invention achieves precise mapping from the pixel coordinates of the pointing point to the pointing coordinates of the projected image, effectively eliminating pointing deviations caused by differences between the sensor and screen coordinate systems. In the above embodiments, the playback of calibration video, the acquisition and processing of event data, the calculation and storage of calibration parameters, and the control of the infrared switcher can all be uniformly coordinated and completed by the same processing unit (such as a microcontroller, embedded processor, or host computer), ensuring the automation and timing accuracy of the entire calibration process.

[0076] This embodiment solves the problem of insufficient infrared response in the calibration process of event-based sensors by cleverly designing an infrared switcher. By designing an alternating black and white calibration pattern suitable for event-based sensors, the screen boundary can be stably and clearly identified in the event data. Through frame compression processing and image analysis algorithms, the position of the screen boundary in the pixel coordinate system of the event-based sensor is accurately extracted. Finally, by calculating the mapping ratio, high-precision coordinate mapping is achieved. The entire technical solution is logically clear and the steps are complete, providing a simple, efficient, and high-precision calibration method for projection pointing systems of event-based sensors, significantly improving the system's practicality and user experience.

[0077] It should be noted that the core of the event-based sensor projection pointing calibration method involved in this invention lies in using an event-based sensor to capture a calibration pattern on the projected image and establishing a mapping relationship between the image coordinate system and the screen physical coordinate system through event stream processing. In one embodiment of this invention, an infrared filter in front of the event-based sensor lens is removed by an infrared switcher (IR-cut) to improve the quality of the event-based sensor's response to the calibration pattern, especially its response sensitivity in the infrared band. However, this step is not a necessary technical feature for implementing this invention. The calibration method of this invention is also applicable to systems without an infrared filter or with a fixed infrared filter; the filter removal and resetting steps can be omitted, and calibration can be completed by directly playing the calibration video and acquiring the event stream. In other words, regardless of whether the event-based sensor is equipped with an infrared filter, and regardless of whether the filter is switchable, any technical solution that achieves projection pointing calibration by calibrating a pattern, acquiring an event stream, extracting the screen boundary, and calculating the mapping relationship falls within the protection scope of this invention.

[0078] This invention also provides a projection pointing system based on an event-based sensor. In one embodiment, the system includes a projector, an event-based sensor, and a processing unit. The projector plays a calibration video containing calibration patterns on a projected image; the event-based sensor captures images of the projected image to obtain an event stream; the processing unit is connected to the event-based sensor and the projector to execute the calibration method as described above. Specifically, during the calibration phase, the processing unit controls the projector to play the calibration video, receives the event stream output by the event-based sensor, processes the event stream to determine the range of the projected image in the sensor pixel coordinate system, and calculates calibration parameters based on this range and the resolution of the projected image; during the pointing phase, the processing unit retrieves the calibration parameters and maps the pointing point pixel coordinates captured by the event-based sensor to the actual pointing coordinates on the projected image. This system achieves high-precision projection pointing interaction through the collaborative work of the above components.

[0079] To better illustrate the present invention, numerous specific details have been provided in the detailed embodiments described above. Those skilled in the art should understand that the present invention can be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of the present invention.

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A projection pointing calibration method for an event-based sensor, characterized in that, Includes the following steps: Play a calibration video, which includes a calibration pattern with periodic brightness changes formed on a projected screen; The calibration video is captured using an event-based sensor to obtain an event stream; The event stream is processed to determine the range of the projected image in the pixel coordinate system of the event-based sensor; The calibration parameters are determined based on the range of the projected image in the pixel coordinate system and the resolution of the projected image. The calibration parameters are used to map the coordinates of the pointing point in the pixel coordinate system to the actual pointing coordinates on the projected screen during pointing operations.

2. The projection pointing calibration method for an event-based sensor according to claim 1, characterized in that: The calibration pattern is set in the boundary area of ​​the projected image and is presented by alternating black and white to indicate the boundary and / or center point of the projected image.

3. The projection pointing calibration method for an event-based sensor according to claim 1, characterized in that, The processing of the event stream includes: Generate a perceptual image that reflects the boundary and / or center point of the calibration pattern to determine the range of the projected image in the pixel coordinate system.

4. The projection pointing calibration method for an event-based sensor according to claim 1, characterized in that, The processing of the event stream includes: The event stream is compressed into frames to accumulate events within a time window and generate event frames. Image analysis is performed on the event frames to extract the boundaries of the calibration pattern, thereby determining the boundaries and / or center point of the projected image in the pixel coordinate system.

5. The projection pointing calibration method for an event-based sensor according to claim 4, characterized in that: The calibration parameters include mapping coefficients calculated based on the range of the projected image in the pixel coordinate system and the resolution of the projected image.

6. The projection pointing calibration method for an event-based sensor according to claim 1, characterized in that, Also includes: Event data is continuously collected over multiple time windows, generating multiple event frames and calculating multiple sets of calibration parameters; The average value of the multiple sets of calibration parameters is calculated or the average value is taken after removing outliers, and the obtained average value is used as the final determined calibration parameter.

7. The projection pointing calibration method for an event-based sensor according to claim 1, characterized in that, The step of mapping the pixel coordinates of the pointing point to the actual pointing coordinates on the projected image includes: A screen coordinate system is established with the predetermined corner point of the projected image as the origin. The actual pointing coordinates are calculated based on the calibration parameters and the offset of the pointing point in the pixel coordinate system relative to the boundary of the projected image in the pixel coordinate system.

8. The projection pointing calibration method for an event-based sensor according to claim 1, characterized in that: When processing the event stream, an edge detection algorithm combined with Hough transform or contour search algorithm is used to extract the boundary of the calibration pattern.

9. A projection pointing system based on an event-driven sensor, characterized in that, include: A projector used to play calibration videos containing calibration patterns on a projected screen; An event-based sensor is used to capture the projected image and acquire an event stream; The processing unit is configured to execute the projection pointing calibration method for an event-based sensor as described in any one of claims 1-8.

10. The event-based sensor projection pointing system according to claim 9, characterized in that, The optical path of the event-based sensor is equipped with an infrared filter and an infrared switch; Before playing the calibration video, the infrared filter is removed by the infrared switcher; After calibration, the infrared filter is reset via the infrared switch.

Images