A projection interaction device and method based on invisible light

Abstract

This invention relates to the field of intelligent projection interaction technology, specifically to a projection interaction device and method based on invisible light. The device includes a housing, a projection optical engine, a camera, a main control chip, and a processing module. An IRCUT dual-filter switcher is installed on the optical path between the camera lens and the image sensor. The filters within the IRCUT dual-filter switcher include a visible light filter and an infrared light filter. The method projects an image through the projection optical engine; the lens captures the projected image through the visible light filter; the processing module identifies the four corner markers of the image and constructs a perspective transformation matrix between the lens and the UI coordinate system; the lens captures the invisible light spot projected by the user's laser pointer through the infrared light filter, processes it to obtain the center coordinates in the lens coordinate system, and then maps it to the UI coordinates through the perspective transformation matrix to achieve interaction. This invention requires no external devices, reducing operational complexity and cost, and improving the convenience and smoothness of interaction.

Description

Technical Field

[0001] This invention relates to the field of intelligent projection interaction technology, and in particular to a projection interaction device and method based on invisible light. Background Technology

[0002] In today's digital office era, the trend towards portable office equipment is becoming increasingly apparent, placing new demands on the form and function of office devices. Smart projection devices, with their inherent advantages, are gradually emerging in scenarios such as meetings and classrooms. Their lightweight and portable characteristics allow users to easily take them to different meeting rooms or classrooms, freeing them from the constraints of fixed installations of traditional large-screen devices and greatly improving the flexibility of use. For example, in companies with multi-regional offices, where employees need to hold meetings in different locations, smart projection devices can be easily carried to meet meeting presentation needs at any time.

[0003] However, existing projection devices have significant shortcomings in terms of interactive experience. For input, they generally lack touch capabilities, relying solely on a mouse, remote control, or external writing tablet. This approach differs greatly from people's natural and fluid interaction habits when interacting with a screen. The disadvantages of traditional projection devices are even more pronounced in writing and annotation scenarios. For example, in a meeting setting, when the meeting host needs to highlight key points or write down their thoughts on the presentation content, using a mouse is neither precise nor convenient, a remote control is insufficient for such fine-tuning, and an external writing tablet increases the complexity and cost of device connectivity, while its writing feel and experience are still significantly inferior to direct touch operation on the screen. Summary of the Invention

[0004] To address the technical problem of inconvenient operation and input during interactive experiences with projection devices in existing technologies, this invention provides a projection interaction device and method based on invisible light.

[0005] The present invention provides the following technical solutions.

[0006] An invisible light-based projection interaction device includes a housing, a projection optical engine, a camera, a main control chip, and a processing module. The projection optical engine and camera are mounted on the housing, while the main control chip and processing module are mounted inside the housing. Both the projection optical engine and camera are connected to the main control chip, which is also connected to the processing module. An IRCUT dual-filter switcher is installed on the optical path between the camera lens and the image sensor. The filters in the IRCUT dual-filter switcher include a visible light filter and an infrared light filter.

[0007] Furthermore, the visible light filter allows visible light of 380~760nm to pass through, and the infrared light filter allows infrared light of 850nm±5nm or 940nm±5nm to pass through.

[0008] Furthermore, the optical axis of the lens is parallel to the projection direction of the projector, and the optical axis of the lens is directly opposite the image area projected by the projector onto the wall; the optical focal plane of the lens is strictly coplanar with the photosensitive surface of the image sensor; it also includes a TOF sensor, which is mounted on the housing and connected to the main control chip.

[0009] A projection interaction method based on invisible light includes the following steps:

[0010] The projected image is projected onto the projection area using a projector.

[0011] The IRCUT dual filter switcher switches to the visible light filter to ensure that the camera can capture the projected image; the camera focuses on the projected image to obtain a clear image; the projection optical engine performs keystone correction based on the clear image and outputs a regular rectangular projection area;

[0012] The processing module identifies the four corner markers of the projected image and determines the position of the projected area in the lens coordinate system based on the four corner markers. It constructs a perspective transformation matrix from the lens coordinate system to the UI coordinate system based on at least four pairs of non-collinear corresponding points of the four corner marker coordinates in the lens coordinate system and the UI coordinates of the projection interaction device.

[0013] After the perspective transformation matrix is ​​constructed, the IRCUT dual filter switcher switches to the infrared filter and acquires the laser spot image in the projection area through the camera;

[0014] The laser spot image frame is converted to grayscale by the processing module, and the threshold is automatically calculated by the big law method for binarization. The contour of the laser spot is obtained by the cv2.findcontours function in the OpenCV library.

[0015] The laser spot is selected based on its roundness and diameter, and the center coordinates of the laser spot in the lens coordinate system are determined.

[0016] The center coordinates of the laser spot in the lens coordinate system are mapped to the UI coordinate system based on the perspective transformation matrix.

[0017] Furthermore, the processing module identifies the four corner markers of the projected image, including the following steps:

[0018] A pure white background is projected onto the projection area using a projector.

[0019] Display QR codes in the four corners of the UI interface;

[0020] The camera captures image frames of the white projected image and the UI QR code.

[0021] Generate a circular search area with the center of each QR code as the origin;

[0022] Apply the Hough line transform within the circular search area to detect horizontal and vertical line segments;

[0023] Find the intersection points of the horizontal and vertical line segments to obtain the four corner markers of the projected image.

[0024] Furthermore, in the grayscale and binarization processing of the laser spot image frame, a weighted average method is used for grayscale conversion, and the calculation formula is as follows:

[0025]

[0026] In the formula: Gray is a grayscale image, R is the value of red, G is the value of green, and B is the value of blue;

[0027] Binarization is achieved using the image thresholding function cv2.threshold from the OpenCV library in conjunction with Otsu's method, and includes the following steps:

[0028] Call the cv2.threshold function;

[0029] Use the grayscale image Gray obtained after grayscale conversion as the input image;

[0030] Set the maxval parameter to 255;

[0031] By specifying and The combined threshold type flag enables the function to automatically calculate the optimal threshold and complete the binarization process.

[0032] Furthermore, in obtaining the contour of the laser spot using the cv2.findcontours function in the OpenCV library, the parameters of the cv2.findcontours function include: the input image is a binary image, the contour retrieval mode is to detect only the outermost contour, and the contour approximation method is to compress redundant points.

[0033] Furthermore, the calculation of the diameter and roundness of the laser spot includes the following steps:

[0034] Erosion and dilation operations are performed on the contour image of the binarized laser spot;

[0035] Traverse each contour and calculate the pixel diameter and pixel area of ​​its circumcircle. The pixel diameter of the circumcircle is denoted as r. Calculate the pixel area S of the circumcircle using the formula for the area of ​​a circle.

[0036] Within the circumcircle, count the number N pixels with a value of 255. The value is considered as the roundness of the light spot.

[0037] Furthermore, the process of screening laser spots based on diameter and roundness includes the following steps:

[0038] During the calibration phase of the projection interaction device, the distance d between the projection device and the projection area is obtained through the TOF sensor. The one-to-one correspondence between the distance d and the laser spot pixel diameter r is calibrated at different distance points to obtain the dr prior parameter library.

[0039] In practical applications of projection interaction devices, the actual distance between the projection device and the projection area is obtained via Time-of-Flight (TOF). Then, the corresponding prior reference laser spot pixel diameter is obtained from the DR prior parameter library for that actual distance. ;

[0040] In a binary image, select pixels with a diameter of 0.8. ~1.2 The outline of the laser spot within the range;

[0041] Construct a confidence function using pixel diameter r and roundness:

[0042]

[0043] In the formula: F is the confidence function; , All are hyperparameters;

[0044] The confidence level of each laser spot is calculated using a confidence function, and the spot with the highest confidence level is selected as the target spot.

[0045] Furthermore, the calculation formula for mapping the center coordinates of the laser spot in the lens coordinate system to the UI coordinate system based on the perspective transformation matrix is ​​as follows:

[0046]

[0047]

[0048] In the formula: These are the coordinates of the center point of the laser spot in the UI coordinate system; The coordinates of the center point of the laser spot in the lens coordinate system; Perspective transformation matrix The elements in.

[0049] Advantages and positive effects of the present invention.

[0050] This invention utilizes a camera in conjunction with a visible light filter to identify the projected image, and a camera in conjunction with an infrared light filter to identify the laser spot emitted by a laser pointer. The coordinates of the laser spot are then projected onto the UI coordinates of the projection device, enabling direct interaction between the projection device and the user's laser pointer. This solves the technical problem of inconvenient input during interactive experiences with existing projection devices. The projector projects the image onto the projection area, and the camera identifies the projected image and its four corner markers to determine the position of the projection area in the lens coordinate system. A perspective transformation matrix is ​​constructed based on the coordinates of the corner markers and the UI coordinates of the interactive projection device. This matrix then maps the center coordinates of the laser spot projected by the user onto the UI coordinates, allowing the user to directly interact with the interactive projection device without external equipment. This reduces operational complexity and cost, and improves the convenience and smoothness of the interaction. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a structural diagram of a projection interaction device based on invisible light provided by the present invention.

[0053] Figure 2 This is a schematic diagram of a projection interaction device based on invisible light, provided by the present invention.

[0054] Figure 3 A flowchart of a projection interaction method based on invisible light provided by the present invention.

[0055] In the picture: 1. Housing; 2. Projector engine; 3. Camera. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0057] This invention provides a projection interaction device based on invisible light, such as... Figures 1-2As shown, the device includes a housing 1, a projection optical engine 2, a camera 3, a TOF sensor, a main control chip, and a processing module. The projection optical engine 2, camera 3, and TOF sensor are mounted on the housing 1, while the main control chip and processing module are installed inside the housing 1. The projection optical engine 2, camera 3, and TOF sensor are all connected to the main control chip, which in turn is connected to the processing module. An IRCUT dual filter switcher is installed on the optical path between the lens of the camera 3 and the image sensor. The filters in the IRCUT dual filter switcher include visible light filters and infrared light filters. The lens and image sensor are inherent components of the camera, both of which are currently standardized components and can be directly adapted for use. The installed IRCUT dual filter switcher is also an existing device. An IRCUT dual filter switcher typically consists of two filters, a power mechanism, a housing, and a circuit control board. The circuit control board controls the movement of the power mechanism to achieve the switching of the two filters. Visible light filters allow visible light in the 380-760nm range to pass through while filtering out infrared light with wavelengths greater than 760nm, preventing color shifts caused by infrared interference and ensuring accurate color projection from the camera. Infrared light filters only allow specific infrared wavelengths of 850nm±5nm or 940nm±5nm to pass through, filtering out visible light, other infrared wavelengths, and all other non-target wavelengths, thus shielding against ambient light interference and ensuring the camera can accurately identify the corresponding wavelength laser spot emitted by the laser pointer.

[0058] The optical axis of the lens is parallel to the projection direction of the projector engine 2, and the lens optical axis is directly opposite the image area projected onto the wall by the projector engine; the optical focal plane of the lens is strictly coplanar with the photosensitive surface of the image sensor. The projector engine 2 supports 720p to 4K image output. The projection interaction device is equipped with peripheral interfaces such as USB / HDMI-IN, supports ISP image signal processing and software SDK expansion, and can dynamically switch between visible light imaging and 940nm infrared narrowband imaging through the IRCUT dual filter switcher; its image sensor has a 120fps output capability, global image 60fps@720p (1280*720), and local specified area image 120fps@(128*128). The local specified area image is achieved through the sensorCCSDigitalCrop protocol and the cooperation of the ISP processing part. The center coordinates of the local image are specified by the subsequent processing method. This method reduces the requirements of ISP capabilities, reduces the overall data bandwidth consumption, and reduces the computational complexity of subsequent processing, making it a highly efficient method.

[0059] An invisible light-based projection interaction method is implemented using an invisible light-based projection interaction device, such as... Figure 3 As shown, it includes the following steps:

[0060] The projected image is projected onto the projection area using a projector.

[0061] The IRCUT dual filter switcher switches to the visible light filter to ensure that the camera can capture the projected image; the camera focuses on the projected image to obtain a clear picture; the projection optical engine performs keystone correction based on the clear picture and outputs a regular rectangular projection area.

[0062] The processing module identifies the four corner markers of the projected image and determines the position of the projected area in the lens coordinate system based on these markers.

[0063] A pure white image is projected onto the projection area using a projector.

[0064] QR codes are displayed in the four corners of the UI interface.

[0065] The camera module captures image frames of the white projected image and the UI QR code.

[0066] A circular search area is generated with the center of each QR code as the origin.

[0067] Apply the Hough line transform within the circular search area to detect horizontal and vertical line segments.

[0068] Find the intersection points of the horizontal and vertical line segments to obtain the four corner markers of the projected image.

[0069] Construct a perspective transformation matrix from the camera coordinate system to the UI coordinate system based on the coordinates of the four corner markers in the camera coordinate system and the UI coordinates of the projection interaction device. .

[0070] After the perspective transformation matrix is ​​constructed, the IRCUT dual filter switcher switches to the infrared filter and acquires the laser spot image in the projection area through the camera.

[0071] The laser spot image frame is processed by the processing module to be converted to grayscale and then binarized.

[0072] The weighted average method is used for grayscale conversion, and the calculation formula is as follows:

[0073]

[0074] In the formula: Gray is a grayscale image, R is the value of red, G is the value of green, and B is the value of blue.

[0075] Binarization is achieved using the image thresholding function cv2.threshold from the OpenCV library in conjunction with Otsu's method, and includes the following steps:

[0076] Call the cv2.threshold function.

[0077] The grayscale image Gray obtained after grayscale conversion is used as the input image.

[0078] Set the maxval parameter to 255.

[0079] By specifying and The combined threshold type flag enables the function to automatically calculate the optimal threshold and complete the binarization process.

[0080] The contour of the laser spot can be obtained using the cv2.findcontours function in the OpenCV library. The parameters of the cv2.findcontours function include: the input image is a binary image, the contour retrieval mode is to detect only the outermost contour, and the contour approximation method is to compress redundant points.

[0081] Based on the outline of the laser spot, its roundness, and its diameter, determine the center coordinates of the laser spot in the lens coordinate system:

[0082] The calculation of the diameter and roundness of the laser spot includes the following steps:

[0083] A combination of erosion and dilation operations is performed on the contour image of the binarized laser spot to optimize the integrity of the spot contour, remove background noise interference, and enhance edge features.

[0084] Traverse each contour after a combination of erosion and dilation operations, and solve for the pixel diameter and pixel area of ​​the circumcircle of the contour. The pixel diameter of the circumcircle is denoted as r; calculate the pixel area S of the circumcircle using the formula for the area of ​​a circle.

[0085] Within the circumcircle, count the number N pixels with a value of 255. The value is considered as the roundness of the light spot.

[0086] The process of selecting laser spots based on diameter and roundness includes the following steps:

[0087] Lasers possess high focusing properties, resulting in minimal variation in the physical size of the laser spot produced by a user using a laser pointer at different distances. Cameras, on the other hand, exhibit a near-far characteristic; when the device is close to the projection area (wall or screen), the laser spot occupies a large area in the captured image, and vice versa. During the calibration phase of the projection interaction device, the distance *d* between the projection device and the projection area is obtained using a TOF sensor. The one-to-one correspondence between distance *d* and the laser spot pixel diameter *r* is calibrated at different distance points, resulting in a priori parameter library for *d*. To ensure the accuracy and practicality of the parameter library, calibration is performed every 20cm within the effective working distance between the projection interaction device and the projection area, simultaneously recording the *d* value and the corresponding laser spot pixel diameter *r* value at each distance.

[0088] In practical applications of projection interaction devices, the actual distance between the projection device and the projection area is obtained via Time-of-Flight (TOF). Then, the corresponding prior reference laser spot pixel diameter is obtained from the DR prior parameter library for that actual distance. .

[0089] In a binary image, select pixels with a diameter of 0.8. ~1.2 The outline of the laser spot within the range.

[0090] Construct a confidence function using pixel diameter r and roundness:

[0091]

[0092] In the formula: F is the confidence function; , All are hyperparameters; =1.0, =0.2.

[0093] The confidence level of each laser spot is calculated using a confidence function, and the spot with the highest confidence level is selected as the target spot.

[0094] The center coordinates of the effective laser spot in the lens coordinate system are mapped to the UI coordinate system based on the perspective transformation matrix. The calculation formula is as follows:

[0095]

[0096]

[0097] In the formula: These are the coordinates of the center point of the laser spot in the UI coordinate system; The coordinates of the center point of the laser spot in the lens coordinate system; Perspective transformation matrix The elements in.

[0098] In the UI coordinate system, the distinction between clickable and background areas, and whether the background area is for handwriting applications, is determined at the system rendering level by combining the active foreground page ID with a predefined name, or through a visual model. UI area characteristics do not need to be acquired in real-time; they are acquired only when UI characteristics change, such as an increase in clickable areas.

[0099] Visual model acquisition utilizes existing UI parser models or trains proprietary UI detection models. Specifically, multiple UI interface images from intelligent systems on numerous projection devices are collected, and clickable areas, such as buttons and text boxes, are marked with rectangles. Common object detection models, such as YOLO, are trained to infer and detect these designated areas during runtime. The user's laser pointer has a button that connects to the projection interaction device via Bluetooth. The laser beam projected by the user is projected onto the UI coordinate system. The user controls the trajectory of the laser beam to interact with the projection interaction device. The projection interaction device displays the current position of the laser beam as a pointer within the projected image, facilitating user operation. When the laser spot falls into the clickable area, and the user presses and releases the button on the laser pointer, the projection device recognizes it as a click event. When the user presses and holds the button and moves the laser spot, the user can drag the button or icon in the area. When the background area is a handwriting application interface, the area is recognized as a handwriting area. When the laser spot falls into the handwriting area, if the user presses and releases the button on the laser pointer, a dot is drawn in the area. If the user presses and holds the button and moves the laser spot, a line is drawn in the area.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of invisible light based projection interaction, characterized in that, The application discloses an invisible light projection interaction device, which comprises a shell (1), a projection light machine (2), a camera (3), a main control chip and a processing module, wherein the projection light machine (2) and the camera (3) are installed on the shell (1), the main control chip and the processing module are installed in the shell (1), the projection light machine (2) and the camera (3) are connected with the main control chip, and the main control chip is connected with the processing module; an IRCUT double-filter switcher is installed on an optical path between a lens and an image sensor of the camera (3), and filters in the IRCUT double-filter switcher comprise a visible light filter and an infrared light filter; an optical axis of the lens is parallel to a projection direction of the projection light machine (2), and the optical axis of the lens is perpendicular to a picture area projected on a wall surface by the projection light machine; an optical focal plane of the lens is coplanar with a photosensitive surface of the image sensor; the invisible light projection interaction device further comprises a TOF sensor, wherein the TOF sensor is installed on the shell (1) and connected with the main control chip; the visible light filter allows visible light of 380-760 nm to pass through, and the infrared light filter allows infrared light of 850 nm±5 nm or 940 nm±5 nm to pass through; the method comprises the following steps: projecting a projection picture to a projection area by the projection light machine; switching the IRCUT double-filter switcher to the visible light filter to ensure that the camera can capture the projection picture; focusing the camera on the projection picture to obtain a clear picture; completing trapezoidal correction of the projection light machine based on the clear picture to output a regular rectangular projection area; identifying four corner mark points of the projection picture by the processing module, determining a position of the projection area in a lens coordinate system according to the four corner mark points, and constructing a perspective transformation matrix from the lens coordinate system to a UI coordinate system according to four pairs of non-collinear corresponding points between the four corner mark point coordinates in the lens coordinate system and the projection interaction device; after the perspective transformation matrix is constructed, switching the IRCUT double-filter switcher to the infrared light filter, and obtaining a laser spot image in the projection area by the camera; performing gray processing on the laser spot image frame by the processing module, automatically calculating a threshold value by majority law to perform binary processing, and obtaining a contour of the laser spot by a cv2.findcontours function in an OpenCV library; screening the laser spot according to the roundness and diameter of the laser spot and obtaining center coordinates of the screened laser spot in the lens coordinate system; mapping the center coordinates of the screened laser spot in the lens coordinate system to the UI coordinate system according to the perspective transformation matrix; and the calculation of the diameter and roundness of the laser spot comprises the following steps: performing erosion and expansion operations on a contour image of the laser spot after binary processing; traversing each contour and solving a pixel diameter and a pixel area of an inscribed circle of the contour, wherein the pixel diameter of the inscribed circle is denoted as r; and calculating the pixel area S of the inscribed circle by a circle area formula; and the screening of the laser spot according to the roundness and diameter of the laser spot comprises the following steps: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ In the circumscribed circle range, the number of pixels with pixel value 255 is counted as N, and the value of is regarded as the roundness of the light spot. ​ In the calibration stage of the projection interaction device, the distance d between the projection device and the projection area is obtained by a TOF sensor, and a one-to-one correspondence between the distance d and the pixel diameter r of the laser spot is calibrated at different distance points to obtain a d-r prior parameter library; In the actual use scene of the projection interaction device, the actual distance between the projection device and the projection area is acquired through the TOF, and the prior reference laser spot pixel diameter corresponding to the actual distance is matched from the d-r prior parameter library ; Filtering out laser spot profiles in the binary image with pixel diameters in the range of 0.8 ~1.2 range A confidence function is constructed using the pixel diameter r and the roundness: In the formula, F is a confidence function; , are hyperparameters; The confidence of each laser spot obtained is calculated using the confidence function, and the spot with the maximum confidence value is selected as the screened laser spot.

2. The invisible light based projection interaction method according to claim 1, characterized in that, The processing module identifies the four corner mark points of the projection picture, including the following steps: A picture with a white background is projected onto the projection area by a projection light machine; A two-dimensional code is displayed at the four corners of the UI interface; An image frame of the white projection picture and the UI two-dimensional code is obtained by a camera; A ring-shaped search area is generated with each two-dimensional code center as the origin; A Hough straight line transformation is applied in the ring-shaped search area to detect horizontal and vertical straight line segments; The intersection points of the horizontal and vertical straight line segments are calculated to obtain the four corner mark points of the projection picture. 3.The invisible light based projection interaction method of claim 1, wherein, In the grayscale and binarization processing of the laser spot image frame, a weighted average method is used for grayscale, and the calculation formula is as follows: In the formula, Gray is the grayscale image, R is the value of red color, G is the value of green color, and B is the value of blue color; The binarization processing is realized by combining the image threshold processing function cv2.threshold in the OpenCV library with the Otsu method, including the following steps: The cv2.threshold function is called; The grayscale image Gray obtained by grayscale is used as the input image; The maxval parameter is set to 255; By specifying the threshold type flag in combination with the function automatically calculates the optimal threshold value to complete the binarization process.

4. The invisible light based projection interaction method according to claim 1, characterized in that, In the process of obtaining the contour of the laser spot by the cv2.findcontours function in the OpenCV library, the parameters of the cv2.findcontours function include: the input image is a binary image, the contour retrieval mode is to detect only the outermost contour, and the contour approximation method is to compress redundant points.

5. The invisible light based projection interaction method according to claim 1, wherein, The calculation formula for mapping the center coordinates of the laser spot in the lens coordinate system to the UI coordinate system according to the perspective transformation matrix is as follows: In the formula: is the coordinate of the laser spot center point in the UI coordinate system; is the coordinate of the laser spot center point in the lens coordinate system; is an element in the perspective transformation matrix .

Images