Multi-mode projection picture interaction device and method

Abstract

The invention relates to an intelligent projection interaction technology, in particular to a multi-mode projection picture interaction device and method. The device comprises a projection device and a laser pen, the projection device is provided with a shell, a projection light machine and a camera, the laser pen comprises a penholder, a laser module and the like, and a pressing type trigger assembly is arranged in the penholder. Pictures are projected through the projection light machine, and the camera can recognize the pictures and laser. The laser pen can switch the flying mouse mode, the handwriting mode and the remote control mode. In the flying mouse mode, the button is touched through the capacitor, and the laser module emits laser. The handwriting mode controls the laser module through pressing of the trigger assembly; and the remote control mode realizes basic control through mechanical keys. Through laser spot positioning and multi-mode switching, interaction convenience and fluency are improved, external equipment is not needed, cost is reduced, and operation habits are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent projection interaction, in particular to a multi-mode projection picture interaction device and method. BACKGROUND

[0002] Under the current wave of digital office, the demand for mobility and convenience of office scenarios continues to rise, and this trend has profoundly reshaped the design logic and functional boundaries of office equipment. Intelligent projection equipment, with its lightweight and easy-to-carry core advantages, has quickly gained popularity in various conference and seminar, teaching and training scenarios, and has gradually become an important choice to replace traditional fixed large screens.

[0003] However, the limitations of existing intelligent projection equipment in interactive experience have become increasingly prominent, becoming a key bottleneck restricting its further penetration. At the operation level, most products lack direct touch interaction capability, and users have to rely on external mouse, dedicated remote control or additional handwriting board to complete instruction input. This operation method is significantly disjointed from the natural interaction habit of "directly touching the screen" formed on smartphones and tablets, not only increasing the operation learning cost, but also easily affecting efficiency in emergency demonstration scenarios due to device adaptation problems. SUMMARY

[0004] In order to solve the technical problems that the existing technology cannot directly interact with the projection equipment and the interaction mode is single, the present application provides a multi-mode projection picture interaction device and method.

[0005] To this end, the present application provides the following technical solutions.

[0006] A multi-mode projection picture interaction device, comprising a projection device and a laser pen, the projection device comprising a shell, a projection light machine, a camera, a main control chip and a processing module, the projection light machine and the camera being installed on the shell, the main control chip and the processing module being installed in the shell, the projection light machine and the camera being connected with the main control chip, and the main control chip being connected with the processing module; the laser pen comprising a pen barrel, a laser module, a micro control unit and a trigger assembly, the pen barrel being a hollow structure, the laser module and the micro control unit being installed inside the pen barrel, the laser module being connected with the micro control unit, the trigger assembly comprising a rod body with an axial through hole and a sensor assembly, the rod body being at least partially arranged in the front end of the pen barrel, the rear end of the rod body being connected with the sensor assembly, the sensor assembly being connected with the micro control unit, and the laser emitted by the laser module being capable of passing through the rod body along the axial direction of the rod body.

[0007] Further, an IRCUT dual filter switcher is installed on the optical path between the lens and the image sensor of the camera, and the filters in the IRCUT dual filter switcher include a visible light filter and an infrared light filter.

[0008] Further, the laser pointer further comprises a Bluetooth module, a capacitive touch press key and / or a mechanical key for input control, the capacitive touch press key comprises a press key, a press bracket and a touch plate, the press key is arranged in the pen barrel, the press bracket is located above the press key and is slidingly fitted to the side wall of the pen barrel, both sides of the press bracket are provided with a rebounding arm, the fixed end of the rebounding arm is connected with the side wall of the pen barrel, and the touch plate is installed on the press bracket; the Bluetooth module, the press key, the touch plate and the mechanical key are connected with the micro control unit, the Bluetooth module can interact with the master control chip, and the mechanical key is installed on the pen barrel.

[0009] Further, the protective cover is slidingly installed on the pen barrel along the length direction of the pen barrel, and the capacitive touch press key and the mechanical key are located on the moving path of the protective cover.

[0010] A multi-mode projection picture interaction method, comprising the following steps: S1. projecting a projection picture to a projection area by a projection light machine; S2. switching the IRCUT double filter switcher to a visible light filter to ensure that the camera can capture the projection picture; focusing the camera on the projection picture to obtain a clear picture; and the projection light machine completing trapezoidal correction based on the clear picture to output a regular rectangular projection area; S3. a processing module recognizing four corner mark points of the projection picture, determining the position of the projection area in the lens coordinate system according to the four corner mark points, and constructing a perspective transformation matrix from the lens coordinate system to the UI coordinate system according to at least four pairs of non-collinear corresponding points of the four corner mark point coordinates in the lens coordinate system and the UI coordinate system; S4. after the perspective transformation matrix is constructed, the IRCUT double filter switcher is switched to an infrared light filter, and the camera is used to obtain a laser spot in the projection area; S5. touching the capacitive touch press key of the laser pointer to enter a mouse mode, the laser module emits laser to the projection area, stopping touching the capacitive touch press key of the laser pointer to exit the mouse mode, the laser module stops emitting laser to the projection area, in the mouse mode, the laser pointer simulates a mouse to realize remote control; or pressing the front end of the laser pointer on the projection area to enter a handwriting mode, the laser module emits laser according to the signal of the sensor assembly, stopping pressing the front end of the laser pointer on the projection area to exit the handwriting mode, the laser module stops emitting laser, in the handwriting mode, the laser pointer simulates a handwriting pen to realize close-range control; S6. mapping the coordinates of the laser spot in the lens coordinate system to the UI coordinate system through the perspective transformation matrix.

[0011] Further, in the step of the laser module emitting laser to the projection area: The exposure value of the image sensor is pre-fixed to 4.5%-5.5% of its maximum exposure value. In a common window-side sunlight scene, the laser spot image obtained by the camera through the infrared filter is collected. The average pixel intensity of the laser spot image is calculated and recorded as the empirical reference value La. Collect the laser spot image obtained in the current environment, calculate the average pixel intensity of the laser spot image, and record it as the current value L; Calculate the ratio of the current value L to the reference value La, recorded as Ra; If Ra> , it is determined that the external light has an impact, and the laser module emits a flashing laser; otherwise it is determined that there is no impact, and the laser module emits a continuous laser; .

[0012] Furthermore, the time duration for the tip of the laser pen to press and trigger the pressure sensor or the time duration for the capacitive touch key to be pressed is recorded as ,when Less than When , a click event or a point drawing operation is triggered; when Greater than When , a sliding event or a mobile writing operation is triggered; .

[0013] Furthermore, in the step of emitting laser light from the laser module, the center coordinates of the laser spot are continuously estimated using the Kalman filter algorithm. The state vector of the Kalman filter algorithm includes 、 、 and ,in and is the position coordinate of the center point of the laser spot at time t, and is the velocity term of the laser spot at time t. Within a period of time, if continuous state estimation can be achieved, the camera increases the frame rate output. If continuous state estimation cannot be achieved, or the trajectory of the laser spot center point obtained by continuous state estimation increases, the camera returns to low frame rate full-screen detection mode.

[0014] Furthermore, in the process of continuous state estimation, the dwell detection algorithm is used to record the time the center point of the laser spot stays in a circular area. Then mark the point as a candidate click point; .

[0015] During the process of triggering the click event, a video of the user holding the laser pen is collected, and for the tth frame of the operation video, a 7-dimensional feature vector is extracted and calculated:

[0016] In the formula: is the position of the laser spot center point in the tth frame, is the speed of the laser spot center point in the tth frame, is the acceleration of the laser spot center point in the tth frame, is the trajectory angle change of the laser spot center point in the current frame relative to the previous frame; mark the best click frame in the video in a frame playback manner; configure a binary classification label for each frame , if the tth frame is the best click frame, ; if the tth frame is a non-click frame, ; use the 7-dimensional feature vector sequence of the continuous N frames as the input of the overall model, so that the overall model outputs the probability that the tth frame is the best click frame ; the overall model is a time sequence model connected with a full connection classification layer; the overall model is trained by taking whether the corresponding frame is the best click frame as a supervision label; use the trained overall model to predict the feature sequence of the real-time video stream, output the probability that each frame is the best click frame , select the best click frame with the maximum probability as the best click target frame, and use the spot center point of the frame as the predicted click point; when the distance between the candidate click point and the predicted click point is less than a set value, the candidate click point is the best click point.

[0017] Further, the step S4 further comprises: after the perspective transformation matrix is constructed, a mechanical button is pressed to enter a remote control mode, functions of the mechanical button include volume adjustment, page switching, projection parameter adjustment, the mechanical button is stopped pressing to exit the remote control mode.

[0018] Advantages and positive effects of the present application.

[0019] The application directly interacts with the laser pen through the projection device without external equipment, reduces the operation complexity and cost, and has multiple interaction modes of remote interaction mode, remote control mode and handwriting mode. The application solves the technical problems of the prior art that cannot directly interact with the projection device and the single interaction mode. The projection light machine projects a projection picture in a projection area, the camera and the visible light filter are used to focus the projection picture by the camera and correct the trapezoidal of the projection light machine, and a perspective transformation matrix of the lens coordinate system to the UI coordinate system is constructed. The camera and the infrared light filter are used to identify the laser spot in the projection picture, when the capacitive touch press key is pressed, the laser pen enters the mouse mode, the laser module emits laser to the projection area at a long distance to realize long-distance control, when the mechanical key is pressed, the laser pen enters the remote control mode, when the laser pen enters the handwriting mode, the front end of the laser pen is pressed in the projection area, the opening and closing of the laser module is controlled by the sensor assembly, the laser module emits laser to the projection area at a short distance to realize short-distance control, the laser spot obtained by the camera is mapped to the UI coordinate system of the projection device to realize the interaction between the laser pen and the projection device. The laser spot positioning combined with the multi-mode switching makes the laser pen not only point, but also realize remote precise interaction in the mouse mode, which is consistent with the natural operation habit, the remote control mode can meet the basic control demand, and the handwriting mode realizes direct writing and marking in the projection area by controlling the laser through the sensor assembly, and the convenience and fluency of the interaction are improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 A structural diagram of a projection device of a multi-mode projection picture interaction device provided by the present application.

[0022] Figure 2 A principle diagram of a projection device of a multi-mode projection picture interaction device provided by the present application.

[0023] Figure 3 A structural diagram of a laser pen of a multi-mode projection picture interaction device provided by the present application.

[0024] Figure 4 A trigger assembly structure diagram of a laser pen of a multi-mode projection picture interaction device provided by the present application.

[0025] Figure 5An explosion structure diagram of a trigger assembly of a laser pen of a multi-mode projection picture interactive device provided by the application.

[0026] Figure 6 An explosion structure diagram of a capacitive touch press key of a laser pen of a multi-mode projection picture interactive device provided by the application.

[0027] Figure 7 A flow chart of a multi-mode projection picture interactive method provided by the application.

[0028] In the figure: 1, a shell; 2, a projection light machine; 3, a camera; 4, a pen barrel; 5, a laser hole; 6, a capacitive touch press key; 7, a mechanical key; 8, a protective cover; 9, a pen tip sleeve; 10, a rod body; 11, a PCB board; 12, a pressure sensor; 13, a press key; 14, a press support; 15, a touch plate; 16, a rebound arm; 17, a touch glass cover plate. DETAILED DESCRIPTION

[0029] In order to enable the personnel in the technical field to better understand the application scheme, the technical scheme in the embodiments of the application will be clearly and completely described below in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by the personnel in the field without making creative efforts should belong to the protection scope of the application.

[0030] The application provides a multi-mode projection picture interactive device, as shown in the figure, comprising a projection device and a laser pen. Figures 1-3 The projection device comprises a shell 1, a projection light machine 2, a camera 3, a main control chip and a processing module, the projection light machine 2 and the camera 3 are installed on the shell 1, the main control chip and the processing module are installed in the shell 1, the projection light machine 2 and the camera 3 are connected with the main control chip, and the main control chip is connected with the processing module.

[0031] The laser pen comprises a pen barrel 4, a laser module and a micro control unit, the pen barrel 4 is of a hollow structure, the laser module and the micro control unit are installed inside the pen barrel 4, the laser module is connected with the micro control unit, and the pen barrel 4 is provided with a pen tip sleeve 9 at the front end in a detachable threaded mode, so that the internal structure of the laser pen is convenient to maintain through the detachable pen tip sleeve 9.

[0032] Further comprising a trigger assembly, one form of the trigger assembly is as shown in the figure. Figures 4-5As shown, the pen includes a rod body 10 with an axial through hole and a sensor assembly, the rod body 10 is at least partially arranged in the front end of the pen tip sleeve 9, the rear end of the rod body 10 is connected with the sensor assembly, the sensor assembly is connected with the micro control unit, the rod body 10 is a hollow structure, and the laser emitted by the laser module can pass through the rod body 10 along the axial direction of the rod body 10. The sensor assembly includes a PCB board 11 and a pressure sensor 12, the rear end of the rod body 10 is connected with the PCB board 11, the pressure sensor 12 is installed between the rod body 10 and the PCB board 11, and the PCB board 11 is installed in the pen barrel 4. When the pen tip sleeve 9 is pressed on the projection area, the front end of the rod body 10 will be subjected to the pushing force applied by the projection area, and the rear end of the rod body 10 will press the PCB board 11, the PCB board 11 will be slightly deformed, the pressure sensor 12 receives the deformation signal of the PCB board 11 and transmits the signal to the micro control unit, and then controls the laser module to emit laser to project in the projection area.

[0033] Another form of the trigger assembly is that the pen tip sleeve 9 is provided with a laser hole 5, the laser emitted by the laser module can pass through the laser hole 5, and a pressure sensor is installed between the pen tip sleeve 9 and the pen barrel 4. Pressing the pen tip sleeve 9 triggers the pressure sensor, and then controls the laser module to emit laser to project in the projection area through the micro control unit.

[0034] An IRCUT dual filter switch is installed on the optical path between the lens and the image sensor of the camera 3, and the filters in the IRCUT dual filter switch include visible light filters and infrared light filters. The lens and the image sensor are inherent components of the camera, both of which belong to the current standardized components and can be directly adapted for use. The installed IRCUT dual filter switch is also an existing device, which is usually composed of two filters, a power mechanism, a shell and a circuit control board. The power mechanism is controlled by the circuit control board to move to realize the switching of the two filters. The visible light filter can allow visible light of 380~760nm to pass through, while filtering out infrared light with a wavelength >760nm, avoiding color deviation caused by infrared interference, and ensuring the color accuracy of the projection picture obtained by the camera. The infrared light filter only allows specific infrared waveband light of 850nm±5nm or 940nm±5nm to pass through, filters out all non-target wavebands such as visible light and other wavelengths of infrared light, thereby shielding environmental light interference and ensuring that the camera can accurately identify the corresponding wavelength laser spot emitted by the laser pen.

[0035] The optical axis of the lens is parallel to the projection direction of the projection light machine 2, and the optical axis of the lens is directly opposite the picture area projected on the wall surface by the projection light machine, and the optical focal plane of the lens is strictly coplanar with the photosensitive surface of the image sensor. The laser pointer further comprises a Bluetooth module and a capacitive touch press key 6 and / or a mechanical key 7 for input control, the Bluetooth module and the mechanical key 7 are connected with the micro-control unit, the Bluetooth module can interact with the master control chip, and the capacitive touch press key 6 and the mechanical key 7 are both mounted on the pen barrel 4. Figure 6 As shown in the figure, the capacitive touch press key 6 comprises a press key 13, a press bracket 14 and a touch plate 15, the press key 13 is arranged in the pen barrel 4, the press bracket 14 is located above the press key 13 and is slidingly fitted to the side wall of the pen barrel 4, both sides of the press bracket 14 are provided with a rebounding arm 16, the fixed end of the rebounding arm 16 is connected with the side wall of the pen barrel 4, the touch plate 15 is mounted on the press bracket 14, and a touch glass cover plate 17 is mounted on the touch plate 15, the touch glass cover plate 17 plays a protective role on the touch plate 15. The press key 13 and the touch plate 15 are both connected with the micro-control unit. When the user touches the touch glass cover plate 17, the touch plate 15 recognizes the electric field of the human body, the touch plate 15 triggers the laser module to continuously emit laser, and when the user presses the touch glass cover plate 17, the press bracket 14 moves towards the press key 13, the press key 13 is pressed, a click event is triggered, and the press bracket 14 is reset through the rebounding arm 16 after being released.

[0036] The protective cover 8 is slidingly mounted on the pen barrel 4 along the length direction of the pen barrel 4, and the capacitive touch press key 6 and the mechanical key are located on the moving path of the protective cover 8.

[0037] A multi-mode projection picture interaction method based on a multi-mode projection picture interaction device, as shown in the figure, comprises the following steps: Figure 7 S1. Project a projection picture to a projection area through a projection light machine.

[0038] S2. The IRCUT double filter switcher is switched to a visible light filter to ensure that the camera can capture the projection picture; the camera focuses on the projection picture to obtain a clear picture; the projection light machine completes trapezoidal correction based on the clear picture and outputs a regular rectangular projection area.

[0039] S3. The four corner mark points of the projection picture are recognized through a visible light module, the position of the projection area in a lens coordinate system is determined according to the four corner mark points, and a perspective transformation matrix from the lens coordinate system to a UI coordinate system is constructed according to at least four pairs of non-collinear corresponding points of the four corner mark point coordinates in the lens coordinate system and the UI coordinate system.

[0040] S4. After the perspective transformation matrix is constructed, the IRCUT double filter switcher is switched to an infrared light filter, and the laser spot in the projection area is obtained through the camera.

[0041] ​S5. Touch the capacitive touch press key 6 of the laser pen, enter the mouse mode, the laser module emits laser to the projection area, stop touching the capacitive touch press key of the laser pen, exit the mouse mode, the laser module stops emitting laser to the projection area, in the mouse mode, the laser pen simulates the mouse to realize remote control. At this time, the protective cover 8 slides to the rear end of the pen barrel 4, and the capacitive touch press key 6 and the press mechanical key are located outside the protective cover 8. The mouse mode refers to that the laser pen can simulate the mouse to operate, the laser module emits laser to the projection area to simulate the cursor displayed on the screen, and the capacitive touch press key simulates the press mouse key.

[0042] Or press the front end of the laser pen on the projection area, enter the handwriting mode, the laser module emits laser according to the signal of the sensor assembly, stop pressing the front end of the laser pen on the projection area, exit the handwriting mode, the laser module stops emitting laser, in the handwriting mode, the laser pen simulates the handwriting pen to realize near distance control. At this time, the protective cover 8 slides to the front end of the pen barrel 4, and the capacitive touch press key 6 and the press mechanical key are located inside the protective cover 8 to avoid accidental touch.

[0043] Or press the mechanical key, enter the remote control mode, the functions of the mechanical key include volume adjustment, page switching, projection parameter adjustment, stop pressing the mechanical key, exit the remote control mode.

[0044] The mouse mode, the handwriting mode and the remote control mode can exist independently, and one of the three modes can be used alone. When the finger touches the capacitive touch press key, the mouse mode is entered, and when the finger leaves the capacitive touch press key, the mouse mode is exited. When the finger presses the mechanical key, the remote control mode is entered, and when the finger leaves the mechanical key, the remote control mode is exited. When the front end of the laser pen is pressed on the projection area, the handwriting mode is entered, and when the front end of the laser pen is away from the projection area, the handwriting mode is exited. The switching of the three operation modes is realized, and at the same time, in the handwriting mode, the capacitive touch press key and the mechanical key are protected by the protective cover and will not trigger the other two operation modes. When using the laser pen, the user generally operates with one hand, and generally does not press the capacitive touch press key and the mechanical key at the same time. If the two are pressed at the same time, the mouse mode and the remote control mode can be executed at the same time, that is, the laser module remotely emits laser, and at the same time, remote control operation is performed through the remote control mode.

[0045] In the mouse mode and the handwriting mode, the step of emitting laser to the projection area by the laser module includes: The exposure value of the image sensor is fixed as 4.5%-5.5% of the maximum exposure value of the image sensor, specifically, the maximum exposure value is 5%, in the common window sunlight scene, the laser spot image obtained by the camera through the infrared light filter is collected, and the average pixel intensity of the laser spot image is calculated, which is recorded as the empirical reference value La.

[0046] Collect the laser spot image obtained in the current environment, calculate the average pixel intensity of the laser spot image, and record it as the current value L.

[0047] Calculate the ratio of the current value L to the reference value La, and record it as Ra.

[0048] If Ra> ,in , The smaller the value, the more sensitive it is. If the value is 0.8, it is determined that the external light has an impact, and the laser module emits a flashing laser with a flashing frequency of 10fps; otherwise, it is determined that there is no impact, and the laser module emits a continuous laser.

[0049] The time duration of pressing the laser pen tip cap 9 to trigger the pressure sensor or the time duration of pressing the capacitive touch key 6 is recorded as ,when Less than When , a click event or a point drawing operation is triggered; when Greater than When , a sliding event or a mobile writing operation is triggered; , the specific option is 100ms. Both handwriting mode and flying mouse mode can trigger click events, point drawing operations when writing, trigger sliding events and mobile writing operations. When the position of the laser spot in the projection screen is a clickable area, when the pressure sensor or capacitive touch key is pressed for less than When the laser spot in the projection screen is located in a background area, when the pressure sensor or capacitive touch key is pressed for less than When the laser spot in the projection screen is located in a sliding area, when the pressure sensor or capacitive touch key is pressed for longer than When the laser spot in the projection screen is located in a background area, when the pressure sensor or capacitive touch key is pressed for longer than When , draw a line in the area to realize mobile writing.

[0050] In the flying mouse mode and handwriting mode, the Kalman filter algorithm is used to continuously estimate the center coordinates of the laser spot. The state vector of the Kalman filter algorithm includes 、 、 and ,in and is the position coordinate of the center point of the laser spot at time t, and is the velocity term of the laser spot at time t; in a period of time, if continuous state estimation can be achieved, the camera 3 increases the frame rate output, if continuous state estimation cannot be achieved, or the trajectory of the center point of the laser spot obtained by continuous state estimation jumps and increases, the camera returns to the low frame rate full screen detection mode.

[0051] In the process of continuous state estimation, the dwell detection algorithm is used to record the time when the center point of the laser spot stays in a certain circular area, and when the dwell time , the point is marked as a candidate click point; wherein , the specific selection is 0.5s.

[0052] The bidirectional LSTM time series model or the GRU time series model is connected to the full connection classification layer to predict the best click point; the jitter during clicking usually produces a ring-shaped trajectory, and the directionality and movement degree are more explicit during sliding; during the triggering of the click event, the video of the user's operation of the laser pen is collected, and for the t-th frame of the operation video, a 7-dimensional feature vector is extracted and calculated:

[0053] In the formula: is the position of the center point of the laser spot in the t-th frame, is the velocity of the center point of the laser spot in the t-th frame, is the acceleration of the center point of the laser spot in the t-th frame, is the trajectory angle change of the center point of the laser spot in the current frame relative to the previous frame.

[0054] The trajectory data in the video is recorded, and the best click frame in the video is marked in the frame playback mode: a binary classification label is configured for each frame , if the t-th frame is the best click frame, then ; if the t-th frame is a non-click frame, then ; the 7-dimensional feature vector sequence of the continuous N frames is taken as the input of the overall model, so that the overall model outputs the probability that the t-th frame is the best click frame; the overall model is a bidirectional LSTM time series model or a GRU time series model connected to a full connection classification layer; the of the corresponding frame is taken as a supervision label to train the overall model; the trained overall model is used to predict the feature sequence of the real-time video stream, and output the probability that each frame is the best click frame, and the best click frame with the maximum probability is selected as the best click target frame, and the spot center point of the frame is taken as the predicted click point; when the distance between the candidate click point and the predicted click point is less than a set value, the candidate click point is the best click point.

[0055] The predicted click point is combined with the candidate click point to obtain the best click point, thereby avoiding shaking of the laser spot caused by user hand shaking and improving operation accuracy.

[0056] The laser spot in the projection area is identified by an infrared recognition module.

[0057] The coordinates of the laser spot in the lens coordinate system are mapped into the UI coordinate system by a perspective transformation matrix.

[0058] In the UI coordinate system, which regions are clickable regions, which regions are background regions, which regions are slideable regions, and whether the background region is a handwriting application are obtained according to a set name or by a visual model in combination with a foreground active page ID in a system rendering bottom layer. The UI region features do not need to be acquired in real time, and are acquired again when the UI page characteristics change, such as an increase in the number of clickable regions.

[0059] The visual model is obtained by using an existing UI parser model or training a special UI detection model. Specifically, a plurality of UI interface images of an intelligent system on a projection device are collected, and clickable regions such as buttons and text boxes are labeled by a rectangular box. A common YOLO target detection model is trained so that these set regions can be detected in the inference stage. When the laser point is in the clickable region, it is determined to be clicked, and when it is in a non-clickable region, it is determined to be slid or written. When the background region is a handwriting application interface, it is determined to be written.

[0060] After the laser spot projected by the user is projected in the UI coordinate system, the user controls the movement track of the laser spot to realize interaction with the projection interaction device. The projection interaction device displays the current position of the laser spot in the form of a pointer in the projectable picture, facilitating user operation.

[0061] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-mode projection screen interaction device, characterized in that: The invention comprises a projection device and a laser pen, wherein the projection device comprises a housing (1), a projection optical machine (2), a camera (3), a main control chip and a processing module, wherein the projection optical machine (2) and the camera (3) are mounted on the housing (1), the main control chip and the processing module are mounted in the housing (1), the projection optical machine (2) and the camera (3) are both connected to the main control chip, and the main control chip is connected to the processing module; the laser pen comprises a pen body (4), a laser module, a micro-control unit and a trigger assembly, wherein the pen body (4) is a hollow structure, the laser module and the micro-control unit are mounted inside the pen body (4), the laser module is connected to the micro-control unit, the trigger assembly comprises a rod body (10) having an axial through hole and a sensor assembly, wherein the rod body (10) is at least partially penetrated through the front end of the pen body (4), the rear end of the rod body (10) is connected to the sensor assembly, and the sensor assembly is connected to the micro-control unit, and the laser emitted by the laser module can pass through the rod body (10) along the axial direction of the rod body (10).

2. The multi-mode projection screen interaction device according to claim 1, characterized in that: An IRCUT dual filter switch is installed on the optical path between the lens of the camera (3) and the image sensor. The filters in the IRCUT dual filter switch include a visible light filter and an infrared light filter.

3. The multi-mode projection screen interaction device according to claim 1, characterized in that: The laser pen further comprises a Bluetooth module and a capacitive touch press key (6) and / or a mechanical key (7) for input control, wherein the capacitive touch press key (6) comprises a press key (13), a press bracket (14) and a touch pad (15), wherein the press key (13) is arranged in the pen body (4), the press bracket (14) is located above the press key (13) and is slidably fitted on the side wall of the pen body (4), and rebound force arms (16) are provided on both sides of the press bracket (14), the fixed end of the rebound force arm (16) is connected to the side wall of the pen body (4), and the touch pad (15) is installed on the press bracket (14); the Bluetooth module, the press key (13), the touch pad (15) and the mechanical key (7) are all connected to the micro-control unit, the Bluetooth module can interact with the main control chip, and the mechanical key (7) is installed on the pen body (4).

4. The multi-mode projection screen interaction device according to claim 3, characterized in that: A protective cover (8) is slidably mounted on the pen shaft (4) along the length direction of the pen shaft (4), and the capacitive touch press key (6) and the mechanical key (7) are located on a moving path of the protective cover (8).

5. A multi-mode projection screen interaction method implemented based on the device according to any one of claims 1 to 4, characterized in that: The following steps are involved: S1. Project the image onto the projection area through a projection light machine; The S2.IRCUT dual filter switcher switches to the visible light filter, ensuring the camera can capture the projected image. The camera focuses the projected image to obtain a clear image. The projector performs keystone correction based on the clear image, outputting a neat rectangular projection area. S3. The processing module identifies the four corner markers of the projection area and determines the position of the projection area in the lens coordinate system according to the four corner markers; constructs a perspective transformation matrix from the lens coordinate system to the UI coordinate system based on at least four pairs of non-collinear corresponding points of the four corner markers in the lens coordinate system and the projection interaction device UI coordinates; S4. After the perspective transformation matrix is ​​constructed, the IRCUT dual-filter switcher switches to the infrared filter, and the camera captures the laser spot within the projection area. S5. Touch the capacitive touch button of the laser pen to enter the flying mouse mode. The laser module emits laser light into the projection area. Stop touching the capacitive touch button of the laser pen to exit the flying mouse mode. The laser module stops emitting laser light into the projection area. In flying mouse mode, the laser pen simulates a mouse for long-distance control. Alternatively, press the tip of the laser pen on the projection area to enter the handwriting mode. The laser module emits laser light according to the signal of the sensor component. Stop pressing the tip of the laser pen on the projection area to exit the handwriting mode. The laser module stops emitting laser light. In handwriting mode, the laser pen simulates a stylus for close-range control. S6. Map the coordinates of the laser spot in the lens coordinate system to the UI coordinate system through the perspective transformation matrix.

6. A multi-mode projection screen interaction method according to claim 5, characterized in that: In the step of the laser module emitting laser light into the projection area: The exposure value of the image sensor is pre-fixed to 4.5%-5.5% of its maximum exposure value. In a common window-side sunlight scene, the laser spot image obtained by the camera through the infrared filter is collected. The average pixel intensity of the laser spot image is calculated and recorded as the empirical reference value La. Collect the laser spot image obtained in the current environment, calculate the average pixel intensity of the laser spot image, and record it as the current value L; Calculate the ratio of the current value L to the reference value La, recorded as Ra; If Ra > , it is determined that the external light has an impact, and the laser module emits a flashing laser; otherwise it is determined that there is no impact, and the laser module emits a continuous laser; .

7. The multi-mode projection screen interaction method according to claim 5, characterized in that: The time duration for the front end of the laser pen to press the trigger sensor component or the time duration for the capacitive touch key to be pressed is recorded as ,when Less than When , a click event or a point drawing operation is triggered; when Greater than When , a sliding event or a mobile writing operation is triggered; .

8. A multi-mode projection screen interaction method according to claim 7, characterized in that: In the step of emitting laser light by the laser module, the center coordinates of the laser spot are continuously estimated using the Kalman filter algorithm. The state vector of the Kalman filter algorithm includes 、 、 and ,in and is the position coordinate of the center point of the laser spot at time t, and is the velocity term of the laser spot at time t. Within a period of time, if continuous state estimation can be achieved, the camera increases the frame rate output. If continuous state estimation cannot be achieved, or the trajectory of the laser spot center point obtained by continuous state estimation increases, the camera returns to low frame rate full-screen detection mode.

9. The multi-mode projection screen interaction method according to claim 8, characterized in that: In the process of continuous state estimation, the dwell detection algorithm is used to record the time the center point of the laser spot stays in a circular area. Then mark the point as a candidate click point; ; During the process of triggering the click event, a video of the user holding the laser pen is collected, and for the tth frame of the operation video, a 7-dimensional feature vector is extracted and calculated: Where: is the position of the center point of the laser spot in the t frame, is the speed of the laser spot center in the t frame, is the acceleration of the center point of the laser spot in the tth frame, The trajectory angle change of the laser spot center point in the current frame relative to the previous frame; Mark the best click frames in the video by frame playback; Configure binary classification labels for each frame , if the tth frame is the best click frame, then ; If the tth frame is a non-click frame, then ; The 7-dimensional feature vector sequence of N consecutive frames is used as the input of the overall model, so that the overall model outputs the probability that the t-th frame is the best click frame. The overall model is a time series model connected to a fully connected classification layer. The overall model is trained using whether the corresponding frame is the best click frame as a supervision label. Use the trained overall model to predict the feature sequence of the real-time video stream and output the probability of each frame being the best click frame , select the best click frame with the maximum probability as the best click target frame, and take the center point of the light spot of the frame as As a predicted click point; When the distance between the candidate click point and the predicted click point is less than the set value, the candidate click point is the optimal click point.

10. The multi-mode projection screen interaction method according to claim 5, characterized in that: The step S5 further includes: pressing a mechanical button to enter a remote control mode, where the functions of the mechanical button include volume adjustment, page switching, and projection parameter adjustment; stopping pressing the mechanical button exits the remote control mode.

Images