A multi-mode projected picture interaction device and method

Abstract

The present application relates to intelligent projection interaction technology, in particular to a multi-mode projection picture interaction device and method. The device contains a projection device and a laser pen, the projection device has a shell, a projection light machine and a camera, the laser pen includes a pen barrel, a laser module and the like, and a press type trigger assembly is arranged in the pen barrel. The projection light machine projects a picture, and the camera can identify the picture and laser. The laser pen can switch between a mouse mode, a handwriting mode and a remote control mode: the mouse mode uses a capacitive touch button and the laser module emits laser; the handwriting mode controls the laser module by pressing the trigger assembly; and the remote control mode realizes basic operation by a mechanical button. Through laser spot positioning and multi-mode switching, the convenience and fluency of interaction are improved, external equipment is not needed, cost is reduced, and the operation habit is matched.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent projection interaction, in particular to a multi-mode projection picture interaction device and method. BACKGROUND

[0002] Under the current wave of digital office, the demand for mobility and convenience of office scenes continues to rise, and this trend has profoundly reshaped the design logic and functional boundaries of office equipment. Intelligent projection equipment, with its lightweight and easy-to-carry core advantages, has quickly gained popularity in various conference and seminar, teaching and training scenarios, and has gradually become an important choice to replace traditional fixed large screens.

[0003] However, the limitations of existing intelligent projection equipment in interactive experience have become increasingly prominent, becoming a key bottleneck restricting its further penetration. At the operation level, most products lack direct touch interaction capability, and users have to rely on external mouse, dedicated remote control or additional handwriting board to complete instruction input. This operation method is significantly disconnected from the natural interaction habit of "directly touching the screen" formed by people on smartphones and tablets, not only increasing the operation learning cost, but also easily affecting efficiency in emergency demonstration scenarios due to device adaptation problems. SUMMARY

[0004] In order to solve the technical problems that the existing technology cannot directly interact with the projection equipment and the interaction mode is single, the present application provides a multi-mode projection picture interaction device and method.

[0005] To this end, the present application provides the following technical solutions.

[0006] A multi-mode projection picture interaction device, comprising a projection device and a laser pen, the projection device comprising a shell, a projection light machine, a camera, a main control chip and a processing module, the projection light machine and the camera being installed on the shell, the main control chip and the processing module being installed in the shell, the projection light machine and the camera being connected with the main control chip, and the main control chip being connected with the processing module; the laser pen comprising a pen barrel, a laser module, a micro control unit and a trigger assembly, the pen barrel being a hollow structure, the laser module and the micro control unit being installed inside the pen barrel, the laser module being connected with the micro control unit, the trigger assembly comprising a rod body with an axial through hole and a sensor assembly, the rod body being at least partially arranged in the front end of the pen barrel, the rear end of the rod body being connected with the sensor assembly, the sensor assembly being connected with the micro control unit, and the laser emitted by the laser module being capable of passing through the rod body along the axial direction of the rod body;

[0007] An IRCUT double filter switcher is installed on the optical path between the lens and the image sensor of the camera, and the filters in the IRCUT double filter switcher include visible light filters and infrared light filters;

[0008] The laser pen further comprises a Bluetooth module, a capacitive touch press key and / or a mechanical key for input control, the capacitive touch press key comprises an event trigger press key, a press bracket and a touch plate, the event trigger press key is arranged in the pen barrel, the press bracket is located above the press key and is slidingly fitted to the side wall of the pen barrel, and the touch plate is installed on the press bracket; the mechanical key is installed on the pen barrel, the Bluetooth module, the event trigger press key, the touch plate and the mechanical key are connected with the micro control unit, and the Bluetooth module can interact with the master control chip.

[0009] The press bracket is provided with a resilient arm on each side, and the fixed end of the resilient arm is connected with the side wall of the pen barrel.

[0010] Further, a protective cover is slidingly installed on the pen barrel along the length direction of the pen barrel, and the capacitive touch press key and the mechanical key are located on the moving path of the protective cover.

[0011] A multi-mode projection picture interaction method, comprising the following steps:

[0012] S1. projecting a projection picture to a projection area by a projection light machine;

[0013] S2. switching the IRCUT double filter switcher to a visible light filter to ensure that the camera can capture the projection picture; the camera focuses on the projection picture to obtain a clear picture; the projection light machine completes trapezoidal correction based on the clear picture to output a regular rectangular projection area;

[0014] S3. the processing module identifies four corner mark points of the projection picture, determines the position of the projection area in the lens coordinate system according to the four corner mark points, and constructs a perspective transformation matrix from the lens coordinate system to the UI coordinate system according to at least four pairs of non-collinear corresponding points of the four corner mark point coordinates in the lens coordinate system and the UI coordinate system;

[0015] S4. after the perspective transformation matrix is constructed, the IRCUT double filter switcher is switched to an infrared light filter, and the camera obtains a laser spot in the projection area;

[0016] S5. touching the capacitive touch press key of the laser pointer to enter the mouse mode, the laser module emits laser to the projection area, stopping touching the capacitive touch press key of the laser pointer to exit the mouse mode, the laser module stops emitting laser to the projection area, in the mouse mode, the laser pointer simulates a mouse to realize remote control; or pressing the front end of the laser pointer on the projection area to enter the handwriting mode, the laser module emits laser according to the signal of the sensor assembly, stopping pressing the front end of the laser pointer on the projection area to exit the handwriting mode, the laser module stops emitting laser, in the handwriting mode, the laser pointer simulates a handwriting pen to realize close-range control;

[0017] S6. mapping the coordinates of the laser spot in the lens coordinate system to the UI coordinate system through a perspective transformation matrix.

[0018] Further, in the step of emitting laser by the laser module:

[0019] Pre-fix the exposure value of the image sensor as 4.5%-5.5% of its maximum exposure value, in a common window sunlight scene, collect the laser spot image obtained by the camera through the infrared light filter, calculate the average pixel intensity of the laser spot image, and record it as the empirical reference value La;

[0020] Collect the laser spot image obtained in the current environment, calculate the average pixel intensity of the laser spot image, and record it as the current value L;

[0021] Calculate the ratio of the current value L and the reference value La, and record it as Ra;

[0022] If Ra> 1, then it is determined that the external light has an impact, and the laser module emits flashing laser; otherwise, it is determined that there is no impact, and the laser module emits continuous laser; wherein .

[0023] Further, the pressing time length of the pen tip sleeve of the laser pen to trigger the pressure sensor or the pressing time length of the capacitive touch press key is recorded as When is less than , a click event or a drawing point operation during writing is triggered; when is greater than , a sliding event or a moving writing operation is triggered; wherein .

[0024] Further, in the step of emitting laser by the laser module, the Kalman filtering algorithm is used for continuous state estimation of the center coordinates of the laser spot, and the state vector of the Kalman filtering algorithm includes , , and , wherein and are the position coordinates of the center point of the laser spot at t frame, and are the velocity terms of the laser spot at t frame; within a period of time, if continuous state estimation can be achieved, the camera increases the frame rate output, if continuous state estimation cannot be achieved, or the trajectory of the center point of the laser spot obtained by continuous state estimation jumps and increases, the camera returns to the low frame rate full screen detection mode.

[0025] ​Further, in the process of continuously estimating the state, a dwell detection algorithm is used to record the time when the center point of the laser spot stays in a certain circular area, and when the dwell time the point is marked as a candidate click point; wherein .

[0026] In the process of triggering the click event, a video of the user's operation of the laser pointer is collected, and for the t-th frame of the operation video, a 7-dimensional feature vector is extracted and calculated:

[0027]

[0028] In the formula: is the position of the center point of the laser spot in the t-th frame, is the speed of the center point of the laser spot in the t-th frame, is the acceleration of the center point of the laser spot in the t-th frame, is the change in the trajectory angle of the center point of the laser spot in the current frame relative to the previous frame;

[0029] The best click frame in the video is marked in a frame playback manner;

[0030] A binary classification label is configured for each frame If the t-th frame is the best click frame, then If the t-th frame is a non-click frame, then ;

[0031] The 7-dimensional feature vector sequence of the continuous N frames is taken as the input of the overall model, so that the overall model outputs the probability that the t-th frame is the best click frame The overall model is a time sequence model connected with a fully connected classification layer; the overall model is trained with the corresponding frame as the supervision label whether it is the best click frame or not;

[0032] The feature sequence of the real-time video stream is predicted by using the trained overall model, and the probability that each frame is the best click frame is output The best click frame with the maximum probability is selected as the best click target frame, and the spot center point of the frame is taken as the predicted click point;

[0033] When the distance between the candidate click point and the predicted click point is less than a set value, the candidate click point is the best click point.

[0034] Further, the step S4 further comprises: after the perspective transformation matrix is constructed, a mechanical button is pressed to enter a remote control mode, and the functions of the mechanical button include volume adjustment, page switching, and projection parameter adjustment; the mechanical button is stopped pressing to exit the remote control mode.

[0035] Advantages and positive effects of the present application.

[0036] The application directly interacts with the laser pen through the projection device without external equipment, reduces the operation complexity and cost, and has multiple interaction modes such as remote interaction mode, remote control mode and handwriting mode. The technical problem that the prior art cannot directly interact with the projection device and the interaction mode is single is solved. The projection light machine projects a projection picture in a projection area, the camera and the visible light filter are used to realize the focusing of the camera on the projection picture and the trapezoidal correction of the projection light machine, and a perspective transformation matrix from the lens coordinate system to the UI coordinate system is constructed; the camera and the infrared light filter are used to identify the laser spot in the projection picture, when the capacitive touch press key is pressed, the laser pen enters the mouse mode, the laser module emits laser to the projection area at a long distance to realize long-distance control; when the mechanical key is pressed, the laser pen enters the remote control mode; when the laser pen enters the handwriting mode, the front end of the laser pen is pressed on the projection area, the opening and closing of the laser module is controlled by the sensor assembly, the laser module emits laser to the projection area at a short distance to realize short-distance control; the laser spot obtained by the camera is mapped to the UI coordinate system of the projection device to realize the interaction between the laser pen and the projection device. Through laser spot positioning combined with multi-mode switching, the laser pen can not only point, but also realize remote precise interaction in the mouse mode, which is consistent with the natural operation habit; the remote control mode can meet the basic control demand; the handwriting mode realizes direct writing and marking in the projection area by controlling the laser with the sensor assembly, and the convenience and fluency of the interaction are improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0038] Figure 1 A structural diagram of a projection device of a multi-mode projection picture interaction device provided by the present application.

[0039] Figure 2 A principle diagram of a projection device of a multi-mode projection picture interaction device provided by the present application.

[0040] Figure 3 A structural diagram of a laser pen of a multi-mode projection picture interaction device provided by the present application.

[0041] Figure 4 A trigger assembly structure diagram of a laser pen of a multi-mode projection picture interaction device provided by the present application.

[0042] Figure 5An exploded view of the triggering component of a laser pointer in a multi-mode projection screen interaction device provided by the present invention.

[0043] Figure 6 An exploded view of the capacitive touch-sensitive button of the laser pen in a multi-mode projection screen interaction device provided by the present invention.

[0044] Figure 7 The flowchart is a multi-mode projection screen interaction method provided by the present invention.

[0045] In the diagram: 1. Housing; 2. Projector engine; 3. Camera; 4. Pen barrel; 5. Laser aperture; 6. Capacitive touch button; 7. Mechanical button; 8. Protective cover; 9. Pen tip cover; 10. Pen body; 11. PCB board; 12. Pressure sensor; 13. Event-triggered button; 14. Press bracket; 15. Touchpad; 16. Rebound arm; 17. Touch glass cover. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] This invention provides a multi-mode projection screen interaction device, such as... Figures 1-3 As shown, the device includes a projection device and a laser pointer. The projection device includes a housing 1, a projection optical engine 2, a camera 3, a main control chip, and a processing module. The projection optical engine 2 and the camera 3 are mounted on the housing 1, and the main control chip and the processing module are mounted inside the housing 1. The projection optical engine 2 and the camera 3 are both connected to the main control chip, and the main control chip is connected to the processing module.

[0048] The laser pointer includes a pen barrel 4, a laser module, and a microcontroller unit. The pen barrel 4 has a hollow structure, and the laser module and microcontroller unit are installed inside the pen barrel 4. The laser module is connected to the microcontroller unit. The front end of the pen barrel 4 is equipped with a detachable pen tip sleeve 9 via a threaded connection. The detachable pen tip sleeve 9 facilitates maintenance of the internal structure of the laser pointer.

[0049] It also includes triggering components, one form of which is... Figures 4-5As shown, the pen includes a rod 10 with an axial through hole and a sensor assembly, the rod 10 is at least partially arranged in the front end of the pen tip sleeve 9, the rear end of the rod 10 is connected with the sensor assembly, the sensor assembly is connected with the micro control unit, the rod 10 is a hollow structure, and the laser emitted by the laser module can pass through the rod 10 along the axial direction of the rod 10. The sensor assembly includes a PCB board 11 and a pressure sensor 12, the rear end of the rod 10 is connected with the PCB board 11, the pressure sensor 12 is installed between the rod 10 and the PCB board 11, and the PCB board 11 is installed in the pen barrel 4. When the pen tip sleeve 9 is pressed on the projection area, the front end of the rod 10 will be subjected to the pushing force applied by the projection area, and the rear end of the rod 10 will press the PCB board 11, the PCB board 11 will be slightly deformed, the pressure sensor 12 receives the deformation signal of the PCB board 11 and transmits the signal to the micro control unit, and then controls the laser module to emit laser to project in the projection area.

[0050] Another form of the trigger assembly is that the pen tip sleeve 9 is provided with a laser hole 5, the laser emitted by the laser module can pass through the laser hole 5, and a pressure sensor is installed between the pen tip sleeve 9 and the pen barrel 4. Pressing the pen tip sleeve 9 triggers the pressure sensor, and then controls the laser module to emit laser to project in the projection area through the micro control unit.

[0051] An IRCUT dual filter switch is installed on the optical path between the lens and the image sensor of the camera 3, and the filters in the IRCUT dual filter switch include visible light filters and infrared light filters. The lens and the image sensor are inherent components of the camera, both of which belong to the current standardized components and can be directly adapted for use. The installed IRCUT dual filter switch is also an existing device, which is usually composed of two filters, a power mechanism, a shell and a circuit control board. The power mechanism is controlled by the circuit control board to move to realize the switching of the two filters. The visible light filter can allow visible light of 380~760nm to pass through, while filtering out infrared light with a wavelength >760nm, avoiding color deviation caused by infrared interference, and ensuring the color accuracy of the projection picture obtained by the camera. The infrared light filter only allows specific infrared waveband light of 850nm±5nm or 940nm±5nm to pass through, filters out all non-target wavebands such as visible light and other wavelengths of infrared light, thereby shielding environmental light interference and ensuring that the camera can accurately identify the corresponding wavelength laser spot emitted by the laser pen.

[0052] The optical axis of the lens is parallel to the projection direction of the projection light machine 2, and the optical axis of the lens is directly opposite the picture area projected on the wall surface by the projection light machine, and the optical focal plane of the lens is strictly coplanar with the photosensitive surface of the image sensor.

[0053] The laser pen further comprises a Bluetooth module and a capacitive touch press key 6 and / or a mechanical key 7 for input control, the Bluetooth module and the mechanical key 7 are connected with the micro-control unit, the Bluetooth module can interact with the master control chip, and the capacitive touch press key 6 and the mechanical key 7 are both mounted on the pen barrel 4. Figure 6 As shown in the figure, the capacitive touch press key 6 comprises an event trigger press key 13, a press bracket 14 and a touch plate 15, the event trigger press key 13 is arranged in the pen barrel 4, the press bracket 14 is located above the event trigger press key 13 and is slidingly fitted on the side wall of the pen barrel 4, both sides of the press bracket 14 are provided with a rebounding arm 16, the fixed end of the rebounding arm 16 is connected with the side wall of the pen barrel 4, the touch plate 15 is mounted on the press bracket 14, and a touch glass cover plate 17 is mounted on the touch plate 15, which protects the touch plate 15. The event trigger press key 13 and the touch plate 15 are both connected with the micro-control unit. When the user touches the touch glass cover plate 17, the touch plate 15 recognizes the electric field of the human body, the touch plate 15 triggers the laser module to continuously emit laser, and when the user presses the touch glass cover plate 17, the press bracket 14 moves towards the event trigger press key 13, the event trigger press key 13 is pressed, a click event is triggered, and the press bracket 14 is reset through the rebounding arm 16 after being released.

[0054] A protective cover 8 is slidingly mounted on the pen barrel 4 along the length direction of the pen barrel 4, and the capacitive touch press key 6 and the mechanical key are located on the moving path of the protective cover 8.

[0055] A multi-mode projection picture interactive method based on a multi-mode projection picture interactive device, as shown in the figure, comprises the following steps: Figure 7

[0056] S1. Project a projection picture to a projection area through a projection light machine.

[0057] S2. Switch the IRCUT double filter switcher to a visible light filter to ensure that the camera can capture the projection picture; focus the camera on the projection picture to obtain a clear picture; and complete trapezoidal correction based on the clear picture by the projection light machine to output a regular rectangular projection area.

[0058] S3. Identify four corner mark points of the projection picture through a visible light module, determine the position of the projection area in a lens coordinate system according to the four corner mark points, and construct a perspective transformation matrix from the lens coordinate system to a UI coordinate system according to at least four pairs of non-collinear corresponding points of the four corner mark point coordinates in the lens coordinate system and the UI coordinate system.

[0059] S4. After the perspective transformation matrix is constructed, switch the IRCUT double filter switcher to an infrared light filter, and obtain a laser spot in the projection area through the camera.

[0060] ​S5. Touch the capacitive touch press key 6 of the laser pen, enter the mouse mode, the laser module emits laser to the projection area, stop touching the capacitive touch press key of the laser pen, exit the mouse mode, the laser module stops emitting laser to the projection area, in the mouse mode, the laser pen simulates the mouse to realize remote control. At this time, the protective cover 8 slides to the rear end of the pen barrel 4, and the capacitive touch press key 6 and the press mechanical key are located outside the protective cover 8. The mouse mode refers to that the laser pen can simulate the mouse to operate, the laser module emits laser to the projection area to simulate the cursor displayed on the screen by the mouse, and the capacitive touch press key simulates the press mouse key.

[0061] Or press the front end of the laser pen on the projection area, enter the handwriting mode, the laser module emits laser according to the signal of the sensor assembly, stop pressing the front end of the laser pen on the projection area, exit the handwriting mode, the laser module stops emitting laser, in the handwriting mode, the laser pen simulates the handwriting pen to realize near distance control. At this time, the protective cover 8 slides to the front end of the pen barrel 4, and the capacitive touch press key 6 and the press mechanical key are located inside the protective cover 8 to avoid accidental touch.

[0062] Or press the mechanical key, enter the remote control mode, the functions of the mechanical key include volume adjustment, page switching, projection parameter adjustment, stop pressing the mechanical key, exit the remote control mode.

[0063] The mouse mode, the handwriting mode and the remote control mode can exist independently, and one of the three modes can be used alone. When the finger touches the capacitive touch press key, the mouse mode is entered, and when the finger leaves the capacitive touch press key, the mouse mode is exited. When the finger presses the mechanical key, the remote control mode is entered, and when the finger leaves the mechanical key, the remote control mode is exited. When the front end of the laser pen is pressed on the projection area, the handwriting mode is entered, and when the front end of the laser pen is away from the projection area, the handwriting mode is exited. The switching of the three operation modes is realized, and at the same time, in the handwriting mode, the capacitive touch press key and the mechanical key are protected by the protective cover and will not trigger the other two operation modes. When the user uses the laser pen, he generally operates with one hand, and generally does not press the capacitive touch press key and the mechanical key at the same time. If the two are pressed at the same time, the mouse mode and the remote control mode can be executed at the same time, that is, the laser module remotely emits laser, and at the same time, remote control operation is performed through the remote control mode.

[0064] In the mouse mode and the handwriting mode, the step of emitting laser to the projection area by the laser module includes:

[0065] The exposure value of the image sensor is fixed as 4.5%-5.5% of the maximum exposure value of the image sensor, specifically, the maximum exposure value is 5%, in the common window sunlight scene, the laser spot image obtained by the camera through the infrared light filter is collected, the average pixel intensity of the laser spot image is calculated, and the average pixel intensity is recorded as the empirical reference value La.

[0066] Collecting the laser spot image obtained under the current environment, calculating the average pixel intensity of the laser spot image, denoted as the current value L.

[0067] Calculate the ratio of the current value L and the reference value La, denoted as Ra.

[0068] If Ra> 0.8, the external light has an impact, and the laser module emits flashing laser with a flashing frequency of 10 fps; otherwise, it is determined that there is no impact, and the laser module emits continuous laser. Wherein , The smaller the value of , the more sensitive the effect is. If is 0.8, it is determined that the external light has an impact, and the laser module emits flashing laser with a flashing frequency of 10 fps; otherwise, it is determined that there is no impact, and the laser module emits continuous laser.

[0069] The length of time when the tip of the laser pen 9 is pressed to trigger the pressure sensor or the length of time when the capacitive touch press key 6 is pressed is denoted as When is less than , the click event or the dot drawing operation during writing is triggered; when is greater than , the sliding event or the moving writing operation is triggered; wherein , the specific selection is 100 ms. Both the handwriting mode and the mouse mode can trigger the click event, the dot drawing operation during writing, the sliding event and the moving writing operation. When the position of the laser spot in the projection image is a clickable area, when the length of time when the pressure sensor or the capacitive touch press key is pressed is less than , a click operation is performed on the area; when the position of the laser spot in the projection image is a background area, when the length of time when the pressure sensor or the capacitive touch press key is pressed is less than , a dot is drawn in the area. When the position of the laser spot in the projection image is a slidable area, when the length of time when the pressure sensor or the capacitive touch press key is pressed is greater than , the area is dragged to slide; when the position of the laser spot in the projection image is a background area, when the length of time when the pressure sensor or the capacitive touch press key is pressed is greater than , a line is drawn in the area to realize moving writing.

[0070] In the mouse mode and the handwriting mode, the Kalman filtering algorithm is used to continuously estimate the center coordinates of the laser spot. The state vector of the Kalman filtering algorithm includes , , and , wherein and are the position coordinates of the center point of the laser spot at t frame, and is the speed term of the laser spot in the t frame; in a period of time, if continuous state estimation can be achieved, the camera 3 increases the frame rate output, if continuous state estimation cannot be achieved, or the trajectory of the center point of the laser spot obtained by continuous state estimation jumps and increases, the camera returns to the low frame rate full screen detection mode.

[0071] In the process of continuous state estimation, the dwell detection algorithm is used to record the time when the center point of the laser spot stays in a certain circular area, and when the dwell time the point is marked as a candidate click point; wherein , the specific selection is 0.5s.

[0072] The bidirectional LSTM time series model or the GRU time series model is connected to the full connection classification layer to predict the best click point; the click jitter usually produces a ring-shaped trajectory, and the sliding direction and movement degree are more clear; in the process of triggering the click event, the video of the user's operation of the laser pen is collected, and for the t frame of the operation video, a 7-dimensional feature vector is extracted and calculated:

[0073]

[0074] In the formula: is the position of the center point of the laser spot in the t frame, is the speed of the center point of the laser spot in the t frame, is the acceleration of the center point of the laser spot in the t frame, is the trajectory angle change of the center point of the laser spot in the current frame relative to the previous frame.

[0075] Record the trajectory data in the video, and mark the best click frame in the video in the frame playback mode: configure a binary classification label for each frame , if the t frame is the best click frame, then ; if the t frame is a non-click frame, then ; the 7-dimensional feature vector sequence of the continuous N frames is taken as the input of the overall model, so that the overall model outputs the probability that the t frame is the best click frame; the overall model is a bidirectional LSTM time series model or a GRU time series model connected to a full connection classification layer; the of the corresponding frame is taken as a supervision label to train the overall model; the trained overall model is used to predict the feature sequence of the real-time video stream, and output the probability that each frame is the best click frame, and the best click frame with the maximum probability is selected as the best click target frame, and the spot center point of the frame is taken as the predicted click point; when the distance between the candidate click point and the predicted click point is less than a set value, the candidate click point is the best click point.

[0076] The predicted click point is combined with the candidate click point to obtain the optimal click point, thereby avoiding shaking of the laser spot caused by user hand shaking and improving operation accuracy.

[0077] The laser spot in the projection area is identified by an infrared recognition module.

[0078] The coordinates of the laser spot in the lens coordinate system are mapped into the UI coordinate system by a perspective transformation matrix.

[0079] In the UI coordinate system, which region is a clickable region, which region is a background region, which region is a slideable region, and whether the background region is a handwriting application are obtained according to a set name or by a visual model in combination with a foreground active page ID in the system rendering bottom layer. The UI region features do not need to be acquired in real time, and are acquired again when the UI page characteristics change, such as an increase in the number of clickable regions.

[0080] The visual model is obtained by using an existing UI parser model or training a special UI detection model. Specifically, a plurality of UI interface images of the intelligent system on the projection device are collected, and the clickable regions such as buttons and text boxes are labeled by a rectangular box. A common YOLO target detection model is trained so that the set regions can be detected in the inference stage. When the laser point is in the clickable region, it is determined to be clicked, and when it is in a non-clickable region, it is determined to be slid or written. When the background region is a handwriting application interface, it is determined to be written.

[0081] After the laser spot projected by the user is projected in the UI coordinate system, the user controls the movement track of the laser spot to realize interaction with the projection interaction device. The projection interaction device displays the current position of the laser spot in the form of a pointer in the projectable picture, facilitating user operation.

[0082] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-mode projected picture interactive device, characterized in that, The application relates to a projection device and a laser pen, the projection device comprising a shell (1), a projection light machine (2), a camera (3), a main control chip and a processing module, the projection light machine (2) and the camera (3) being installed on the shell (1), the main control chip and the processing module being installed in the shell (1), the projection light machine (2) and the camera (3) being connected with the main control chip, and the main control chip being connected with the processing module; the laser pen comprising a pen barrel (4), a laser module, a micro control unit and a trigger assembly, the pen barrel (4) being a hollow structure, the laser module and the micro control unit being installed inside the pen barrel (4), the laser module being connected with the micro control unit, the trigger assembly comprising a rod body (10) with an axial through hole and a sensor assembly, the rod body (10) being at least partially arranged in the front end of the pen barrel (4), the rear end of the rod body (10) being connected with the sensor assembly, the sensor assembly being connected with the micro control unit, and the laser emitted by the laser module being capable of penetrating the rod body (10) along the axial direction of the rod body (10); An IRCUT double filter switcher is arranged on the optical path between the lens and the image sensor of the camera (3), and the filters in the IRCUT double filter switcher comprise visible light filters and infrared light filters; The laser pen further comprises a Bluetooth module, a capacitive touch press key (6) and a mechanical key (7) for input control, the capacitive touch press key (6) comprising an event trigger press key (13), a press bracket (14) and a touch plate (15), the event trigger press key (13) being arranged in the pen barrel (4), the press bracket (14) being arranged above the event trigger press key (13) and being slidably connected with the side wall of the pen barrel (4), and the touch plate (15) being arranged on the press bracket (14); the mechanical key (7) is arranged on the pen barrel (4), the Bluetooth module, the event trigger press key (13), the touch plate (15) and the mechanical key (7) are connected with the micro control unit, and the Bluetooth module can interact with the main control chip.

2. The multi-mode projected picture interactive device according to claim 1, wherein, Resilient arms (16) are arranged on the two sides of the press bracket (14), and the fixed ends of the resilient arms (16) are connected with the side wall of the pen barrel (4).

3. The multi-mode projected picture interactive device of claim 1, wherein, A protective cover (8) is slidably arranged on the pen barrel (4) along the length direction of the pen barrel (4), and the capacitive touch press key (6) and the mechanical key (7) are arranged on the moving path of the protective cover (8).

4. A multi-modal projected picture interaction method implemented by the apparatus according to any one of claims 1-3, characterized in that, The application further discloses a method for using the projection device and the laser pen, and the method comprises the following steps: S1. projecting a projection picture to a projection area by the projection light machine; S2. switching the IRCUT double filter switcher to the visible light filter to ensure that the camera can capture the projection picture, focusing the camera on the projection picture to obtain a clear picture, and completing trapezoidal correction based on the clear picture by the projection light machine to output a regular rectangular projection area; S3. identifying four corner mark points of the projection area by the processing module, determining the position of the projection area in the lens coordinate system according to the four corner mark points, and constructing a perspective transformation matrix from the lens coordinate system to the UI coordinate system according to at least four pairs of non-collinear corresponding points of the four corner mark point coordinates in the lens coordinate system and the projection interactive device UI coordinates. S4. After the perspective transformation matrix is constructed, the IRCUT dual filter switch is switched to the infrared light filter, and the camera is used to acquire the laser spot in the projection area; S5. The capacitive touch press key of the laser pen is pressed to enter the mouse mode, the laser module emits laser to the projection area, the capacitive touch press key of the laser pen is stopped to exit the mouse mode, the laser module stops emitting laser to the projection area, in the mouse mode, the laser pen simulates a mouse to realize remote control; or the front end of the laser pen is pressed on the projection area to enter the handwriting mode, the laser module emits laser according to the signal of the sensor assembly, the front end of the laser pen is stopped to press on the projection area to exit the handwriting mode, the laser module stops emitting laser, in the handwriting mode, the laser pen simulates a handwriting pen to realize close-range control; S6. The coordinates of the laser spot in the lens coordinate system are mapped into the UI coordinate system through the perspective transformation matrix.

5. The multi-modal projected picture interaction method of claim 4, wherein, In the step of the laser module emitting laser to the projection area: The exposure value of the image sensor is fixed as 4.5%-5.5% of the maximum exposure value, in the common window sunlight scene, the laser spot image acquired by the camera through the infrared light filter is collected, the average pixel intensity of the laser spot image is calculated, and the average pixel intensity is recorded as an empirical reference value La; The laser spot image acquired in the current environment is collected, the average pixel intensity of the laser spot image is calculated, and the average pixel intensity is recorded as a current value L; The ratio of the current value L and the reference value La is calculated, and the ratio is recorded as Ra; If Ra , then it is determined that the ambient light has an influence, and the laser module emits flickering laser; otherwise, it is determined that there is no influence, and the laser module emits continuous laser; wherein .

6. The multi-modal projected picture interaction method of claim 4, wherein, The length of pressing the front end of the laser pen to trigger the sensor assembly or the length of pressing the capacitive touch key is recorded as When is less than , a click event or a drawing point operation is triggered when writing; when is greater than , a sliding event or a moving writing operation is triggered; wherein .

7. The multi-modal projected picture interaction method of claim 6, wherein, In the step of emitting laser light by the laser module, the center coordinates of the laser spot are continuously estimated by using a Kalman filtering algorithm, a state vector of the Kalman filtering algorithm comprises 、 、 and , wherein and are position coordinates of the center point of the laser spot at t frame, and are velocity terms of the laser spot at t frame; in a period of time, if continuous state estimation can be achieved, the camera increases the frame rate output, if continuous state estimation cannot be achieved, or the trajectory of the center point of the laser spot obtained by continuous state estimation jumps and increases, the camera returns to a low frame rate full screen detection mode.

8. The multi-modal projected picture interaction method of claim 7, wherein, In the process of continuous state estimation, a dwell detection algorithm is used to record the time when the center point of the laser spot stays in a certain circular area, and when the dwell time the point is marked as a candidate click point; wherein ; In the process of triggering the click event, the video of the user holding the laser pen operation is collected, for the t-th frame of the operation video, a 7-dimensional feature vector is extracted and calculated: In the formula: is the position of the laser spot center point in the tth frame, is the velocity of the laser spot center point in the tth frame, is the acceleration of the laser spot center point in the tth frame, is the trajectory angle change of the laser spot center point in the current frame relative to the previous frame; The best click frame in the video is marked in the frame playback mode; configuring a binary classification label for each frame , if the t-th frame is the best click frame, then ; If the tth frame is a non-click frame, then ; The 7-dimensional feature vector sequence of continuous N frames is taken as an input of an overall model, and the overall model outputs a probability that the t-th frame is the optimal click frame The overall model is a time sequence model connected with a full connection classification layer, and the overall model is trained with the supervision label of whether the corresponding frame is the optimal click frame The trained overall model is used to predict the feature sequence of a real-time video stream, and the probability of each frame being the best click frame is output The best click target frame with the maximum probability is selected, and the spot center point of the frame is used as the predicted click point ​ When the distance between the candidate click point and the predicted click point is less than a set value, the candidate click point is the best click point.

9. The multi-modal projected picture interaction method of claim 4, wherein, The step S5 further includes: pressing the mechanical key to enter the remote control mode, the functions of the mechanical key include volume adjustment, page switching, and projection parameter adjustment, and the remote control mode is exited by stopping pressing the mechanical key.

Images