Projection interaction method based on infrared light spots, projection equipment and storage medium

Through the projection interaction method based on infrared spots, the target coordinates of the infrared spot in the projection original image are calculated and the light spots are displayed, which solves the problem of low operating efficiency of remote control buttons during projector operation, and improves user experience and operation efficiency.

CN119011792APending Publication Date: 2024-11-22深圳市当智科技有限公司
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410968448.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When using a projector, users need to use the remote control to control the projection screen, which affects the operation experience and efficiency.

Method used

Using the projection interaction method based on infrared spots, the infrared spot coordinates and conversion matrix in the infrared shooting screen are obtained, and the target coordinates of the infrared spot in the projection original image are calculated, and the light spots are displayed at this position, thereby realizing any control operation.

Benefits of technology

It improves the user's experience and efficiency in projector operation, and realizes precise control of the projected image through the visualization of light spots, reducing the dependence on the physical buttons of the remote control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011792A_ABST
    Figure CN119011792A_ABST
Patent Text Reader

Abstract

The invention discloses an infrared facula-based projection interaction method, projection equipment and a storage medium, and the method comprises the following steps: obtaining an infrared shooting picture, and recognizing the coordinates of infrared facula in the infrared shooting picture; a main conversion matrix between the infrared shooting picture and the original projection picture is obtained, the main conversion matrix is composed of a first sub-conversion matrix and a second sub-conversion matrix, and the first sub-conversion matrix is a conversion matrix between the infrared shooting picture and the visible light shooting picture; the second sub-conversion matrix is a conversion matrix between the visible light shooting picture and the projection original picture; based on the infrared light spot coordinates and the main conversion matrix, obtaining target coordinates corresponding to the infrared light spot coordinates in the projection original image; and displaying the light spot at the position of the target coordinate in the projection original image. According to the invention, the light spot can be displayed at the position corresponding to the infrared light spot in the projection picture, so that any required control operation can be further realized based on the light spot.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of projection equipment, and in particular to a projection interaction method based on infrared light spots, a projection equipment and a storage medium. Background Art

[0002] With the popularity of projectors, users are fond of using large-size projection screens for entertainment and work. However, in the process of operating the projector, it is inevitable to control the projected screen, such as moving the selection box, selecting the controls in the screen, clicking patterns, sliding, etc. These operations can be completed with a mouse on the computer, but when using a projector, these control instructions generally require users to use a remote control to implement them, and use the physical buttons of the remote control to move the selection box to the specified screen area, which affects the user's operating experience and efficiency. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a projection interaction method, projection device and storage medium based on infrared light spots, which can display light spots at the positions corresponding to the infrared light spots in the projection screen, so that any required control operations can be further realized based on the light spots.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] In a first aspect, the present invention discloses a projection interaction method based on infrared light spots, comprising the following steps:

[0006] Acquire an infrared shooting picture, and identify the coordinates of the infrared spot in the infrared shooting picture;

[0007] Acquire a main conversion matrix between the infrared shooting picture and the projection original image, wherein the main conversion matrix is ​​composed of a first sub-conversion matrix and a second sub-conversion matrix, wherein the first sub-conversion matrix is ​​a conversion matrix between the infrared shooting picture and the visible light shooting picture, and the second sub-conversion matrix is ​​a conversion matrix between the visible light shooting picture and the projection original image;

[0008] Based on the infrared spot coordinates and the main conversion matrix, obtain the target coordinates corresponding to the infrared spot coordinates in the projection original image;

[0009] A light spot is displayed at the position of the target coordinates in the projection original image.

[0010] Preferably, the step of acquiring a main conversion matrix between the infrared photographed image and the projection original image comprises:

[0011] Acquire a preset first sub-conversion matrix between the infrared shooting picture and the visible light shooting picture;

[0012] Projecting the feature map onto a target projection surface and acquiring a visible light shooting picture, wherein the visible light shooting picture includes the feature map;

[0013] Obtaining a second sub-transformation matrix based on the coordinates of the feature map in the original projection image and the visible light captured image respectively;

[0014] A main conversion matrix is ​​obtained according to the first sub-conversion matrix and the second sub-conversion matrix.

[0015] Preferably, before obtaining the target coordinates corresponding to the infrared spot coordinates in the projection original image based on the infrared spot coordinates and the main conversion matrix, the method further includes: obtaining the projection mapping area corresponding to the projection original image in the infrared shooting screen according to the vertex coordinates of the projection original image and the inverse matrix operation of the main conversion matrix;

[0016] The method of obtaining the target coordinates corresponding to the infrared spot coordinates in the projection original image based on the infrared spot coordinates and the main conversion matrix includes: when the infrared spot coordinates are located in the projection mapping area, obtaining the target coordinates corresponding to the infrared spot coordinates in the projection original image based on the infrared spot coordinates and the main conversion matrix.

[0017] Preferably, before obtaining the target coordinates corresponding to the infrared spot coordinates in the projection original image based on the infrared spot coordinates and the main conversion matrix, the method includes:

[0018] Acquire a first real-time motion parameter of an inertial sensor in a remote controller that emits an infrared light spot; calculate theoretical light spot coordinates of the infrared light spot in the infrared shooting picture according to the infrared light spot coordinates, the first real-time motion parameter and a dead reckoning algorithm; use the theoretical light spot coordinates as new infrared light spot coordinates; or,

[0019] Acquire the current projection distance, the first spatial attitude angle and the preset shooting picture viewing angle of the projector, and establish a first spatial coordinate system, wherein the first spatial coordinate system includes an infrared shooting area corresponding to the infrared shooting picture, and infrared mapping coordinates corresponding to the infrared spot coordinates; acquire the distance from the wall and the second spatial attitude angle of the remote control that emits the infrared spot; obtain the starting coordinates and starting orientation of the remote control in the first spatial coordinate system according to the infrared mapping coordinates, the distance from the wall and the second spatial attitude angle; acquire the second real-time motion parameters of the remote control, and calculate the theoretical spot coordinates of the infrared spot in the infrared shooting picture based on the starting coordinates, the starting orientation and the second real-time motion parameters in the first spatial coordinate system; and use the theoretical spot coordinates as the new infrared spot coordinates.

[0020] Preferably, after acquiring the main conversion matrix between the infrared shooting picture and the projection original image, the method includes:

[0021] Based on the main transformation matrix and the projection area in the projection original image, determining a projection mapping area corresponding to the projection area in the captured image;

[0022] When the theoretical light spot coordinates are outside the projection mapping area; or when the theoretical light spot coordinates are within the projection mapping area but cannot be identified in the infrared shooting picture, the remote controller is instructed to reduce the brightness of the emitted infrared light spot.

[0023] Preferably, after acquiring the main conversion matrix between the infrared shooting picture and the projection original image, the method includes:

[0024] Obtaining a projection mapping area corresponding to the projection original image in the infrared shooting picture according to the vertex coordinates of the projection original image and the inverse matrix operation of the main transformation matrix;

[0025] When the theoretical light spot coordinates are located within the projection mapping area, and the infrared light spot coordinates cannot be identified from the infrared shooting picture within a second threshold time, the remote controller is instructed to turn off or reduce the emission brightness of the light spot.

[0026] Preferably, before acquiring the infrared shooting picture and identifying the coordinates of the infrared spot in the infrared shooting picture, the following steps are also included:

[0027] Acquire the projection distance and the pitch angle of the projector, and calculate the initial pitch angle range of the remote control by combining the projection distance, the pitch angle of the projector and the preset use distance of the remote control emitting the infrared light spot;

[0028] Acquire the current pitch angle of the remote controller, and determine whether the current pitch angle of the remote controller is within the initial pitch angle range;

[0029] When the current pitch angle of the remote controller is within the initial pitch angle range, driving the remote controller to emit a light spot at a transmission power lower than a first threshold;

[0030] An infrared shooting picture is acquired, and when the coordinates of the infrared light spot in the infrared shooting picture are identified, the remote controller is instructed to increase the brightness of the emitted light spot.

[0031] Preferably, after displaying the light spot at the position of the target coordinates in the projection original image, the method further comprises:

[0032] Get the button control instructions of the remote control;

[0033] An operation corresponding to the key control instruction is performed at a position corresponding to the target coordinates in the projection original image.

[0034] Preferably, after acquiring the infrared shooting picture and identifying the coordinates of the infrared spot in the infrared shooting picture, the method includes:

[0035] The attitude angle change parameter of the inertial sensor in the remote controller is obtained, and when the real-time attitude angle change parameter is less than a third threshold, the change of the infrared spot coordinate is ignored.

[0036] In a second aspect, the present invention discloses a projection device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program can be read and executed by the processor to implement the projection interaction method as described in the first aspect.

[0037] In a third aspect, the present invention discloses a storage medium, characterized in that a computer program is stored in the storage medium, wherein the computer program is configured to be executable by a processor to execute the projection interaction method described in the first aspect.

[0038] Compared with the prior art, the beneficial effect of the present invention lies in: the projection interaction method based on infrared light spots disclosed in the present invention identifies the coordinates of the infrared light spots according to the infrared shooting picture, and obtains the main conversion matrix between the infrared shooting picture and the projection original picture, so as to combine the infrared light spot coordinates with the main conversion matrix to obtain the target coordinates corresponding to the infrared light spot coordinates in the projection original picture, and display the light spot at the position of the target coordinates, so that the light spot can be displayed at the position corresponding to the infrared light spot in the projection picture, so that any required control operation can be further realized based on the light spot. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the hardware structure of the remote control;

[0040] Figure 2 It is a schematic diagram of the hardware structure of the projector;

[0041] Figure 3 It is a flow chart of the projection interaction method based on infrared light spots disclosed in the first embodiment of the present invention;

[0042] Figure 4 It is a schematic diagram of the mapping relationship between the infrared shooting screen coordinate system, the visible light shooting screen coordinate system and the projection original image coordinate system;

[0043] Figure 5 It is a schematic diagram of the movement of light spots in the infrared shooting picture;

[0044] Figure 6 It is a schematic diagram of the position relationship of the remote control relative to the target projection surface;

[0045] Figure 7 It is a schematic diagram of the positional relationship between the projector, remote control and infrared shooting screen;

[0046] Figure 8 A schematic diagram of the hardware structure of a remote controller according to another embodiment;

[0047] Fig. 9 is the remote control's pitch field of view;

[0048] Fig.10 It is a schematic diagram of the positional relationship between the projector and the remote control relative to the projected image. DETAILED DESCRIPTION

[0049] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0050] It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and circuit / signal communication.

[0051] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0052] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0053] Embodiment 1 of the present invention discloses a projection interaction method based on infrared light spots, wherein in this embodiment, infrared light spots are emitted by a remote controller, and the hardware structure diagram of the remote controller 10 is shown in FIG. Figure 1 As shown, the corresponding hardware structure diagram of the projector 20 is as follows Figure 2 shown.

[0054] The remote controller 10 mainly includes an infrared laser 11, a power drive circuit 12, a Bluetooth remote control SOC (System on Chip) chip 13 and a first IMU (Inertial Measurement Unit) unit 14; wherein the power drive circuit 12 is used to drive the infrared laser 11 to emit infrared light. In one embodiment of the present scheme, after the position of the infrared light spot in the projection screen is identified by the infrared camera on the projector, the projection screen displays a bright spot at the position to represent the position of the infrared light spot, thereby visualizing the infrared light spot. The Bluetooth remote control SOC chip 13 can control the on and off of the infrared laser 11 through the power drive circuit 12, as well as the length of time of on, that is, the pulse width of the on PWM waveform; the first IMU unit 14 is used to detect the posture of the remote controller 10; the remote controller 10 can also establish a Bluetooth connection with the projector 20 through the Bluetooth remote control SOC chip 13. Specifically, the Bluetooth remote control SOC chip 13 is an integrated circuit with a dedicated purpose, which integrates modules such as the processor (CPU), graphics processing unit (GPU), memory controller (Memory Controller), and wireless radio frequency chip (RF); the remote control 10 generally extracts and calculates various data through the CPU, and sends and receives wireless signals through the wireless radio frequency chip (including the Bluetooth module).

[0055] The projector 20 mainly includes a first camera 21, a second camera 22, an optical machine 23, a SOC chip 24, a BT module 25, and a second IMU unit 26. Among them, the first camera 21 is a visible light camera that can shoot RGB three-color visible light images; the second camera 22 is an infrared camera with an infrared filter, which can image infrared light of a specific band; the BT (BlueTooth) module 25 is used to establish a Bluetooth connection with the remote control 10, which is used for the key value / voice data / status synchronous communication of the remote control; the second IMU unit 26 is used to detect changes in the three spatial attitude angles (pitch angle, rotation angle and flight angle) of the projector 20, which is used for the calculation of the projector 20's own automatic trapezoidal / autofocus function. Among them, the filter band of the second camera 22 can be a narrow band, which only passes infrared rays of a specific band, such as 980nm and 850nm; it can also be a broadband filter, which filters out the visible light band and retains the near-infrared band, such as filtering out visible light less than 780nm and retaining the infrared band above 780nm; the light spot scattered by the infrared laser emitted by the infrared laser 11 on the wall / curtain can be easily imaged by the infrared camera (second camera 22), and the trajectory of the infrared light spot can be identified by comparing the moving light spot between the previous and next two frames of images.

[0056] like Figure 3As shown, the present invention implements a disclosed projection interaction method based on infrared light spots, comprising the following steps:

[0057] S1: Acquire an infrared shooting picture and identify the coordinates of the infrared spot in the infrared shooting picture;

[0058] S2: Obtain the main conversion matrix between the infrared shooting picture and the projection original picture, the main conversion matrix is ​​composed of a first sub-conversion matrix and a second sub-conversion matrix, the first sub-conversion matrix is ​​the conversion matrix between the infrared shooting picture and the visible light shooting picture, which represents the coordinate conversion relationship between the same light point position in the infrared shooting picture and the visible light shooting picture; the second sub-conversion matrix is ​​the conversion matrix between the visible light shooting picture and the projection original picture, which represents the conversion relationship between a coordinate in the visible light shooting picture and its mapping coordinate in the projection original picture;

[0059] Wherein, obtaining the red-primary conversion matrix in step S2 specifically includes:

[0060] S21: Acquire a preset first sub-conversion matrix between the infrared shooting picture and the visible light shooting picture;

[0061] The infrared camera is used to capture the light spot scattered by the low-power infrared laser, while the visible light camera is used for the auto-focus and auto-trapezoidal functions of the projection. In order to obtain the correspondence between the light spot coordinates recognized by the infrared camera and the projection optical machine screen coordinates, the infrared camera and the visible light camera are calibrated with external parameters in advance to obtain the conversion matrix H1 from the infrared camera screen coordinates to the visible light camera screen coordinates; the conversion matrix H1 is the first sub-conversion matrix, and the conversion matrix H1 can be obtained by any of the following two specific steps:

[0062] (1) A spatial coordinate system can be established by the distance between the infrared camera and the visible light camera on the projector, as well as their respective field of view angles. Based on these known hardware parameters, the spatial relative position relationship between the shooting range of the infrared camera and the shooting range of the visible light camera can be determined from the spatial coordinate system, and the conversion matrix H1 can be calculated based on this. For example, the coordinates of the four vertices of the infrared shooting picture in the spatial coordinate system can be obtained based on the field of view angle parameters of the infrared camera. Similarly, the coordinates of the four vertices of the visible light shooting picture in the spatial coordinate system can be obtained based on the known optical parameters. By taking one of the vertices of the infrared shooting picture as the origin of the coordinate system, an infrared shooting picture coordinate system is established, and the coordinates of the visible light shooting picture in the infrared shooting picture coordinate system can be obtained, so that the conversion matrix H1 between the infrared shooting picture coordinate system and the visible light shooting picture coordinate system can be established.

[0063] (2) Before the equipment leaves the factory, the coordinates of the light spot at the same position on the wall in the infrared shooting picture and the visible light shooting picture are measured. According to the coordinates of this light spot in the two shooting pictures, a mapping matrix H1 can be established, such as: Z 可见 =H1*Z 红外 , Z 可见 is the coordinate of the light spot in the visible light shooting picture, Z 红外 is the coordinate of the light spot in the infrared shooting picture.

[0064] S22: Projecting the feature map onto the target projection surface and acquiring a visible light shooting picture, where the visible light shooting picture includes the feature map;

[0065] S23: obtaining a second sub-transformation matrix based on the coordinates of the feature map in the original projection image and the visible light captured image;

[0066] In this step, the conversion matrix H2 from the visible light camera screen coordinates to the projection screen coordinates after the projector performs trapezoidal correction is obtained. The conversion matrix H2 is the second sub-conversion matrix. When the projector performs trapezoidal correction, it is generally necessary to project feature images such as a chessboard / QR code, which need to be photographed by a visible light camera. The following specific steps can be used to obtain the conversion matrix H2:

[0067] When the host of the projector is moved, the projector may be tilted relative to the screen. At this time, the projector projects a chessboard calibration image. After shooting with the visible light camera, the coordinates of the chessboard corner points are identified and compared with the coordinates of the original points on the chessboard. The conversion matrix H2 from the visible light camera image coordinates to the projection image coordinates is calculated.

[0068] That is Z 原图 =H2*Z 可见 .

[0069] The infrared shooting screen coordinate system and the visible light shooting screen coordinate system can use a vertex of the shooting screen as the coordinate system origin, and the XOZ plane coincides with the shooting screen. The coordinates of other vertices of the shooting screen in the shooting coordinate system are set according to the resolution value of the shooting screen. Similarly, the projection original image coordinate system can use a vertex of the projection original image as the coordinate system origin, and the XOZ plane coincides with the projection original image. The coordinates of other vertices of the projection original image in the projection original image coordinate system are set according to the resolution value of the projection original image.

[0070] Assuming that the infrared camera identifies the coordinate of the infrared laser spot in the infrared shooting screen coordinate system as Z0, the mapping coordinate of Z0 in the projection original image coordinate system is Z1 = H2 × H1 × Z0; Figure 4As shown, the coordinates of the light spot A in the infrared shooting screen coordinate system 100 are calculated by the conversion matrix H1 to obtain A1 in the visible light shooting screen coordinate system 200, and the coordinates of A1 are calculated by the conversion matrix H2 to obtain A2 in the projection original image coordinate system 300; the coordinates of the light spot B are calculated by the conversion matrix H1 to obtain B1 in the visible light shooting screen coordinate system 200, and the coordinates of B1 are calculated by the conversion matrix H2 to obtain B2 in the projection original image coordinate system 300. It can be seen that A is a light spot outside the projection screen 400, and B is a light spot inside the projection screen 400. The projection original image 500 is the image to be projected in the projector. The projection original image 500 in the projection original image coordinate system 300 has been subjected to trapezoidal correction. Although it is trapezoidal, the image after being projected onto the target projection surface will appear as a rectangle; the feature map 600 can be, for example, a checkerboard calibration map.

[0071] S24: Obtain a main conversion matrix according to the first sub-conversion matrix and the second sub-conversion matrix.

[0072] Through the transformation matrices H1 and H2, the four vertices C11, C12, C13, and C14 in the projection original image 500 can be respectively converted into C01, C02, C03, and C04 in the infrared shooting screen coordinate system 100, so the processor can also directly determine the position of the light spot in the infrared shooting screen coordinate system 100.

[0073] S3: Based on the infrared spot coordinates and the main transformation matrix, the target coordinates corresponding to the infrared spot coordinates in the projection original image are obtained;

[0074] In some embodiments, before executing step S3, the projection mapping area corresponding to the projection original image in the infrared shooting screen is obtained based on the vertex coordinates of the projection original image and the inverse matrix operation of the main transformation matrix; correspondingly, step S3 includes: when the infrared spot coordinates are located in the projection mapping area, based on the infrared spot coordinates and the main transformation matrix, the target coordinates corresponding to the infrared spot coordinates in the projection original image are obtained.

[0075] like Figure 4 As shown, the area covered by the projection original image 500 in the projection original image coordinate system 300 is the projection area. Combined with the inverse matrix of the main conversion matrix, the corresponding projection mapping area in the shooting coordinate system can be determined, wherein the projection mapping area is the area covered by the projection screen 400 in the infrared shooting screen coordinate system 100; further based on the coordinates of the light spot B and the main conversion matrix, the target coordinates corresponding to the light spot B2 in the projection original image 500 can be obtained. For example, by operating the coordinates of C11, C12, C13, and C14 in the projection original image coordinate system with the inverse matrix of the main conversion matrix, the corresponding coordinates C01, C02, C03, and C04 in the shooting coordinate system can be obtained.

[0076] In some embodiments, before executing step S3, the theoretical spot coordinates of the infrared spot in the infrared shooting picture are calculated and used as new infrared spot coordinates. The theoretical spot coordinates of the infrared spot in the infrared shooting picture can be calculated in any of the following two ways.

[0077] The first method includes the following steps:

[0078] The first real-time motion parameter of the inertial sensor in the remote controller is obtained; and the theoretical spot coordinates of the infrared spot in the infrared shooting picture are calculated according to the infrared spot coordinates, the first real-time motion parameter and the dead reckoning algorithm.

[0079] In a specific example, the remote control emits an infrared laser, and the infrared camera of the projector obtains the infrared shooting picture. After the screen coordinates of the infrared laser spot are identified from the infrared shooting picture, the spot track can be calculated by integration calculation combined with the IMU real-time parameters reported by the remote control through the Bluetooth channel. The IMU real-time parameters include linear acceleration and angular velocity. By integrating the linear acceleration and angular velocity over time, the spatial movement distance of the remote control and the rotation angle around three axes, namely the three attitude angles, can be calculated respectively.

[0080] refer to Figure 5 , the first IMU unit on the remote control 10 can measure changes in its angle and acceleration. When the IMU in the remote control 10 recognizes that its spatial position is always changing within a range, it is assumed that the spatial position of the remote control remains unchanged. Then a corresponding relationship between the moving distance of the light spot and the changing angle of the remote control can be established, that is, the ratio of the angle change of the remote control when the light spot moves from K1 to K2 is obtained: k = d1 / θ1. Specifically, the user presses the calibration button of the remote control for the first time. At this time, the infrared camera of the projector shoots the infrared spot K1 emitted by the remote control and records the current angle 1; then the user slightly changes the direction of the remote control and presses the calibration button again. At this time, the infrared camera of the projector shoots the infrared spot K2 emitted by the remote control and records the current angle 2. Angle 2 minus angle 1 is θ1, and the lateral offset between K2 and K1 in the infrared shooting picture is d1.

[0081] Figure 5 The example above only uses the lateral movement, and the ratio k can also be used to calculate the longitudinal movement. In order to further improve the accuracy, the ratio of the longitudinal spot offset to the angle change can also be calibrated and calculated separately using the method in the previous paragraph, and different k values ​​are used for the subsequent longitudinal and lateral spot offset calculations.

[0082] Furthermore, by multiplying the change angle △θ of the remote control by the ratio k, the change △d of the light spot position can be obtained, that is, △d=k*△θ. For example, d2=k*θ2, d1=k*θ1; from K1 to K2 to K3, each calculation cycle can be a preset value, that is, each cycle can measure the latest coordinates of the light spot in the infrared shooting screen coordinate system. In multiple cycles, the remote control moves from K1→K2→K3→K4 in sequence, and the lateral angle changes θ1, θ2, θ3 measured by the remote control IMU are combined with the previously calculated k value, so that the lateral offsets d1, d2, d3 can be known. Then, the lateral offsets such as d1, d2, d3 are accumulated on the basis of the K1 coordinate recorded at the beginning of the infrared shooting screen, and finally the theoretical lateral coordinates of the light spot (such as the K4 position) can be calculated.

[0083] In another specific example, other existing methods of performing track estimation through IMU integration may also be used, which is not limited to the integration method in the above example.

[0084] The second method includes the following steps:

[0085] The current projection distance, the first spatial attitude angle and the preset shooting picture viewing angle of the projector are obtained to establish a first spatial coordinate system, wherein the first spatial coordinate system includes an infrared shooting area corresponding to the infrared shooting picture and infrared mapping coordinates corresponding to the infrared spot coordinates; the distance from the wall and the second spatial attitude angle of the remote control are obtained; according to the infrared mapping coordinates, the distance from the wall and the second spatial attitude angle, the starting coordinates and the starting direction of the remote control in the first spatial coordinate system are obtained; the second real-time motion parameters of the remote control are obtained, and based on the starting coordinates, the starting direction and the second real-time motion parameters in the first spatial coordinate system, the theoretical spot coordinates of the infrared spot in the infrared shooting picture are calculated.

[0086] In this calculation method, TOF sensors need to be set on the remote control and the projector respectively. The TOF sensor on the remote control predicts whether the remote control is pointing at the projection screen, and controls the laser to turn on to protect the infrared laser from accidentally hurting the human eye.

[0087] The TOF sensors installed on the remote control and projector can calculate the distance from the wall (the distance from the target projection surface) respectively, and emit two symmetrical beams of light with a known angle. Figure 6As shown, the angle φ0 between the two beams of light is known, and the distance L1 and the distance L2 can be calculated according to the time of light return. According to these parameters, the inclination angle of the remote control 10 relative to the target projection surface 30 in the horizontal direction, that is, the flight angle φ1 of the remote control 10 relative to the target projection surface 30 can be calculated. The distance L3 (the distance from the wall) can be measured by the TOF sensor. Combining the distance L3 and the flight angle φ1, plus the pitch angle and rotation angle measured by the first IMU unit in the remote control, the spatial coordinates of the remote control 10 relative to the light spot P can be accurately calculated.

[0088] like Figure 7 As shown, the TOF sensor and the second IMU unit provided on the projector 20, as well as the known field of view angle of the infrared shooting picture 700, can establish a projector coordinate system with the optical center of the projector as the origin. In this coordinate system, the coordinates of the four vertices of the infrared shooting picture 700 can be calculated. Based on the light spot P identified in the infrared shooting picture 700 at the initial time, the initial position and orientation of the remote control 10 in the projector coordinate system can be deduced. Subsequently, the new position coordinates and orientation of the remote control 10 are calculated at regular intervals according to the changes in the three attitude angles and the distance from the wall of the remote control 10, and then the latest position of the light spot P relative to the projection area can be directly calculated by a simple spatial geometry method.

[0089] In this calculation method, the integral calculation process of the light spot track is omitted to avoid errors, thereby improving the calculation accuracy of the theoretical light spot coordinates. That is, the light spot is identified in the infrared shooting picture, and the coordinates of the light spot in the infrared shooting picture coordinate system are bound to the current position and attitude angle (flight angle, pitch angle, rotation angle) of the remote control.

[0090] In the above two processes of calculating the theoretical light spot coordinates in the shooting coordinate system, when the spatial position change of the remote controller exceeds the second threshold, the process returns to step S1 to re-identify the infrared light spot coordinates in the infrared shooting picture from the infrared shooting picture. Specifically, when the first IMU unit of the remote controller detects that it is in a stationary state, the laser emission can be turned off; when the spatial position change of the remote controller is calculated by the first IMU unit of the remote controller to exceed the second threshold, for example, when the remote controller exceeds 10 cm in any direction of the x, y, and z axes, the track calculation of the light spot coordinates is restarted.

[0091] In another embodiment, a TOF sensor may be used to determine the theoretical light spot coordinates. Specifically, Figure 8As shown, when the projector is equipped with a TOF sensor, the TOF sensor 15 in the remote controller 10 and the TOF sensor of the projector can use the same band, such as the 940nm band. The TOF sensor 15 in the remote controller 10 can emit a predefined coded signal. When the coded signal is recognized by the TOF sensor in the projector, it means that the remote controller 10 is pointing to the optical screen of the projector, and a signal is sent to the remote controller 10 via Bluetooth to restore its laser power to normal.

[0092] S4: Display the light spot at the target coordinate position in the projected original image.

[0093] The light spot is, for example, a circular light spot, a mouse-shaped light spot, etc., so that a pointer cursor of a corresponding pattern can be displayed in the projection picture projected by the projector.

[0094] Furthermore, in order to realize the interaction of the light spot position, after step S4, the following steps are included:

[0095] Get the button control instructions of the remote control;

[0096] The operation corresponding to the key control instruction is performed at the position corresponding to the target coordinates in the original projection image. The key instruction can be the key value generated by the corresponding key on the remote control. The remote control 10 is connected to the projector 20 via Bluetooth, and the key value can be reported to the projector host via Bluetooth BLE. After the key value is reported to the host, the projector executes. Among them, a certain key on the hardware of the remote control 10, such as the confirmation key (OK key), is defined as a touch control key. When the remote control 10 points to a certain position of the projection screen, the second camera 22 of the projector 20 recognizes the position of the scattered light spot and displays a prompt cursor at the corresponding coordinate position of the projection screen; further, when the OK key is pressed, it indicates that a touch down event is generated at the coordinate. When the OK key is pressed and moved, it indicates a touch move event. When the OK key pops up, it indicates a touch up event.

[0097] In some embodiments, the step S1 further includes the following steps:

[0098] A1: Get the projection distance and pitch angle of the projector, and calculate the initial pitch angle range of the remote control based on the projection distance, pitch angle and the preset remote control usage distance;

[0099] The first IMU unit of the remote control and the second IMU unit of the projector are both calibrated in the horizontal direction. Specifically, they can be calibrated before leaving the factory so that the remote control and the projector are placed on a horizontal plane for IMU calibration.

[0100] like Fig. 9 As shown, based on the default projection distance D and the pitch field angle θ of the projector (i.e., the optical and mechanical field angle of the projector), combined with Fig.10 , according to the preset use distance of the remote controller 10 (generally 2m), calculate the range of the initial pitch angle θk of the remote controller 10, for example, the value range of θk is 0-70°, within which the laser of the remote controller 10 can point to the projection screen of the projector; wherein, the pitch angle θk is the angle between the remote controller and the lines connecting the highest point and the lowest point of the projection screen height h at the preset use distance, and the pitch field angle θt is the angle between the projection angle of the projector in the vertical direction and the lines connecting the highest point and the lowest point of the projection screen height h. θt and θk can be calculated in advance.

[0101] A2: Get the current pitch angle of the remote controller and determine whether the current pitch angle of the remote controller is within the initial pitch angle range;

[0102] The current pitch angle of the remote controller is acquired through the first IMU unit of the remote controller, and it is determined whether the current pitch angle is within the range of the initial pitch angle θk.

[0103] A3: When the current pitch angle of the remote controller is within the initial pitch angle range, the remote controller is driven to emit a light spot at a transmission power lower than a first threshold;

[0104] When it is determined that the current pitch angle is within the range of the initial pitch angle θk, the infrared laser driving the remote control works normally through pulse conduction, but at this time the pulse width (duty cycle) can be controlled to the minimum value that the infrared camera can identify the heat dissipation spot to prevent the laser from being too bright when it is first emitted. The infrared camera on the projector can identify low-power infrared spots by increasing the exposure time.

[0105] A4: Obtain the infrared shooting picture, and when the infrared spot coordinates in the infrared shooting picture are identified, instruct the remote controller to increase the brightness of the emitted light spot.

[0106] Through steps A1 to A3, the irradiation angle and brightness of the laser when the remote control is initially started can be limited. Further, in step A4, when the infrared camera recognizes the pulsed laser spot, the normal exposure time is restored, and the remote control is notified through the Bluetooth channel to increase the laser emission power. At this time, the projector begins to normally recognize the screen coordinates of the spot.

[0107] In some embodiments, the method after step S1 includes: obtaining a real-time attitude angle change parameter of an inertial sensor in a remote controller, and when the real-time attitude angle change parameter is less than a third threshold, ignoring the change of the infrared spot coordinates.

[0108] Specifically, the remote control is equipped with a first IMU unit, and the inertial sensor in the first IMU unit can detect the angular velocity and linear acceleration of the remote control itself, and then the attitude angle change and displacement change of the remote control itself can be calculated by the time integration method. Secondly, an angle change threshold (i.e., the third threshold) can be set. When the change in any attitude angle (flight angle, pitch angle, rotation angle) measured in real time is less than the angle change threshold, the spatial change of the remote control is ignored, and the target coordinates in the original projection image are not changed, thereby achieving the purpose of anti-shake.

[0109] In some embodiments, step S2 includes:

[0110] B1: Based on the main transformation matrix and the projection area in the projection original image, determine the projection mapping area corresponding to the projection area in the captured image;

[0111] B2: When the theoretical light spot coordinates are outside the projection mapping area; or when the theoretical light spot coordinates are within the projection mapping area but cannot be identified in the infrared shooting picture, the remote controller is instructed to reduce the brightness of the emitted infrared light spot.

[0112] In step B2, the remote controller is specifically instructed to reduce the brightness of the emitted light spot in any of the following two situations: ① when the theoretical light spot coordinates are outside the projection mapping area; ② when the theoretical light spot coordinates are within the projection mapping area, but the light spot cannot be identified by the camera. Among them, when the theoretical light spot coordinates are outside the projection mapping area, the projector transmits a signal through the wireless communication module to instruct the remote controller to reduce the brightness of the emitted light spot. If the theoretical light spot coordinates are within the projection mapping area again, the remote controller is instructed to increase the brightness of the emitted light spot. Specifically, the projector sends a command to the remote controller to instruct the remote controller to restore the normal laser emission power.

[0113] In a specific example, when the infrared camera cannot recognize the infrared light spot emitted by the remote control, the light spot track is continuously calculated according to the above two methods of calculating the theoretical light spot coordinates in the shooting coordinate system. If it is found that the theoretical light spot coordinates exceed the projection screen range, an instruction is sent through the Bluetooth channel to the pointing remote control hardware to reduce the laser power or turn off the laser emission.

[0114] The coordinates of the projection image in the infrared shooting image can be calculated based on the conversion matrices H1 and H2 calculated above. Figure 4The coordinates of the vertices are (0,0), (1960,0), (0,1080), and (1960,1080). By substituting these coordinates into the inverse matrix operations of H2 and H1 in turn, the coordinates of these vertices in the coordinate system of the infrared shooting screen can be calculated, which can then be used to determine whether the theoretical spot coordinates calculated by the previous steps are in the projection screen.

[0115] When the infrared camera fails to recognize the infrared light spot and then re-recognizes the light spot coordinates in the infrared shooting picture, the above-mentioned integral calculation error is corrected according to the real coordinates recognized in the infrared shooting picture. For example, when the light spot is re-recognized in the infrared shooting picture, the track of the theoretical light spot coordinates is recalculated with the coordinates of the light spot as the starting coordinates.

[0116] Through steps B1 to B2, the laser in the remote control can be further made to work continuously only when it is pointed at the projection screen area, and the laser will not work when it is pointed at other directions; therefore, this solution can identify whether the infrared light spot leaves the projection screen area, and can automatically reduce or turn off the brightness of the light spot of the remote control after leaving the projection screen area to prevent damage to the human eye.

[0117] In some other embodiments, step S2 includes:

[0118] C1: According to the vertex coordinates of the projection original image and the inverse matrix operation of the main transformation matrix, the projection mapping area corresponding to the projection original image in the infrared shooting screen is obtained;

[0119] C2: When the theoretical light spot coordinates are within the projection mapping area, and the infrared light spot coordinates cannot be identified from the infrared shooting picture within the second threshold time, the remote control is instructed to turn off or reduce the emission brightness of the light spot.

[0120] When the track of the calculated theoretical light spot coordinates is within the projection screen, but the infrared camera cannot identify the infrared light spot within the second threshold time (such as within 1s), the remote control is instructed to turn off the laser emission. The second threshold is calculated based on the laser emission frequency, and is limited to being harmless to the human eye; specifically, when the laser emission power is large, the brightness of the emitted laser is high, and the maximum time allowed to irradiate the human eye should be short, otherwise it will damage the human eye; when the calculated theoretical light spot coordinates are within the projection screen, but the infrared camera cannot identify the light spot, it is considered that the light spot may irradiate the eyes of the user standing in the projection screen area, so the timing starts at this time. When this state lasts for more than the second threshold time, the remote control is instructed to reduce the brightness of the emitted light spot to prevent the laser from irradiating the human eye for a long time.

[0121] In the projection interaction method based on infrared light spots disclosed in the first embodiment of the present invention, a laser light source (infrared light) irradiates a target projection surface (wall / curtain, etc.) to scatter a light spot, and an infrared camera captures the light spot position, calculates the position of the light spot position in the infrared shooting screen, and then calculates the target coordinates of the infrared light spot in the projection original image through the corresponding conversion matrix, and displays the light spot at the position of the target coordinates. Furthermore, by emitting an infrared laser from the infrared laser in the remote control to irradiate the projection screen of the projector and identifying the reflected light spot, the absolute coordinates (theoretical light spot coordinates) of the pointing point in the projection screen can be directly and accurately calculated; after obtaining the absolute coordinates of the pointing projection original image coordinate system, the corresponding touch indicator cursor can be displayed on the coordinates, and in conjunction with the remote control button, the pointing touch function can be realized through the reporting of key events (down / up) and absolute coordinates.

[0122] Embodiment 2 of the present invention discloses a projection device, including a processor and a memory, wherein a computer program is stored in the memory, and the computer program can be read and executed by the processor to implement the projection interaction method in embodiment 1.

[0123] The memory is a storage device that stores programs and data executed by the processor in a non-volatile manner. The memory is composed of semiconductor storage elements such as magnetic storage devices, flash read-only memories (ROM), or other types of non-volatile storage devices. The memory may also include random access memories (RAM) that constitute the working area of ​​the processor. The memory stores the control program executed by the processor. The processor may be composed of a single processor or a combination of multiple processing groups. The processor executes the control program to control various parts of the projection device.

[0124] Embodiment 3 of the present invention discloses a storage medium, in which a computer program is stored. The computer program is configured to be executable by a processor to execute the projection interaction method in embodiment 1.

[0125] Optionally, the storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0126] The background section of the present invention may contain background information about the problem or environment of the present invention, rather than describing the prior art by others. Therefore, the content contained in the background section is not an admission by the applicant that the prior art is available.

[0127] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of mutual contradiction, the technical personnel in this field can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope defined by the attached claims.

Claims

1. A projection interaction method based on infrared light spots, characterized in that: The following steps are involved: Acquire an infrared shooting picture, and identify the coordinates of the infrared spot in the infrared shooting picture; Acquire a main conversion matrix between the infrared shooting picture and the projection original image, wherein the main conversion matrix is ​​composed of a first sub-conversion matrix and a second sub-conversion matrix, wherein the first sub-conversion matrix is ​​a conversion matrix between the infrared shooting picture and the visible light shooting picture, and the second sub-conversion matrix is ​​a conversion matrix between the visible light shooting picture and the projection original image; Based on the infrared spot coordinates and the main conversion matrix, obtain the target coordinates corresponding to the infrared spot coordinates in the projection original image; A light spot is displayed at the position of the target coordinates in the projection original image.

2. The projection interaction method according to claim 1, characterized in that: The obtaining of the main conversion matrix between the infrared shooting picture and the projection original picture comprises: Acquire a preset first sub-conversion matrix between the infrared shooting picture and the visible light shooting picture; Projecting the feature map onto a target projection surface and acquiring a visible light shooting picture, wherein the visible light shooting picture includes the feature map; Obtaining a second sub-transformation matrix based on the coordinates of the feature map in the original projection image and the visible light captured image respectively; A main conversion matrix is ​​obtained according to the first sub-conversion matrix and the second sub-conversion matrix.

3. The projection interaction method according to claim 1, characterized in that: Before obtaining the target coordinates corresponding to the infrared spot coordinates in the projection original image based on the infrared spot coordinates and the main conversion matrix, the method further includes: obtaining the projection mapping area corresponding to the projection original image in the infrared shooting screen according to the vertex coordinates of the projection original image and the inverse matrix operation of the main conversion matrix; The method of obtaining the target coordinates corresponding to the infrared spot coordinates in the projection original image based on the infrared spot coordinates and the main conversion matrix includes: when the infrared spot coordinates are located in the projection mapping area, obtaining the target coordinates corresponding to the infrared spot coordinates in the projection original image based on the infrared spot coordinates and the main conversion matrix.

4. The projection interaction method according to claim 1, characterized in that: Before obtaining the target coordinates corresponding to the infrared spot coordinates in the projection original image based on the infrared spot coordinates and the main conversion matrix, the method includes: Acquire a first real-time motion parameter of an inertial sensor in a remote controller that emits an infrared light spot; calculate theoretical light spot coordinates of the infrared light spot in the infrared shooting picture according to the infrared light spot coordinates, the first real-time motion parameter and a dead reckoning algorithm; use the theoretical light spot coordinates as new infrared light spot coordinates; or, Acquire the current projection distance, the first spatial attitude angle and the preset shooting picture viewing angle of the projector, and establish a first spatial coordinate system, wherein the first spatial coordinate system includes an infrared shooting area corresponding to the infrared shooting picture, and infrared mapping coordinates corresponding to the infrared spot coordinates; acquire the distance from the wall and the second spatial attitude angle of the remote control that emits the infrared spot; obtain the starting coordinates and starting orientation of the remote control in the first spatial coordinate system according to the infrared mapping coordinates, the distance from the wall and the second spatial attitude angle; acquire the second real-time motion parameters of the remote control, and calculate the theoretical spot coordinates of the infrared spot in the infrared shooting picture based on the starting coordinates, the starting orientation and the second real-time motion parameters in the first spatial coordinate system; and use the theoretical spot coordinates as the new infrared spot coordinates.

5. The projection interaction method according to claim 4, characterized in that: After acquiring the main conversion matrix between the infrared shooting picture and the projection original picture, the method includes: Based on the main transformation matrix and the projection area in the projection original image, determining a projection mapping area corresponding to the projection area in the captured image; When the theoretical light spot coordinates are outside the projection mapping area; or when the theoretical light spot coordinates are within the projection mapping area but cannot be identified in the infrared shooting picture, the remote controller is instructed to reduce the brightness of the emitted infrared light spot.

6. The projection interaction method according to claim 4, characterized in that: After acquiring the main conversion matrix between the infrared shooting picture and the projection original picture, the method includes: Obtaining a projection mapping area corresponding to the projection original image in the infrared shooting picture according to the vertex coordinates of the projection original image and the inverse matrix operation of the main transformation matrix; When the theoretical light spot coordinates are located within the projection mapping area, and the infrared light spot coordinates cannot be identified from the infrared shooting picture within a second threshold time, the remote controller is instructed to turn off or reduce the emission brightness of the light spot.

7. The projection interaction method according to claim 1, characterized in that: After displaying the light spot at the position of the target coordinates in the projection original image, the method further comprises: Get the button control instructions of the remote control; An operation corresponding to the key control instruction is performed at a position corresponding to the target coordinates in the projection original image.

8. The projection interaction method according to claim 1, characterized in that: After acquiring the infrared shooting picture and identifying the coordinates of the infrared spot in the infrared shooting picture, the following steps are included: A real-time attitude angle change parameter of an inertial sensor in a remote controller is obtained, and when the real-time attitude angle change parameter is less than a third threshold, a change in the infrared spot coordinate is ignored.

9. A projection device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program can be read by the processor to be executed to implement the projection interaction method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to be executable by a processor to execute the projection interaction method described in any one of claims 1 to 8.

Citation Information

Cited By

  • Projection interaction device and method based on invisible light

    CN120786044A