Wireless control method and device for display screen

Abstract

The invention relates to the technical field of wireless control, in particular to a wireless control method and device for a display screen. The method comprises the following steps: identifying a pixel position of a light spot in each image; based on the internal reference of the camera, the spatial position of the lens, the spatial position of the infrared light source, the pixel position of the light spot and the radius of the reflective ball, the spherical center spatial position of the reflective ball is determined; determining a cursor position generated when the handheld device points to the display screen on the basis of the sphere center space positions of the two reflective spheres and a rotation transformation matrix and a translation transformation matrix of a space coordinate system converted from a screen coordinate system of the display screen, so as to wirelessly control the display screen by using the cursor position; wherein the spherical center space positions of the two reflective balls and the cursor position are collinear. According to the technical scheme, the click operation on the display picture can be freely controlled through the direction set by the handheld device.

Description

Technical Field

[0001] The present invention relates to the field of wireless control technology, and in particular to a method and device for wirelessly controlling a display screen. Background Art

[0002] With the development of wireless technology, traditional desktop mice can no longer meet users' needs for interactive experience. In a conference room or classroom, users can use some wireless control devices to wirelessly control the display screens of terminal devices in the conference room or classroom, such as turning pages or simply moving the pictures displayed on the display screen, but they cannot freely control click operations such as deleting and confirming the displayed pictures like desktop mice.

[0003] Therefore, there is an urgent need to provide a wireless control method and device for a display screen to solve the above technical problems. Summary of the invention

[0004] The invention provides a wireless control method and device for a display screen, which can freely control the click operation on the display screen through the direction set by a handheld device.

[0005] According to a first aspect, the present invention provides a wireless control method for a display screen, which is applied to a controller in a wireless control system, wherein the wireless control system comprises a display screen, two cameras, an infrared light source, a handheld device and the controller, wherein the handheld device is provided with two reflective balls, the display screen, the camera and the infrared light source are coplanar, the infrared light source is provided between the two cameras, the infrared light source is used to emit infrared rays, the reflective balls are used to reflect the infrared rays, the camera is used to photograph the handheld device so that each generated image contains two light spots, and the method comprises:

[0006] Identifying the pixel position of the light spot in each of the images;

[0007] Determine the spatial position of the center of the reflective ball based on the intrinsic parameters and lens spatial position of the camera, the spatial position of the infrared light source, the pixel position of the light spot, and the radius of the reflective ball;

[0008] Based on the spatial positions of the centers of the two reflective balls, the rotation transformation matrix and the translation transformation matrix of the display screen's screen coordinate system converted to the spatial coordinate system, the cursor position generated when the handheld device points to the display screen is determined, so that the display screen is wirelessly controlled using the cursor position; wherein the spatial positions of the centers of the two reflective balls and the cursor position are collinear.

[0009] According to a second aspect, the present invention provides a wireless control device for a display screen, which is applied to a controller in a wireless control system. The wireless control system includes a display screen, two cameras, an infrared light source, a handheld device and the controller. The handheld device is provided with two reflective balls. The display screen, the camera and the infrared light source are coplanar. The infrared light source is provided between the two cameras. The infrared light source is used to emit infrared rays. The reflective balls are used to reflect the infrared rays. The camera is used to shoot the handheld device so that each generated image contains two light spots. The device includes:

[0010] A light spot recognition unit, configured to recognize a pixel position of the light spot in each of the images;

[0011] A first determining unit is configured to determine the spatial position of the center of the reflective ball based on the intrinsic parameters and the spatial position of the lens of the camera, the spatial position of the infrared light source, the pixel position of the light spot, and the radius of the reflective ball;

[0012] The second determination unit is configured to determine the cursor position generated by the handheld device pointing to the display screen based on the spatial positions of the centers of the two reflective balls, the rotation transformation matrix and the translation transformation matrix of the screen coordinate system of the display screen converted to the spatial coordinate system, so as to wirelessly control the display screen using the cursor position; wherein the spatial positions of the centers of the two reflective balls and the cursor position are collinear.

[0013] According to the wireless control method and device of the display screen provided by the present invention, the pixel position of the light spot of the image is first identified, so that the spatial position of the center of the reflective ball can be determined, and the cursor position generated by the handheld device pointing to the display screen can be further determined, so as to wirelessly control the display screen using the cursor position, for example, triggering a click operation on the relevant content at the current cursor position through the confirmation key on the handheld device. Therefore, the above technical solution can freely control the click operation on the display screen through the direction set by the handheld device. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A schematic flow chart of a method for wirelessly controlling a display screen according to an embodiment is shown;

[0016] Figure 2A schematic block diagram of a wireless control device for a display screen according to an embodiment is shown;

[0017] Figure 3 A schematic structural diagram of a wireless control system according to an embodiment is shown;

[0018] Figure 4 An image with infrared spots captured by a camera according to one embodiment is shown;

[0019] Figure 5 A schematic diagram of a scenario in which a handheld device points to a display screen to generate a cursor position according to an embodiment is shown;

[0020] Figure 6 A schematic plan view showing the positions of a reflective ball, a camera and an infrared light source according to an embodiment is shown;

[0021] Figure 7 A schematic diagram of performing nine-point calibration on a screen according to an embodiment is shown. DETAILED DESCRIPTION

[0022] The solution provided by the present invention is described below in conjunction with the accompanying drawings.

[0023] Figure 1 A flow chart of a method for wirelessly controlling a display screen according to an embodiment is shown. It is understood that the method can be executed by any device, equipment, platform, or device cluster having computing and processing capabilities. Figure 1 and Figure 3 As shown, the method is applied to a controller in a wireless control system, the wireless control system includes a display screen, two cameras, an infrared light source, a handheld device and a controller, the handheld device is provided with two reflective balls, the display screen, the camera and the infrared light source are coplanar, the infrared light source is provided between the two cameras, the infrared light source is used to emit infrared rays, the reflective balls are used to reflect infrared rays, the camera is used to shoot the handheld device so that each generated image contains two light spots (such as Figure 4 shown), including:

[0024] Step 100, identifying the pixel position of the light spot in each image;

[0025] Step 102: Determine the spatial position of the center of the reflective ball based on the intrinsic parameters of the camera and the spatial position of the lens, the spatial position of the infrared light source, the pixel position of the light spot, and the radius of the reflective ball;

[0026] Step 104: Based on the spatial positions of the centers of the two reflective balls and the rotational transformation matrix and the translational transformation matrix of the screen coordinate system of the display screen converted to the spatial coordinate system, determine the cursor position (e.g. Figure 5As shown), the display screen can be wirelessly controlled using the cursor position; wherein the center spatial positions of the two reflective balls and the cursor position are collinear.

[0027] In this embodiment, the pixel position of the light spot of the image is first identified, so that the spatial position of the center of the reflective ball can be determined, and the cursor position generated by the handheld device pointing to the display screen can be further determined, so that the display screen can be wirelessly controlled using the cursor position, for example, by triggering a click operation on the relevant content at the current cursor position through the confirmation key on the handheld device. Therefore, the above technical solution can freely control the click operation on the display screen through the pointing setting of the handheld device.

[0028] For step 100, since the infrared light source on the screen is invisible to the naked eye, it can appear as a bright light spot on the camera. Therefore, the infrared light spot generated by the reflective ball on the speech pen (i.e., the handheld device) in the image can be effectively identified by means of RCNN algorithm, YOLO algorithm, etc.

[0029] like Figure 6 As shown, taking into account the requirements of cost and recognition accuracy, the present invention adopts a spatial positioning solution of a single light source and dual cameras, and the spatial positioning problem of one of the reflective balls is used as an example to illustrate.

[0030] Let the position of the infrared light source be I =

[000] , the spatial position of the camera lens (i.e., the lens node, which can be determined based on the horizontal displacement between the two and the infrared light source) be o1 = [0α0], o2 = [0-α0], the focal length of the camera (i.e., the vertical distance from the imaging plane to the lens node) be β, the radius of the reflective ball be R, and the light spot imaging positions be u1 and u2. Taking the light spot position u1 in camera 1 as an example, assuming is the pixel position of the light spot on the image, then where λ x and λ y These are two correction coefficients after considering the camera's internal parameters.

[0031] In order to find the center position C of the reflective ball, we need to use the following physical laws:

[0032] The three points q1, o1, and u1 are collinear, i.e., q1=o1+k1(o1-u1); the three points q2, o2, and u2 are collinear, i.e., q2=o2+k2(o2-u2); the four points C, q1, o1, and I are coplanar, i.e., (o1-I)×(q1-o1)·(C-o1)=0; the four points C, q2, o2, and I are coplanar, i.e., (o2-I)×(q2-o2)·(C-o2)=0; where × and · are the outer product and inner product operations of vectors, respectively;

[0033] According to the law of reflection, the normal Bisecting the two rays The angle between Similarly, we can know: in,

[0034] Integrating the above formulas, we can know that:

[0035]

[0036] By solving the above equations, k1, k2, and C can be obtained.

[0037] As a preferred implementation, the distances from the infrared light source to the two cameras are equal, and the spatial position of the center of the reflective ball is determined by the following formula:

[0038]

[0039] q1=o1+k1(o1-u1)

[0040] q2=o2+k2(o2-u2)

[0041] Where q1 and q2 are two reflection points on the surface of the reflective ball, o1 and o2 are the spatial positions of the lenses of the two cameras, C is the center point of any reflective ball, I is the spatial position of the infrared light source, R is the radius of the reflective ball, k1 and k2 are the first and second numbers to be solved, u1 and u2 are the imaging positions of the light spots of the two cameras, u1 and u2 are both related to the intrinsic parameters of the camera, the spatial position of the lens and the pixel position of the light spot, and × and · are the outer product and inner product operations of the vectors, respectively.

[0042] Although the above calculation method can obtain the spatial position of the center of the reflective ball, the process is relatively complicated and prone to falling into local optimality, and it will consume a lot of computing resources. In order to avoid affecting the accuracy of the results as much as possible, while improving the calculation speed and reducing the consumption of computing resources, you can consider optimizing the above calculation method. In fact, relative to the distance from the reflective ball to the display, the radius of the reflective ball is very small, that is, the two points q1 and q2 are very close. Therefore, we can simplify the above formula: that is, on the basis of q1=q2, calculate the optimal solution under the constraints of the following set of equations:

[0043]

[0044] From the first formula in this formula group, we can get make Therefore, k1 and k2 can be estimated using the least squares method So we can estimate the position of point q1 as Substituting it into the formula group, there are three formulas left, and we can get:

[0045]

[0046] Let C = [C x C y C z ,but In this nonlinear system of equations, only C=[C x C y C z ] is an unknown quantity, so the spatial position of the center of the reflective ball can be quickly calculated through an iterative calculation method.

[0047] It can be understood that through the above method, we can accurately calculate the spatial positions of the two reflective balls, which are recorded as C1 and C2 respectively (the spatial position C of the center of the reflective ball in the above calculation method can refer to both C1 and C2.), connect the two points C1 and C2, and select the ray in the direction of the infrared light source coordinate

[000] (Among them, the reflective ball of C1 is located at the end of the pen).

[0048] As a preferred implementation, let q1=q2, and the spatial position of the center of the reflective ball is further determined by the following formula:

[0049]

[0050] In the formula, is the estimator of q1, C x , C y , C z is the coordinate of C in the xyz direction.

[0051] Considering that in actual use, it is usually difficult to ensure that the handheld device is completely parallel to the display screen and the infrared light source is completely on the plane of the display, that is, the actual screen coordinate system and the infrared light source coordinate system may be deflected or translated. Assume that g = [g x g y ] point, which is R in the infrared light source coordinate system I g+T I , in R I and T I Under known conditions, we can solve the simultaneous equations C2+k3(C1-C2)=R I g+T I Calculate k3 and g (that is, solve the three unknown quantities k3, g in three equations x , g y ), and finally determine the screen position g that the pen is pointing to.

[0052] As a preferred implementation, the cursor position is determined by the following formula:

[0053] C2+k3(C1-C2)=R I g+T I

[0054] In the formula, C1 and C2 are the center points of the two reflective balls, k3 is the third number to be solved, and R I is the rotation transformation matrix from the screen coordinate system to the space coordinate system, T I is the translation transformation matrix from the screen coordinate system to the space coordinate system, and g is the cursor position in the screen coordinate system.

[0055] As a preferred implementation, the rotation transformation matrix and the translation transformation matrix are obtained by calibrating and solving at least six known display points on the screen. Figure 7 As shown, the nine-point calibration method can be used to solve the problem.

[0056] When the equipment is first installed, R I and T I is unknown and needs to be calibrated, so we propose a nine-point calibration method, that is, when used for the first time, nine points in the following positions are displayed on the screen in sequence, and the user is required to point to these nine points with a pen in turn. Therefore, we can get 9 known rays and the 9 known g positions, substituting them into the equation C2+k3(C1-C2)=R I g+T I Since each set of equations contains 3 equations, 9 sets of equations can obtain a total of 27 equations; and the variable to be solved is R I The 6 unknowns in T I There are 3 unknowns in the equations and 9 k3 variables in the 9 equations, a total of 18; therefore, 27 equations can support the solution of these 18 variables and can accurately solve the spatial transformation relationship. In theory, the rotation transformation matrix and the translation transformation matrix can be obtained by calibrating and solving the six known display points on the screen (that is, 18 equations can support the solution of these 18 variables).

[0057] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0058] According to another embodiment, the present invention provides a wireless control device for a display screen. Figure 2A schematic block diagram of a wireless control device for a display screen according to an embodiment is shown. It is understood that the device can be implemented by any device, equipment, platform and device cluster with computing and processing capabilities. Figure 2 As shown, the device is applied to a controller in a wireless control system, the wireless control system includes a display screen, two cameras, an infrared light source, a handheld device and the controller, the handheld device is provided with two reflective balls, the display screen, the camera and the infrared light source are coplanar, the infrared light source is provided between the two cameras, the infrared light source is used to emit infrared rays, the reflective ball is used to reflect the infrared rays, the camera is used to shoot the handheld device so that each generated image contains two light spots, the device includes: a light spot recognition unit 200, a first determination unit 202 and a second determination unit 204. The main functions of each component unit are as follows:

[0059] The light spot recognition unit 200 is configured to recognize the pixel position of the light spot in each of the images;

[0060] A first determining unit 202 is configured to determine the spatial position of the center of the reflective ball based on the intrinsic parameters and the spatial position of the lens of the camera, the spatial position of the infrared light source, the pixel position of the light spot, and the radius of the reflective ball;

[0061] The second determination unit 204 is configured to determine the cursor position generated by the handheld device pointing to the display screen based on the spatial positions of the centers of the two reflective balls, the rotation transformation matrix and the translation transformation matrix of the screen coordinate system of the display screen converted to the spatial coordinate system, so as to wirelessly control the display screen using the cursor position; wherein the spatial positions of the centers of the two reflective balls and the cursor position are collinear.

[0062] As a preferred implementation, the distances from the infrared light source to the two cameras are equal, and the spatial position of the center of the reflective ball is determined by the following formula:

[0063]

[0064] q1=o1+k1(o1-u1)

[0065] q2=o2+k2(o2-u2)

[0066] Wherein, q1 and q2 are two reflection points on the surface of the reflective ball, o1 and o2 are the spatial positions of the lenses of the two cameras, C is the center point of any of the reflective balls, I is the spatial position of the infrared light source, R is the radius of the reflective ball, k1 and k2 are the first number to be solved and the second number to be solved, u1 and u2 are the imaging positions of the light spots of the two cameras, u1 and u2 are both related to the intrinsic parameters of the camera, the spatial position of the lens and the pixel position of the light spot, and × and · are the outer product and inner product operations of the vectors, respectively.

[0067] As a preferred implementation, let q1=q2, the spatial position of the center of the reflective ball is further determined by the following formula:

[0068]

[0069] In the formula, is the estimator of q1, C x , C y , C z is the coordinate of C in the xyz direction.

[0070] As a preferred implementation, the cursor position is determined by the following formula:

[0071] C2+k3(C1-C2)=R I g+T I

[0072] Where C1 and C2 are the center points of the two reflective balls, k3 is the third number to be solved, and R I is the rotation transformation matrix from the screen coordinate system to the space coordinate system, T I is the translation transformation matrix for converting the screen coordinate system to the space coordinate system, and g is the cursor position in the screen coordinate system.

[0073] As a preferred implementation, the rotation transformation matrix and the translation transformation matrix are obtained by calibrating and solving at least six known display points on the screen.

[0074] According to another embodiment of the present invention, there is also provided an electronic device, comprising a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the Figure 1 The method described.

[0075] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0076] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0077] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A wireless control method for a display screen, characterized in that: A controller used in a wireless control system, the wireless control system comprises a display screen, two cameras, an infrared light source, a handheld device and the controller, the handheld device is provided with two reflective balls, the display screen, the camera and the infrared light source are coplanar, the infrared light source is provided between the two cameras, the infrared light source is used to emit infrared rays, the reflective balls are used to reflect the infrared rays, the camera is used to photograph the handheld device so that each generated image contains two light spots, the method comprises: Identifying the pixel position of the light spot in each of the images; Determine the spatial position of the center of the reflective ball based on the intrinsic parameters and lens spatial position of the camera, the spatial position of the infrared light source, the pixel position of the light spot, and the radius of the reflective ball; Based on the spatial positions of the centers of the two reflective balls, the rotation transformation matrix and the translation transformation matrix of the display screen's screen coordinate system converted to the spatial coordinate system, the cursor position generated when the handheld device points to the display screen is determined, so that the display screen is wirelessly controlled using the cursor position; wherein the spatial positions of the centers of the two reflective balls and the cursor position are collinear.

2. The method according to claim 1, characterized in that The distances from the infrared light source to the two cameras are equal, and the spatial position of the center of the reflective ball is determined by the following formula: q1=o1+k1(o1-u1) q2=o2+k2(o2-u2) Wherein, q1 and q2 are two reflection points on the surface of the reflective ball, o1 and o2 are the spatial positions of the lenses of the two cameras, C is the center point of any of the reflective balls, I is the spatial position of the infrared light source, R is the radius of the reflective ball, k1 and k2 are the first number to be solved and the second number to be solved, u1 and u2 are the imaging positions of the light spots of the two cameras, u1 and u2 are both related to the intrinsic parameters of the camera, the spatial position of the lens and the pixel position of the light spot, and × and · are the outer product and inner product operations of the vectors, respectively.

3. The method according to claim 2, characterized in that Let q1=q2, the spatial position of the center of the reflective ball is further determined by the following formula: In the formula, is the estimator of q1, C x , C y , C z is the coordinate of C in the xyz direction.

4. The method according to claim 2 or 3, characterized in that: The cursor position is determined by the following formula: C2+k3(C1-C2)=R I g+T I Where, C1 and C2 are the center points of the two reflective balls, k3 is the third number to be solved, and R I is the rotation transformation matrix from the screen coordinate system to the space coordinate system, T I is the translation transformation matrix for converting the screen coordinate system to the space coordinate system, and g is the cursor position in the screen coordinate system.

5. The method according to claim 4, characterized in that The rotation transformation matrix and the translation transformation matrix are obtained by calibrating and solving at least six known display points on the screen.

6. A wireless control device for a display screen, characterized in that: A controller used in a wireless control system, the wireless control system includes a display screen, two cameras, an infrared light source, a handheld device and the controller, the handheld device is provided with two reflective balls, the display screen, the camera and the infrared light source are coplanar, the infrared light source is provided between the two cameras, the infrared light source is used to emit infrared rays, the reflective balls are used to reflect the infrared rays, the camera is used to shoot the handheld device so that each generated image contains two light spots, the device includes: A light spot recognition unit, configured to recognize a pixel position of the light spot in each of the images; A first determining unit is configured to determine the spatial position of the center of the reflective ball based on the intrinsic parameters and the spatial position of the lens of the camera, the spatial position of the infrared light source, the pixel position of the light spot, and the radius of the reflective ball; The second determination unit is configured to determine the cursor position generated by the handheld device pointing to the display screen based on the spatial positions of the centers of the two reflective balls, the rotation transformation matrix and the translation transformation matrix of the screen coordinate system of the display screen converted to the spatial coordinate system, so as to wirelessly control the display screen using the cursor position; wherein the spatial positions of the centers of the two reflective balls and the cursor position are collinear.

7. The device according to claim 6, characterized in that The distances from the infrared light source to the two cameras are equal, and the spatial position of the center of the reflective ball is determined by the following formula: q1=o1+k1(o1-u1) q2=o2+k2(o2-u2) Wherein, q1 and q2 are two reflection points on the surface of the reflective ball, o1 and o2 are the spatial positions of the lenses of the two cameras, C is the center point of any of the reflective balls, I is the spatial position of the infrared light source, R is the radius of the reflective ball, k1 and k2 are the first number to be solved and the second number to be solved, u1 and u2 are the imaging positions of the light spots of the two cameras, u1 and u2 are both related to the intrinsic parameters of the camera, the spatial position of the lens and the pixel position of the light spot, and × and · are the outer product and inner product operations of the vectors, respectively.

8. The device according to claim 7, characterized in that Let q1=q2, the spatial position of the center of the reflective ball is further determined by the following formula: In the formula, is the estimator of q1, C x , C y , C z is the coordinate of C in the xyz direction.

9. The device according to claim 7 or 8, characterized in that The cursor position is determined by the following formula: C2+k3(C1-C2)=R I g+T I Where, C1 and C2 are the center points of the two reflective balls, k3 is the third number to be solved, and R I is the rotation transformation matrix from the screen coordinate system to the space coordinate system, T I is the translation transformation matrix for converting the screen coordinate system to the space coordinate system, and g is the cursor position in the screen coordinate system.

10. The device according to claim 9, characterized in that The rotation transformation matrix and the translation transformation matrix are obtained by calibrating and solving at least six known display points on the screen.

Images