Method and device for determining position of mouse pointer of air mouse

Abstract

The invention discloses a method and a device for determining the position of a mouse pointer of an air mouse, relates to the technical field of man-machine interaction and electronic equipment control, and mainly aims to improve the consistency between the pointing direction of the air mouse and the position of the mouse pointer on a screen so as to improve the user experience. According to the main technical scheme, the method comprises the steps of obtaining a screen image corresponding to the direction of an air mouse; extracting a quadrilateral contour of the screen image by using a contour extraction algorithm; determining a geometric center point of the quadrilateral contour by using a geometric center point determination method; determining a mouse pointer position corresponding to the air mouse on the screen based on the geometric center point by using a position determination method; and sending the mouse pointer position to the host so as to display the mouse pointer on the mouse pointer position of the screen by using the host. The method and the device are used for determining the position of the air mouse pointing to the corresponding mouse pointer on the screen.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction and electronic equipment control, and in particular to a method and device for determining the position of an air mouse pointer. Background Art

[0002] With the popularity of smart devices, air mouse (abbreviated as "air mouse") is widely used in large-screen devices such as TVs and projectors as a convenient interactive tool. At present, the method for determining the position of the mouse pointer corresponding to the air mouse is to use internal sensors such as gyroscopes and accelerometers to capture the user's movements and convert them into the movement of the mouse pointer on the screen, thereby greatly improving the convenience of operation.

[0003] However, in actual use, the gyroscope inside the air mouse may cause inaccurate detection data due to long-term operation, changes in ambient temperature, power fluctuations, and impacts caused by the device during movement. Similarly, the accelerometer may also be affected by electronic noise and changes in ambient temperature, causing deviations in its detection data. Since the gyroscope and accelerometer detect different data based on the different directions of the air mouse and convert these data into the position of the mouse pointer on the screen, when the data of these two sensors are inaccurate, the direction of the air mouse and the position of the mouse pointer on the screen will deviate, seriously affecting the user experience. Summary of the invention

[0004] In view of the above problems, the present invention provides a method and device for determining the position of an air mouse pointer, the main purpose of which is to improve the consistency between the direction of the air mouse and the position of the mouse pointer on the screen, thereby improving the user experience.

[0005] In order to solve the above technical problems, the present invention proposes the following solutions:

[0006] In a first aspect, the present invention provides a method for determining the position of an empty mouse pointer, the method comprising:

[0007] Get the screen image corresponding to the empty mouse pointing;

[0008] Extracting the quadrilateral outline of the screen image using an outline extraction algorithm;

[0009] Determine the geometric center point of the quadrilateral outline using a geometric center point determination method;

[0010] Using a position determination method, based on the geometric center point, determine the corresponding mouse pointer position of the empty mouse on the screen;

[0011] The mouse pointer position is sent to a host so that the host can display the mouse pointer at the mouse pointer position on the screen.

[0012] In a second aspect, the present invention provides a device for determining the position of an empty mouse pointer, the device comprising:

[0013] An image acquisition unit, used to acquire a screen image corresponding to the pointing of the air mouse;

[0014] A contour extraction unit, used for extracting a quadrilateral contour of the screen image acquired by the image acquisition unit using a contour extraction algorithm;

[0015] A center point determination unit, used to determine the geometric center point of the quadrilateral outline extracted by the outline extraction unit using a geometric center point determination method;

[0016] A position determination unit, used to determine the position of the mouse pointer corresponding to the air mouse on the screen based on the geometric center point determined by the center point determination unit using a position determination method;

[0017] The position sending unit is used to send the mouse pointer position determined by the position determining unit to the host, so that the host can display the mouse pointer at the mouse pointer position on the screen.

[0018] In order to achieve the above-mentioned purpose, according to the third aspect of the present invention, a storage medium is provided, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the method for determining the position of the empty mouse pointer of the first aspect mentioned above.

[0019] In order to achieve the above object, according to a fourth aspect of the present invention, a processor is provided, the processor being used to run a program, wherein the method for determining the position of an empty mouse pointer according to the first aspect is executed when the program is run.

[0020] By means of the above technical scheme, the present invention provides a method and device for determining the position of an air mouse pointer, which can realize automatic confirmation of the position of the mouse pointer by identifying the quadrilateral outline in the screen image and calculating its geometric center point. This method can ensure the correct display position of the mouse pointer even when the data provided by the gyroscope or sensor inside the air mouse is inaccurate, thereby ensuring a good operation experience for the user. First, the image of the current screen is captured by the air mouse. Subsequently, the quadrilateral outline in the screen image is identified using an image processing algorithm. Next, the geometric center point determination algorithm is used to calculate the geometric center point of the quadrilateral outline. Compared with the vertices of the quadrilateral, the geometric center point is more stable and less susceptible to hand shaking, thereby improving the accuracy of the mouse pointer position. The position of the geometric center point intuitively reflects the user's intention to operate the air mouse, so that the movement of the mouse pointer is more in line with the user's expectations. Finally, the calculated geometric center point position is sent to the host in a timely manner, and the host displays the corresponding mouse pointer position on the screen. This method greatly shortens the response time and enhances the real-time performance of the system. The user can observe the changes in the position of the mouse pointer in real time, and the immediate operation feedback improves the user experience and makes the entire interaction process smoother and more natural.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 A flow chart of a method for determining the position of an empty mouse pointer provided by an embodiment of the present invention is shown;

[0024] Figure 2 A flow chart of another method for determining the position of an empty mouse pointer provided by an embodiment of the present invention is shown;

[0025] Figure 3 A block diagram showing a device for determining the position of an empty mouse pointer provided by an embodiment of the present invention is shown;

[0026] Figure 4 A block diagram showing the composition of another device for determining the position of an empty mouse pointer provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0028] Aiming at the problem that the measured data of the gyroscope and accelerometer inside the current air mouse are affected by external factors and the air mouse pointing is inconsistent with the mouse pointer position on the screen, the inventor has proposed a method for determining the position of the air mouse mouse pointer through a lot of creative work. The method can maintain the consistency of the position of the air mouse and the mouse pointer even if there are errors in the gyroscope and accelerometer inside the air mouse through the cooperation between the air mouse and the host corresponding to the screen. Specifically, when implementing the method, the required equipment includes an air mouse, a screen and a host corresponding to the screen. Among them, the air mouse can be equipped with a gyroscope, an accelerometer, a high-definition camera and a control program; of course, the air mouse can also be equipped with only a high-definition camera and a control program. The camera of the air mouse is responsible for shooting the screen image and sending the image to the control program. After receiving the screen image, the control program extracts the quadrilateral outline in the screen image and determines the geometric center point of the quadrilateral outline according to the geometric center point determination method. Subsequently, the position determination method is used to determine the display position of the mouse pointer corresponding to the air mouse on the screen according to the geometric center point. Finally, the determined mouse pointer position is sent to the host, and the host displays the mouse pointer on the screen. It can also be seen from this that the execution subject of the method for determining the position of the air mouse pointer provided by the embodiment of the present invention is the control program inside the air mouse.

[0029] Further, a method for determining the position of an empty mouse pointer provided by an embodiment of the present invention is described in detail. The specific execution steps are as follows: Figure 1 As shown, including:

[0030] 101. Get the screen image corresponding to the empty mouse pointing.

[0031] In this step, you can obtain the screen image corresponding to the mouse pointing at the following two times:

[0032] The first case is when the air mouse detects a change in its own posture using its internal gyroscope and accelerometer. At this point, the air mouse can send a posture change event to the host. After receiving the posture change event, the host calculates the theoretical position of the mouse pointer that should appear on the screen based on the posture data. Then, the host obtains the current actual displayed pointer position. By comparing the theoretically calculated pointer position with the actual displayed pointer position, if the difference between the two is large, it is determined that the air mouse is pointing at an inconsistent position with the mouse pointer position. In this case, the host can communicate with the air mouse and instruct the high-definition camera inside the air mouse to capture the screen image facing the air mouse.

[0033] In addition, the correction mechanism triggering strategy can be integrated into the air mouse. Specifically, the gyroscope and accelerometer inside the air mouse can detect specific gestures, such as shaking several times in a row. When this specific gesture is detected, the high-definition camera is triggered to capture the image of the screen currently facing. This method triggers the shooting mechanism through simple user interaction (such as shaking the air mouse), which can ensure the consistency and smoothness of the operation.

[0034] In the second case, there is no need to rely on the host. The high-definition camera inside the air mouse can capture the screen image facing the air mouse in real time.

[0035] 102. Use the contour extraction algorithm to extract the quadrilateral contour of the screen image.

[0036] After acquiring the screen image, the acquired screen image is first converted into a grayscale image to simplify subsequent processing. Next, the grayscale image is smoothed (for example, using Gaussian blur) to reduce the impact of noise on edge detection. Subsequently, the smoothed image is edge detected using a contour extraction algorithm (such as Canny edge detection) to highlight the edge features in the image. Then, based on the edge detection results, computer vision technology (such as OpenCV's findContours function) is used to find all contours in the image. Next, the contours that are approximately quadrilaterals are screened out from all the detected contours. Finally, among the screened quadrilateral contours, the best contour with the largest area or that meets other conditions is selected as the target quadrilateral, and the target quadrilateral is determined as the quadrilateral contour of the screen image.

[0037] The reason why the best outline with the largest area or satisfying other conditions is selected as the quadrilateral outline of the screen image is:

[0038] Usually, the boundary contour of the screen is the largest quadrilateral contour. Therefore, selecting the quadrilateral contour with the largest area can improve the reliability of detection and avoid misselecting smaller non-boundary contours. If multiple quadrilateral contours have similar areas, the best contour can be further screened based on other conditions (such as shape regularity, position rationality, etc.). For example, you can judge which contour is more likely to be the boundary contour of the screen based on their position in the image.

[0039] In summary, selecting the best contour with the largest area or satisfying other conditions can improve the robustness of the algorithm and better identify the screen boundary even in the case of poor image quality or interference.

[0040] 103. Use the geometric center point determination method to determine the geometric center point of the quadrilateral outline.

[0041] After successfully extracting the quadrilateral outline, the geometric center point is more stable than the vertices of the quadrilateral and is less susceptible to hand shaking, which helps improve the accuracy of mouse pointer positioning. In addition, the position of the geometric center point directly maps the user's intention to operate the air mouse, making the movement of the mouse pointer more in line with the user's expectations. Therefore, in this step, the geometric center point determination algorithm will be used to calculate the geometric center point of the quadrilateral outline.

[0042] It should be noted that due to the relative angle between the air mouse and the screen, the screen image captured by the built-in high-definition camera of the air mouse may produce a distorted quadrilateral outline. Therefore, when applying the geometric center point determination method, both the standard quadrilateral and the distorted quadrilateral can be considered to ensure the accuracy and adaptability of the algorithm.

[0043] That is, determine whether the outline of the quadrilateral is a standard quadrilateral; if not, use the geometric center point determination method corresponding to the non-standard quadrilateral (deformed quadrilateral) to determine the geometric center point of the outline of the quadrilateral; if so, use the geometric center point determination method corresponding to the standard quadrilateral to determine the geometric center point of the outline of the quadrilateral.

[0044] It should be noted that if the quadrilateral outline obtained from the captured screen image is a distorted quadrilateral, it can be a distorted convex quadrilateral outline. Because the screen itself is a planar object, its four corners are right angles. Even if the image is deformed due to the distortion of the viewing angle or the camera lens, no concave corners will be formed, because the four corners of the screen are always expanded outwards, not concave inwards.

[0045] 104. Utilize the position determination method to determine the mouse pointer position corresponding to the empty mouse on the screen based on the geometric center point.

[0046] 105. Send the mouse pointer position to the host so that the host can display the mouse pointer at the mouse pointer position on the screen.

[0047] In step 104 and step 105, after the geometric center point is obtained, since the geometric center point is obtained by the screen image captured by the camera, the geometric center point is located in the camera coordinate system. What needs to be determined in the end is the position of the mouse pointer on the screen. Therefore, it is necessary to use a coordinate conversion method to convert the coordinates of the geometric center point in the camera coordinate system into the coordinates in the screen coordinate system. The converted coordinates of the geometric center point are determined as the corresponding mouse pointer position of the empty mouse on the screen, and the mouse pointer position is sent to the host so that the host displays the mouse pointer at the corresponding position on the screen.

[0048] In this embodiment, a center point can be determined by using multiple images to determine the corresponding mouse pointer position; of course, a center point can also be determined by only using one screen image to determine the corresponding mouse pointer position.

[0049] In addition, it should be noted that when the air mouse uses a camera to capture the screen image in real time, the air mouse may continuously output the newly calculated mouse pointer position to the host. In this case, the host needs to ensure that the mouse pointer moves smoothly from the current position to the new position to provide a smooth user experience.

[0050] Based on the above Figure 1 It can be seen from the implementation method that the present invention provides a method for determining the position of the mouse pointer of an air mouse, which can realize automatic confirmation of the position of the mouse pointer by identifying the quadrilateral outline in the screen image and calculating its geometric center point. This method can ensure the correct display position of the mouse pointer even when the data provided by the gyroscope or sensor inside the air mouse is inaccurate, thereby ensuring a good operating experience for the user. First, the image of the current screen is captured by the air mouse. Subsequently, the quadrilateral outline in the screen image is identified using an image processing algorithm. Next, the geometric center point determination algorithm is used to calculate the geometric center point of the quadrilateral outline. Compared with the vertices of the quadrilateral, the geometric center point is more stable and less susceptible to hand shaking, thereby improving the accuracy of the mouse pointer position. The position of the geometric center point intuitively reflects the user's intention to operate the air mouse, so that the movement of the mouse pointer is more in line with the user's expectations. Finally, the calculated geometric center point position is sent to the host in a timely manner, and the host displays the corresponding mouse pointer position on the screen. This method greatly shortens the response time and enhances the real-time performance of the system. The user can observe the changes in the position of the mouse pointer in real time, and the immediate operation feedback improves the user experience and makes the entire interaction process smoother and more natural.

[0051] Furthermore, as a Figure 1The embodiment shown in the figure is refined and expanded. The embodiment of the present invention also provides another method for determining the position of the empty mouse pointer, such as Figure 2 As shown, the specific steps are as follows:

[0052] 201. Obtain the screen image corresponding to the empty mouse pointing.

[0053] Among them, the implementation method of step 201 is the same as that of step 101, and can achieve the same technical effect and solve the same technical problem, and will not be repeated here.

[0054] 202. Use a contour extraction algorithm to extract the quadrilateral contour of the screen image.

[0055] In this step, in order to prevent the situation that the captured screen image is too dark due to the environment (such as too low brightness, etc.), and the quadrilateral outline of the screen image cannot be extracted, it can be detected whether the captured screen image meets the quadrilateral outline extraction standard. If it does not meet the standard, the screen image can be processed, and the processing measures include adjusting the image contrast, brightness, removing noise and enhancing the outline.

[0056] Among them, the extraction criteria may include the following:

[0057] The brightness of the screen image can be sufficient to ensure that edge features in the image can be clearly identified.

[0058] The contrast of the screen image is such that different areas in the image can be distinguished.

[0059] The clarity of the screen image is satisfactory: the edges and details in the image can be extracted.

[0060] The noise in the image should be as little as possible to reduce the interference with contour detection.

[0061] In addition, the contrast and brightness of the image can be adjusted and the noise can be removed according to the actual situation of the screen image. For example, if the image is overexposed, the brightness of the screen image can be appropriately reduced; on the contrary, if the image is too dark, the brightness of the screen image can be appropriately increased.

[0062] There is no specific limitation on the specific algorithm for contour enhancement of the screen image, as long as the contour enhancement effect can be achieved without introducing additional noise.

[0063] It should be noted that the above embodiment is performed on the assumption that one screen image is captured. However, in order to improve the accuracy of contour extraction, multiple screen images can also be captured at the same angle (since the screen content is dynamically changeable, the brightness of multiple images may change). Therefore, the stability of contour detection can be improved by combining the information of multiple frames of images and using inter-frame fusion technology.

[0064] 203. Determine the geometric center point of the quadrilateral outline using the geometric center point determination method.

[0065] In this step, it can be determined whether the outline of the quadrilateral is a standard quadrilateral; if not, the geometric center point of the outline of the quadrilateral is determined by the method for determining the geometric center point corresponding to the non-standard quadrilateral, which is the diagonal intersection method; on the contrary, if it is, the geometric center point of the outline of the quadrilateral is determined by the method for determining the geometric center point corresponding to the standard quadrilateral, which is the vertex average method.

[0066] For example, the geometric center point calculation for a non-standard quadrilateral is:

[0067] Find the two endpoints of the diagonal of the quadrilateral, let them be A(x1,y1) and C(x3,y3).

[0068] Find the two endpoints of another pair of diagonals, assuming they are B(x2,y2) and D(x4,y4).

[0069] Calculate the intersection of the two diagonals as the geometric center point. Since the quadrilateral is convex, each diagonal divides the quadrilateral into two triangles, calculate the centroid of each triangle, and then take the average of the two centroids as the geometric center point.

[0070] The calculation formula of the diagonal intersection method is as follows:

[0071] For diagonals AC and BD, their intersection point P(xp,yP) can be calculated by the following formula:

[0072]

[0073] Calculation of the geometric center point of a standard quadrilateral:

[0074] If the quadrilateral is a rectangle or a square, the average value of the vertices of the quadrilateral can be directly calculated as the geometric center point:

[0075]

[0076] 204. Using the position determination method, the mouse pointer position corresponding to the empty mouse on the screen is determined based on the geometric center point.

[0077] In this step, the screen size corresponding to the screen image, including the screen width and screen height, can be obtained first; then the boundary of the camera coordinate system for shooting the screen image can be obtained. Finally, the coordinate conversion method is used to obtain the corresponding mouse pointer position on the screen according to the screen size, the boundary of the camera coordinate system and the geometric center point calculated in step 203. This step actually converts the geometric center point in step 203 into the coordinates in the screen coordinate system.

[0078] It should be noted that the above screen size can be manually measured and recorded in advance, and then transmitted to the air mouse through the communication between the host and the air mouse. This ensures that the air mouse can accurately perform corresponding coordinate conversion according to the actual size of the screen during operation.

[0079] Assuming the geometric center point is (xc, yc) and the pixel coordinates of the screen (mouse pointer position) are (xs, ys), the following formula can be used for conversion:

[0080]

[0081] Where (ximg_min,yimg_min) and (ximg_max,yimg_max) are the boundaries of the camera coordinate system, and screen_width and screen_height are the width and height of the screen.

[0082] It should be noted that the above formula is applicable to the case where the actual width and height of the screen are known. However, if the specific size of the screen cannot be directly obtained, and only the relative size ratio of the screen width and height is known (such as the common 16:9), the following method can be used to calculate the coordinates of the geometric center point in the screen coordinate system:

[0083] Calculate the initial coordinates:

[0084]

[0085] Then, use the relative aspect ratio, such as 16:9, to calculate the relative coordinates in the screen coordinate system, assuming that the width of the screen is 1 unit length and the height is 9 / 16 unit length:

[0086] xs=xinitial (Formula 9)

[0087]

[0088] Where (ximg_min,yimg_min) and (ximg_max,yimg_max) are the boundaries of the camera coordinate system.

[0089] The (xs, ys) obtained in this way is a coordinate position relative to the screen size. This coordinate is valid no matter what the actual screen size is.

[0090] Of course, if you want to obtain the actual pixel coordinates, you can assume a screen width and height based on the relative ratio of 16:9, and then use Formula 5 and Formula 6 to calculate the coordinates of the geometric center point in the screen coordinate system.

[0091] 205. Send the mouse pointer position to the host, so that the host can display the mouse pointer at the mouse pointer position on the screen.

[0092] Among them, the implementation method of step 205 is the same as that of step 105, and can achieve the same technical effect and solve the same technical problem, and will not be repeated here.

[0093] In addition, it should be noted that there is a situation in the embodiment of the present invention, that is, the camera inside the air mouse captures the screen image in real time, and outputs the corresponding mouse pointer display position in real time according to the screen image. In this case, the host needs to ensure that the mouse pointer moves smoothly from the current position to the new position. The specific execution steps are as follows:

[0094] 1. Determine the current mouse pointer position and the new mouse pointer position sent by the empty mouse;

[0095] 2. Create a moving path from the current mouse pointer position to the new mouse pointer position based on the current mouse pointer position and the new mouse pointer position sent by the air mouse according to the linear interpolation algorithm or the spline interpolation algorithm;

[0096] 3. Move the mouse pointer from the current mouse pointer position to the new mouse pointer position according to the moving path.

[0097] It should be noted that, when the host moves the mouse pointer from the current position to the new position according to the movement path, visual feedback effects can be taken into account. For example, a temporary auxiliary line can be displayed on the screen, extending from the current pointer position to the new position, to prompt the user of the upcoming movement.

[0098] In practical applications, it is crucial to choose a linear interpolation algorithm or a spline interpolation algorithm to implement the planning of the moving path according to specific needs. For simple scenarios, that is, when only the starting point and the end point need to be directly connected without requiring a smooth transition, the linear interpolation algorithm is the most appropriate choice. During the linear interpolation process, speed control can also be introduced as needed to make the mouse pointer show different moving speeds in different time periods, for example, to achieve the effect of acceleration or deceleration. In addition, linear interpolation also allows the path to be dynamically adjusted to bypass obstacles or specific areas on the screen.

[0099] On the contrary, when it is necessary to generate a curve that passes through multiple points and achieves smooth transitions between these points, the spline interpolation algorithm shows its advantages. Spline interpolation is particularly suitable for mobile path planning in complex scenes. It can ensure that the path remains smooth and natural while passing through each specified point. Therefore, the spline interpolation algorithm is a better choice when it involves complex path planning and requires a high degree of smoothness.

[0100] 206. Use the fusion correction algorithm to determine the new mouse pointer position corresponding to the empty mouse on the screen.

[0101] After obtaining the position of the mouse pointer, in order to further improve the accuracy of the obtained position of the mouse pointer, a fusion correction algorithm may be used to determine a new position of the mouse pointer corresponding to the empty mouse on the screen.

[0102] Among them, the specific execution steps of the fusion correction algorithm are as follows:

[0103] 1. When the direction of the air mouse remains unchanged, obtain the attitude angle data obtained from the gyroscope and accelerometer inside the air mouse. The accelerometer measures the acceleration of gravity, which can be used to estimate the attitude angle. The formula is:

[0104]

[0105] Where ax, ay, az are the readings of the accelerometer on the three axes.

[0106] 2. Using a sensor fusion algorithm, the attitude angle data obtained by the air mouse gyroscope and the accelerometer are fused to obtain fused attitude angle data, wherein the sensor fusion algorithm can be a sensor fusion algorithm such as a complementary filter or a Kalman filter.

[0107] The formula is:

[0108] θfusion(t)=α·θgyro(t)+(1-α)θaccel(t) (Formula 12)

[0109] Among them, θfusion(t) is the attitude angle after fusion, θgyro(t) is the attitude angle calculated by the gyroscope, θaccel(t) is the attitude angle calculated by the accelerometer, and α is the fusion factor (usually between 0 and 1).

[0110] 3. Determine the new mouse pointer position corresponding to the air mouse on the screen based on the fused posture angle data and the mouse pointer position, including:

[0111] (1) Obtain the offset data based on the fused attitude angle data and the preset compensation coefficient.

[0112] Conversion from attitude angle to offset:

[0113] Assume that the attitude angle θfusion describes the tilt angle of the air mouse relative to the screen.

[0114] The tilt angle can be converted into the offset of the pointer on the screen. The specific formula is:

[0115] dx=kx×θfusion (Formula 13)

[0116] dy=ky×θfusion (Formula 14)

[0117] Among them, kx and ky are preset compensation coefficients used to adjust the size of the offset.

[0118] The values ​​of kx and ky can be determined by experiments or experience to ensure that the offset is appropriate.

[0119] It should be noted that θfusion is an angle value, which can be radians or degrees, depending on the implementation. If radians are used, kx and ky will be in screen pixels, with the number of pixels per radian. If degrees are used, the angle needs to be converted to radians before calculation (1 degree = π / 180 radians).

[0120] (2) Determine the new mouse pointer position corresponding to the empty mouse on the screen based on the offset data and the mouse pointer position.

[0121] This step is to add the offset to the original calculated mouse pointer position. The formula is:

[0122] (x fusion, y fusion) = (xs + dx, ys + dy) (Formula 15)

[0123] Among them, (xs, ys) is the original calculated mouse pointer position.

[0124] It should be noted that the execution of step 206 is initiated when it is detected that the mouse pointer position calculated by the above steps is still inaccurate, as an additional calibration method. The conditions for triggering the calibration are not specified in detail here. It can be manual intervention or further detection by the host after applying the mouse pointer position obtained by the above steps. In order to improve the efficiency of subsequent calibration, it is recommended to perform this detection while the empty mouse pointing remains unchanged, so that the previously calculated mouse pointer position can be directly added to the detected offset to obtain a new mouse pointer position.

[0125] It should be noted that, as mentioned above, the air mouse can be equipped with a gyroscope, an accelerometer, a high-definition camera, and a control program; of course, the air mouse can also be equipped with only a high-definition camera and a control program. When a fusion correction algorithm is needed to determine the position of the new mouse pointer corresponding to the air mouse on the screen, the air mouse should be equipped with a gyroscope, an accelerometer, a high-definition camera, and a control program. However, if a fusion correction algorithm is not needed to determine the position of the new mouse pointer, the air mouse only needs to be equipped with a high-definition camera and a control program. Regardless of the configuration used, both situations fall within the scope of protection of the embodiments of the present invention.

[0126] Furthermore, as a response to the above Figure 1 In order to realize the method shown in the figure, the embodiment of the present invention also provides a device for determining the position of the empty mouse pointer, which is used for Figure 1 The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Figure 3 As shown, the device comprises:

[0127] An image acquisition unit 301 is used to acquire a screen image corresponding to the pointing of the air mouse;

[0128] A contour extraction unit 302, configured to extract a quadrilateral contour of the screen image acquired by the image acquisition unit 301 using a contour extraction algorithm;

[0129] A center point determination unit 303, used to determine the geometric center point of the quadrilateral outline extracted by the outline extraction unit 302 using a geometric center point determination method;

[0130] A position determination unit 304 is used to determine the position of the mouse pointer corresponding to the air mouse on the screen based on the geometric center point determined by the center point determination unit 303 using a position determination method;

[0131] The position sending unit 305 is used to send the mouse pointer position determined by the position determining unit 304 to the host, so that the host can display the mouse pointer at the mouse pointer position on the screen.

[0132] Furthermore, as a response to the above Figure 2 In order to realize the method shown in the figure, the embodiment of the present invention also provides another device for determining the position of the empty mouse pointer, which is used for determining the position of the empty mouse pointer. Figure 2 The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Figure 4 As shown, the device comprises:

[0133] An image acquisition unit 301 is used to acquire a screen image corresponding to the pointing of the air mouse;

[0134] A contour extraction unit 302, configured to extract a quadrilateral contour of the screen image acquired by the image acquisition unit 301 using a contour extraction algorithm;

[0135] A center point determination unit 303, used to determine the geometric center point of the quadrilateral outline extracted by the outline extraction unit 302 using a geometric center point determination method;

[0136] A position determination unit 304 is used to determine the position of the mouse pointer corresponding to the air mouse on the screen based on the geometric center point determined by the center point determination unit 303 using a position determination method;

[0137] The position sending unit 305 is used to send the mouse pointer position determined by the position determining unit 304 to the host, so that the host can display the mouse pointer at the mouse pointer position on the screen.

[0138] In an optional implementation manner, the center point determination unit 303 includes:

[0139] A shape determination module 3031 is used to determine whether the quadrilateral outline is a standard quadrilateral;

[0140] A first center point determination module 3032 is used to determine the geometric center point of the quadrilateral outline by using a geometric center point determination method corresponding to a non-standard quadrilateral if the shape determination module 3031 determines no, wherein the geometric center point determination method corresponding to the non-standard quadrilateral is a diagonal intersection method;

[0141] The second center point determination module 3033 is used to determine the geometric center point of the quadrilateral outline using the geometric center point determination method corresponding to the standard quadrilateral if the shape judgment module 3031 determines yes, and the geometric center point determination method corresponding to the standard quadrilateral is the vertex average method.

[0142] In an optional implementation manner, the location determination unit 304 includes:

[0143] A size acquisition module 3041 is used to acquire the screen size corresponding to the screen image, where the screen size includes screen width and screen height;

[0144] A boundary acquisition module 3042, used to acquire the boundary of the camera coordinate system for capturing the screen image;

[0145] The position determination module 3043 is used to obtain the corresponding mouse pointer position of the air mouse on the screen based on the screen size obtained by the size acquisition module 3041, the boundary of the camera coordinate system obtained by the boundary acquisition module 3042 and the geometric center point by using a coordinate conversion method.

[0146] In an optional implementation, after the position determination unit 304 determines the position of the mouse pointer corresponding to the air mouse on the screen based on the geometric center point using a position determination method, the device further includes a position correction unit 306, and the position correction unit 306 is specifically used to:

[0147] The fusion correction algorithm is used to determine the new mouse pointer position corresponding to the empty mouse on the screen.

[0148] In an optional implementation manner, when the position correction unit 306 determines the new mouse pointer position corresponding to the empty mouse on the screen by using the fusion correction algorithm, it is specifically used to:

[0149] When the direction of the air mouse remains unchanged, acquiring attitude angle data obtained from the gyroscope and accelerometer of the air mouse;

[0150] The sensor fusion algorithm is used to fuse the attitude angle data obtained by the air mouse gyroscope and the accelerometer to obtain the fused attitude angle data;

[0151] A new mouse pointer position corresponding to the air mouse on the screen is determined according to the fused posture angle data and the mouse pointer position.

[0152] In an optional implementation manner, when the position correction unit 306 determines the new mouse pointer position corresponding to the air mouse on the screen according to the fused posture angle data and the mouse pointer position, it is specifically used to:

[0153] Obtaining offset data according to the fused attitude angle data and a preset compensation coefficient;

[0154] A new mouse pointer position corresponding to the empty mouse on the screen is determined according to the offset data and the mouse pointer position.

[0155] In an optional implementation, after the image acquisition unit 301 acquires the screen image corresponding to the air mouse pointing, the device further includes an image adjustment unit 307, and the image adjustment unit 307 is specifically used to:

[0156] Detecting whether the screen image meets the quadrilateral contour extraction standard;

[0157] If not, the screen image is processed, wherein the processing includes adjusting the image contrast, brightness, noise removal and edge enhancement of the screen image.

[0158] Furthermore, an embodiment of the present invention further provides a storage medium, wherein the storage medium is used to store a computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the above Figure 1-2 The method for determining the position of the empty mouse pointer described in .

[0159] Furthermore, an embodiment of the present invention further provides a processor, the processor is used to run a program, wherein the program executes the above Figure 1-2 The method for determining the position of the empty mouse pointer described in .

[0160] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0161] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, but do not represent the advantages and disadvantages of the embodiments.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0164] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0174] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for determining the position of an empty mouse pointer, characterized in that: The method comprises: Get the screen image corresponding to the empty mouse pointing; Extracting the quadrilateral outline of the screen image using an outline extraction algorithm; Determine the geometric center point of the quadrilateral outline using a geometric center point determination method; Using a position determination method, based on the geometric center point, determine the corresponding mouse pointer position of the empty mouse on the screen; The mouse pointer position is sent to a host so that the host displays the mouse pointer at the mouse pointer position on the screen.

2. The method according to claim 1, characterized in that Determining the geometric center point of the quadrilateral outline using a geometric center point determination method includes: Determine whether the quadrilateral outline is a standard quadrilateral; If not, the geometric center point of the quadrilateral outline is determined using a method for determining the geometric center point corresponding to a non-standard quadrilateral, wherein the method for determining the geometric center point corresponding to the non-standard quadrilateral is a diagonal intersection method; If so, the geometric center point of the quadrilateral outline is determined using a method for determining the geometric center point corresponding to a standard quadrilateral, wherein the method for determining the geometric center point corresponding to the standard quadrilateral is a vertex average method.

3. The method according to claim 1, characterized in that Using a position determination method, determining the mouse pointer position corresponding to the empty mouse on the screen based on the geometric center point includes: Acquire the screen size corresponding to the screen image, where the screen size includes screen width and screen height; Obtaining the boundary of the camera coordinate system for capturing the screen image; The coordinate conversion method is used to obtain the corresponding mouse pointer position of the air mouse on the screen based on the screen size, the boundary of the camera coordinate system and the geometric center point.

4. The method according to claim 1, characterized in that: After determining the position of the mouse pointer corresponding to the air mouse on the screen based on the geometric center point using the position determination method, the method further includes: The fusion correction algorithm is used to determine the new mouse pointer position corresponding to the empty mouse on the screen.

5. The method according to claim 4, characterized in that The fusion correction algorithm is used to determine the new mouse pointer position corresponding to the empty mouse on the screen, including: When the direction of the air mouse remains unchanged, acquiring attitude angle data obtained from the gyroscope and accelerometer of the air mouse; The sensor fusion algorithm is used to fuse the attitude angle data obtained by the air mouse gyroscope and the accelerometer to obtain the fused attitude angle data; A new mouse pointer position corresponding to the air mouse on the screen is determined according to the fused posture angle data and the mouse pointer position.

6. The method according to claim 5, characterized in that Determining a new mouse pointer position corresponding to the air mouse on the screen according to the fused posture angle data and the mouse pointer position includes: Obtaining offset data according to the fused attitude angle data and a preset compensation coefficient; A new mouse pointer position corresponding to the empty mouse on the screen is determined according to the offset data and the mouse pointer position.

7. The method according to claim 1, characterized in that After acquiring the screen image corresponding to the air mouse pointing, the method further includes: Detecting whether the screen image meets the quadrilateral contour extraction standard; If not, the screen image is processed, wherein the processing includes adjusting the image contrast, brightness, noise removal and edge enhancement of the screen image.

8. A device for determining the position of an empty mouse pointer, characterized in that: The device comprises: An image acquisition unit, used to acquire a screen image corresponding to the pointing of the air mouse; A contour extraction unit, used for extracting the quadrilateral contour of the screen image acquired by the image acquisition unit by using a contour extraction algorithm; A center point determination unit, used to determine the geometric center point of the quadrilateral outline extracted by the outline extraction unit using a geometric center point determination method; A position determination unit, used to determine the position of the mouse pointer corresponding to the air mouse on the screen based on the geometric center point determined by the center point determination unit using a position determination method; The position sending unit is used to send the mouse pointer position determined by the position determining unit to the host, so that the host can display the mouse pointer at the mouse pointer position on the screen.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the method for determining the position of an air mouse pointer as described in any one of claims 1 to 7.

10. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the method for determining the position of an air mouse pointer as described in any one of claims 1 to 7.

Images