Cursor remote control method for projecting fingertip to screen and equipment interaction system
By accurately acquiring the three-dimensional coordinates of the fingertip through a dual-camera system and combining it with specific gestures to control the cursor, the accuracy and cost-effectiveness issues of non-contact touchscreen methods are solved, making it suitable for catering and medical scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-04-07
AI Technical Summary
Existing contactless touchscreen methods struggle to balance accuracy and cost-effectiveness; monocular cameras cannot determine the physical location of the fingertip in space, and 3D cameras are prohibitively expensive.
It uses dual cameras to capture images simultaneously, solves the three-dimensional coordinates of the fingertip through a system of linear equations, and controls cursor movement and clicks by specific gestures, thereby reducing hardware costs.
It achieves high precision and economy in non-contact cursor remote control, avoids cross-contamination of bacteria, reduces public health risks, and is suitable for catering and medical scenarios.
Smart Images
Figure CN121807188A_ABST
Abstract
Description
Technical Field
[0001] This application relates primarily to the field of device interaction technology, and in particular to a method for remotely controlling a cursor by projecting fingertip onto a screen, a device interaction system, a computer-readable storage medium, and a computer program product. Background Technology
[0002] In commercial and industrial environments (such as catering and medical settings), touchscreen devices pose problems of cross-infection and operational inconvenience. One solution is to completely replace contactless touchscreen control by using a non-contact clicking method, where the cursor on the screen is controlled by the movement of the fingertip at a distance from the screen, and a click is triggered by a gesture.
[0003] There are several technical approaches to current contact gesture interaction technology. One approach is to use a monocular camera, typically mounted on the device with the lens facing the user. It controls software elements on the screen by recognizing single-frame gestures or dynamic gestures in media streams. The main drawback is that it cannot determine the spatial physical position of the fingertip using a single camera. Another approach is to use a 3D camera (including structured light, TOF, etc.), which is similar to the basic idea of a monocular camera but adds 3D information to assist in accurate finger spatial icon detection. However, this approach is more expensive.
[0004] In other words, monocular cameras lack the ability to determine the physical location of the fingertip, resulting in insufficient interaction accuracy. While 3D cameras can acquire 3D information for accurate detection, their high cost makes it difficult to balance the accuracy and cost-effectiveness requirements of contactless interaction scenarios. Summary of the Invention
[0005] The technical problem to be solved by this application is to provide a cursor remote control method and device interaction system that projects fingertips onto the screen, thereby addressing the difficulty of existing non-contact touch screen methods in simultaneously meeting the accuracy and cost requirements of non-contact interaction scenarios.
[0006] To address the aforementioned technical problems, this application provides a method for remotely controlling a cursor projected onto a screen using fingertip projection, comprising: simultaneously capturing a first image and a second image containing a gesture using two cameras; detecting a target fingertip feature point and at least two sets of reference feature points from the first image and the second image respectively; obtaining the pixel coordinates of the target fingertip feature point in the first image and the second image, and the pixel coordinates of the at least two sets of reference feature points; constructing a system of linear equations based on the pixel coordinates of the target fingertip feature point in the first image and the second image, and preset parameters of the two cameras, solving the system of linear equations to obtain the three-dimensional coordinates of the target fingertip feature point in a stereoscopic vision coordinate system; transforming the three-dimensional coordinates to the screen coordinate system to obtain the screen pixel coordinates of the target fingertip feature point, and controlling the cursor to move to the screen pixel coordinates; calculating the pixel distance between each of the at least two sets of reference feature points, and triggering or not triggering a cursor click command based on the comparison result of the pixel distances.
[0007] Optionally, the two cameras are referred to as the first camera and the second camera, respectively. Constructing the linear equation system includes: constructing a first collinearity equation based on the pixel coordinates of the target fingertip feature point in the first image and the preset parameters of the first camera; constructing a second collinearity equation based on the pixel coordinates of the target fingertip feature point in the second image and the preset parameters of the second camera; and combining the first collinearity equation and the second collinearity equation to obtain the linear equation system.
[0008] Optionally, the system of linear equations is:
[0009]
[0010] Where (u1, v1) are the pixel coordinates of the target fingertip feature point in the first image.
[0011] (u2, v2) are the pixel coordinates of the target fingertip feature point in the second image, and (X, Y, Z) are the three-dimensional coordinates of the target fingertip feature point in the stereo vision coordinate system. These are the preset parameters for the first camera. These are the preset parameters for the second camera.
[0012] Optionally, the preset parameters of the first camera are obtained by the dot product of the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera, and the preset parameters of the second camera are obtained by the dot product of the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera.
[0013] Optionally, transforming the three-dimensional coordinates to the screen coordinate system to obtain the screen pixel coordinates of the target fingertip feature point includes: transforming the three-dimensional coordinates (X, Y, Z) in the stereo vision coordinate system to the three-dimensional coordinates (Xs, Ys, Zs) in the screen coordinate system; extracting the two-dimensional coordinates (Xs, Ys) from the three-dimensional coordinates (Xs, Ys, Zs) in the screen coordinate system, and using the two-dimensional coordinates (Xs, Ys) as the screen pixel coordinates of the target fingertip feature point.
[0014] Optionally, the three-dimensional coordinates can be transformed to the screen coordinate system using the following formula:
[0015]
[0016] Where (X, Y, Z) are the three-dimensional coordinates in the stereo vision coordinate system, and (Xs, Ys, Zs) are the three-dimensional coordinates in the screen coordinate system. and This is the transformation matrix from the stereo vision coordinate system to the screen coordinate system.
[0017] Optionally, the method further includes calibrating the two cameras to obtain the preset parameters, including: performing single-target calibration on the two cameras using a calibration board to obtain the intrinsic parameter matrices of the two cameras respectively; performing dual-target calibration to obtain the extrinsic parameter matrices of the two cameras respectively; and calibrating the positional relationship between the two cameras and the screen plane to obtain the transformation matrix from the stereo vision coordinate system to the screen coordinate system.
[0018] Optionally, the method further includes: obtaining a first distortion coefficient of the camera that captured the first image, and performing distortion correction on the first image using the first distortion coefficient; obtaining a second distortion coefficient of the camera that captured the second image, and performing distortion correction on the second image using the second distortion coefficient.
[0019] Optionally, the at least two sets of reference feature points include a first reference point pair and a second reference point pair. The pixel Euclidean distance between the first reference point pair and the pixel Euclidean distance between the second reference point pair are calculated. When the pixel Euclidean distance between the first reference point pair is less than the pixel Euclidean distance between the second reference point pair, a cursor click command is triggered.
[0020] To address the aforementioned technical problems, this application provides a device interaction system, comprising: a dual-camera module for simultaneously capturing a first image and a second image containing a gesture; a key point detection module for detecting a target fingertip feature point and at least two sets of reference feature points from the first image and the second image respectively, obtaining the pixel coordinates of the target fingertip feature point in the first image and the second image, and the pixel coordinates of the at least two sets of reference feature points; a coordinate calculation module for constructing a system of linear equations based on the pixel coordinates of the target fingertip feature point in the first image and the second image, and preset parameters of the two cameras, solving the system of linear equations to obtain the three-dimensional coordinates of the target fingertip feature point in a stereoscopic vision coordinate system; a motion mapping module for converting the three-dimensional coordinates to a screen coordinate system to obtain the screen pixel coordinates of the target fingertip feature point, and controlling the cursor to move to the screen pixel coordinates; and a click control module for calculating the pixel distance between each of the at least two sets of reference feature points, and triggering or not triggering a cursor click command based on the comparison result of the pixel distances.
[0021] Optionally, the dual-camera module includes two monocular cameras, each monocular camera having a field of view that covers at least three corners of the screen. The fields of view of the two monocular cameras overlap, and the overlapping area covers the entire space of the user's gesture operation in front of the screen. It can also simultaneously capture images of any fingertip resting within the overlapping area, adapting to the palm recognition needs of left-handed and right-handed users.
[0022] Optionally, the two monocular cameras are arranged symmetrically along the horizontal edge of the screen, and are in one of the following two layouts: Layout 1: The two monocular cameras are respectively installed in the upper left corner and the upper right corner of the screen; Layout 2: The two monocular cameras are respectively installed in the lower left corner and the lower right corner of the screen.
[0023] Optionally, it further includes: a calibration module for calibrating the two cameras to obtain the preset parameters, including using a calibration board to perform single-target calibration on the two cameras respectively to obtain the intrinsic parameter matrix of each of the two cameras; performing dual-target calibration on the two cameras to obtain the extrinsic parameter matrix; calibrating the positional relationship between the two cameras and the screen plane to obtain the transformation matrix from the stereo vision coordinate system to the screen coordinate system.
[0024] Optionally, it further includes: an image calibration module, configured to obtain a first distortion coefficient of the camera that captured the first image, and perform distortion correction on the first image using the first distortion coefficient; and to obtain a second distortion coefficient of the camera that captured the second image, and perform distortion correction on the second image using the second distortion coefficient.
[0025] To address the aforementioned technical problems, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a processor, cause the processor to perform the method described above.
[0026] To address the aforementioned technical problems, this application provides a computer program product containing instructions that, when executed by a processor, cause the processor to perform the method described above.
[0027] Compared with the prior art, this application has the following advantages:
[0028] This application's fingertip projection cursor remote control method and device interaction system completely replace traditional contact-based touchscreen control through a fully contactless operation process of "fingert movement to control the cursor and specific gestures to trigger clicks." In high-frequency, multi-user scenarios such as restaurant ordering and medical consultations, it can completely avoid users' direct hand contact with the screen surface, solving the problem of cross-transmission of bacteria and viruses caused by repeated touches at the source of interaction, and reducing public health and safety risks. Specifically, this application uses dual cameras to simultaneously acquire images and solves the three-dimensional coordinates of the fingertip based on the principle of binocular visual triangulation, accurately obtaining the spatial position of the fingertip within the operating area in front of the screen, ensuring the synchronization and accuracy of cursor movement and fingertip actions. Attached Figure Description
[0029] The accompanying drawings are included to provide a further understanding of this application. They are incorporated into and constitute a part of this application. The drawings illustrate embodiments of this application and, together with this specification, serve to explain the principles of this application.
[0030] Figure 1 This is a flowchart of a method for remotely controlling a cursor by projecting a fingertip onto a screen, according to an embodiment of this application.
[0031] Figure 2 This is a diagram showing the relative positions of two cameras and the screen according to an embodiment of this application.
[0032] Figure 3 This is a schematic diagram of hand joint feature points according to an embodiment of this application.
[0033] Figure 4 This is a schematic diagram of a first gesture according to an embodiment of this application.
[0034] Figure 5 This is a schematic diagram of a second gesture according to an embodiment of this application.
[0035] Figure 6 This is a schematic diagram showing the position of the target fingertip feature point and the cursor according to an embodiment of this application.
[0036] Figure 7This is a system block diagram of a device interaction system according to an embodiment of this application. Detailed Implementation
[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.
[0038] This application provides a method for remotely controlling a cursor projected from a fingertip onto a screen. The method converts the pixel coordinates of a two-dimensional image of the fingertip captured by two cameras into three-dimensional spatial coordinates in the screen coordinate system, projects this into a cursor on the two-dimensional plane of the screen, and finally triggers a click on the screen cursor by touching it with the thumb and middle finger.
[0039] Figure 1 This is a flowchart of a cursor remote control method for projecting a fingertip onto a screen, according to an embodiment of this application. Figure 1 As shown, the cursor remote control method 100, which projects a fingertip onto the screen, includes:
[0040] Step S1: Simultaneously capture a first image and a second image containing the gesture using two cameras.
[0041] Figure 2 This is a diagram showing the relative positions of two cameras and the screen according to an embodiment of this application. Figure 2 As shown, a first camera and a second camera are respectively installed at the upper left and upper right corners of the screen, both facing diagonally across the screen. The field of view of each camera covers at least three corner points of the screen. The fields of view of the first and second cameras overlap, covering the entire area of the user's hand gestures in front of the screen, and can simultaneously capture images of any fingertip resting within this overlapping area, adapting to the palm recognition needs of both left-handed and right-handed users. In this application, the first and second cameras are monocular cameras.
[0042] In other embodiments, the first camera and the second camera are respectively mounted at the lower left and lower right corners of the screen. That is, the two monocular cameras of this application are symmetrically arranged along the horizontal edge of the screen, and are in one of the following two layouts:
[0043] Layout 1: Two monocular cameras are respectively installed in the upper left and upper right corners of the screen;
[0044] Layout 2: Two monocular cameras are installed in the lower left and lower right corners of the screen, respectively.
[0045] In other words, the two cameras should be arranged either from the top left and top right, or from the bottom left and bottom right. If both are on the left or both on the right, the palm of the hand may not be recognized in some cases, depending on whether the user is left-handed or right-handed. Diagonal placement cannot achieve binocular positioning.
[0046] Step S2: Detect the target fingertip feature points and at least two sets of reference feature points from the first image and the second image, respectively.
[0047] In the first and second images, specific fingertip feature points are located using gesture recognition algorithms (such as deep learning-based keypoint detection models or traditional contour analysis + corner detection algorithms).
[0048] Figure 3 This is a schematic diagram of hand joint feature points according to an embodiment of this application. For example... Figure 3 As shown, the hand joint feature points include thumb fingertip feature point A1, index fingertip feature point B1, middle finger third joint feature point C3, and ring finger third joint feature point D3. In this application, index fingertip feature point B1 is used as the target fingertip feature point to control "cursor movement".
[0049] The thumb tip feature point A1 and the middle finger third joint feature point C3 are used as one set of reference feature points, and the middle finger third joint feature point C3 and the ring finger third joint feature point D3 are used as another set of reference feature points to determine the "click action".
[0050] This application defines two key gestures:
[0051] The first gesture resembles a simulated pistol stance and is used for cursor pointing. The vertical intersection of the index finger tip and the screen is the cursor's display position. Figure 4 As shown, the vertical intersection point of the screen pointed to by the fingertip feature point B1 is the cursor display position, that is, the position of the cursor on the screen moves as the position of the fingertip feature point B1 moves.
[0052] Gesture two: Press your thumb down and interlock it with your middle finger, forming the following shape. Figure 5 The gesture shown triggers a cursor click operation. Figure 5 As shown, the position of the index fingertip feature point B1 determines the position of the cursor on the screen. When the distance between the thumb fingertip feature point A1 and the middle finger third joint feature point C3 is less than the distance between the middle finger third joint feature point C3 and the ring finger third joint feature point D3, the cursor click operation is triggered.
[0053] Step S3: Obtain the pixel coordinates of the target fingertip feature points in the first image and the second image, respectively, as well as the pixel coordinates of at least two sets of reference feature points.
[0054] The pixel coordinates of the target fingertip feature point B1 in the first image are P1(u1, v1); the pixel coordinates of the target fingertip feature point B1 in the second image are P2(u2, v2).
[0055] Step S4: Based on the pixel coordinates of the target fingertip feature point in the first and second images, and the preset parameters of the two cameras, construct a system of linear equations, solve the system of linear equations, and obtain the three-dimensional coordinates of the target fingertip feature point in the stereo vision coordinate system.
[0056] Two cameras form a binocular vision system. In this system, the target fingertip feature point B1, the optical center O1 of the first camera, and the optical center O2 of the second camera form a triangle. In the first image, the image point P1 (u1, v1) of B1 lies on the line connecting O1 and B1, and in the second image, the image point P2 (u2, v2) lies on the line connecting O2 and B1.
[0057] For the first camera, a first collinearity equation is constructed based on the pixel coordinates of the target fingertip feature point in the first image and the preset parameters of the first camera:
[0058] (1)
[0059] Where S1 is the scale factor of the first camera, (u1, v1) are the pixel coordinates of the target fingertip feature point in the first image, (X, Y, Z) are the three-dimensional coordinates of the target fingertip feature point in the stereo vision coordinate system, K1 is the intrinsic parameter matrix of the first camera, I is the identity matrix, and [I 0] is the extrinsic parameter matrix of the first camera.
[0060] For the second camera, a second collinearity equation is constructed based on the pixel coordinates of the target fingertip feature point in the second image and the preset parameters of the second camera:
[0061] (2)
[0062] Where S2 is the scale factor of the second camera, (u2, v2) are the pixel coordinates of the target fingertip feature point in the second image, (X, Y, Z) are the three-dimensional coordinates of the target fingertip feature point in the stereo vision coordinate system, K2 is the intrinsic parameter matrix of the second camera, [RT] is the extrinsic parameter matrix of the second camera, and [RT] is the rotation matrix R and translation vector T of the second camera relative to the first camera obtained by performing dual-target positioning with the first camera as the origin.
[0063] For formula (1), where, (3)
[0064] M1 is a 3x4 projection matrix, which can be represented as:
[0065] (4)
[0066] Substituting formulas (3) and (4) into formula (1) and expanding, for the first camera:
[0067]
[0068] Substituting (1c) into (1a) and (1b), eliminate S1:
[0069]
[0070] We obtain two linear equations:
[0071]
[0072] For formula (2), let's denote... M2 is a 3x4 projection matrix, which can be represented as:
[0073] (5)
[0074] Similarly, for the second camera, expanding and eliminating S2, we obtain two linear equations:
[0075]
[0076] Combining the four linear equations into matrix form yields a system of linear equations:
[0077]
[0078] The above system of linear equations can be expressed in the form AX = B, where:
[0079] A is a 4×3 matrix (overdetermined system).
[0080] X = [X, Y, Z] represents the three-dimensional coordinates of the unknown target fingertip feature point in the stereo vision coordinate system.
[0081] B is a 4×1 vector.
[0082] Since this is an overdetermined system (the number of equations exceeds the number of unknowns), the least squares method can be used to solve it.
[0083]
[0084] By solving the above system of linear equations, we can find the coordinates (X, Y, Z) that best fit all the equations, which lie in the stereo vision coordinate system.
[0085] Step S5: Transform the 3D coordinates to the screen coordinate system to obtain the screen pixel coordinates of the target fingertip feature point, and control the cursor to move to the screen pixel coordinates.
[0086] The "visual-screen transformation matrix" obtained during the calibration phase can be used to transform (X,Y,Z) in the stereo vision coordinate system into (Xs,Ys,Zs) in the screen coordinate system, thus establishing a direct relationship between spatial points and the screen.
[0087] In one embodiment, transforming three-dimensional coordinates to screen coordinates includes:
[0088] The following formula is used to transform the 3D coordinates to the screen coordinate system:
[0089]
[0090] Where (X, Y, Z) are the 3D coordinates in the stereo vision coordinate system, and (Xs, Ys, Zs) are the 3D coordinates in the screen coordinate system. and This is the transformation matrix from the stereo vision coordinate system to the screen coordinate system.
[0091] Extract the two-dimensional coordinates (Xs, Ys) from the three-dimensional coordinates (Xs, Ys, Zs) in the screen coordinate system, and use the two-dimensional coordinates (Xs, Ys) as the screen pixel coordinates of the target fingertip feature point.
[0092] Figure 6 This is a schematic diagram showing the position of the target fingertip feature point and the cursor according to an embodiment of this application. Figure 6 As shown, the system moves the cursor to the screen pixel coordinates of the target fingertip feature point in real time, achieving a synchronous effect of "fingertip movement, cursor movement".
[0093] Step S6: Calculate the pixel distance between at least two sets of reference feature points, and trigger or not trigger the cursor click command based on the comparison result of the pixel distance.
[0094] In some embodiments, at least two sets of reference feature points include a first reference point pair and a second reference point pair. The pixel Euclidean distance between the first reference point pair and the second reference point pair is calculated. When the pixel Euclidean distance between the first reference point pair is less than the pixel Euclidean distance between the second reference point pair, a cursor click command is triggered.
[0095] like Figure 5 As shown, calculate the pixel Euclidean distance (Dist_a1c3) between the thumb tip feature point A1 and the middle finger third joint feature point C3, and the pixel Euclidean distance (Dist_c3d3) between the middle finger third joint feature point C3 and the ring finger third joint feature point D3.
[0096] Assume that the pixel coordinates of the feature point C3 of the third joint of the middle finger in the first image are (u3, v3);
[0097] The pixel coordinates of feature point D3 of the third joint of the ring finger in the first image are (u4, v4);
[0098] The formula for calculating the pixel Euclidean distance Dist_c3d3 is as follows:
[0099] .
[0100] The pixel Euclidean distance Dist_a1c3 can also be calculated based on the pixel coordinates of the thumb tip feature point A1 and the middle finger third joint feature point C3 in the first image.
[0101] In some embodiments, the pixel Euclidean distance Dist_a1c3 and pixel Euclidean distance Dist_c3d3 can also be calculated based on the pixel coordinates of the thumb tip feature point A1, the middle finger third joint feature point C3, and the ring finger third joint feature point D3 in the second image.
[0102] Trigger conditions are set: when Dist_a1c3 < Dist_c3d3 (usually corresponding to "finger bending" or "pinch" actions), the "cursor click" command is triggered; otherwise, the "cursor not click" state is maintained. This application determines the interaction based on pixel distance, eliminating the need for an additional pressure sensor and relying solely on visual features.
[0103] In some embodiments, prior to gesture interaction, the two cameras are calibrated to obtain preset parameters. These preset parameters include an intrinsic parameter matrix, a rotation matrix between the two cameras, a translation vector, and a vision-screen transformation matrix. Calibration is the "calibration step" of the entire system, aiming to eliminate camera hardware errors, establish spatial correlation between the two cameras, and define the positional correspondence between the vision system and the screen, providing accurate parameters for subsequent coordinate calculations. This includes:
[0104] The two cameras were individually calibrated using a calibration board to obtain the intrinsic parameter matrices K1, K2 and distortion coefficients dist1, dist2 for each camera.
[0105] Using the first camera as the origin, for dual-target positioning, the extrinsic parameter matrix of the first camera is:
[0106] [I 0], where I is the identity matrix; the extrinsic parameter matrix of the second camera is [RT], where R is the rotation matrix of the second camera relative to the first camera, and T is the translation vector of the second camera relative to the first camera.
[0107] By calibrating the positional relationship between the two cameras and the screen plane, the transformation matrix from the stereo vision coordinate system to the screen coordinate system is obtained. , .
[0108] Optionally, in some embodiments, the method further includes correcting the original images captured by the two cameras using the obtained distortion coefficients to eliminate distortion caused by the wide-angle lens and obtain an image conforming to the pinhole imaging model. Specifically, a first distortion coefficient of the camera capturing the first image is obtained, and distortion correction is performed on the first image using the first distortion coefficient; a second distortion coefficient of the camera capturing the second image is obtained, and distortion correction is performed on the second image using the second distortion coefficient.
[0109] This application also provides a device interaction system. Figure 7 This is a system block diagram of a device interaction system according to an embodiment of this application. Figure 7 As shown, the device interaction system 700 includes: a dual-camera module 71, a key point detection module 72, a coordinate calculation module 73, a motion mapping module 74, and a click control module 75.
[0110] The dual-camera module 71 is used to simultaneously capture a first image and a second image containing gestures. The keypoint detection module 72 is used to detect the target fingertip feature point and at least two sets of reference feature points from the first and second images respectively, obtaining the pixel coordinates of the target fingertip feature point in the first and second images, and the pixel coordinates of the at least two sets of reference feature points. The coordinate calculation module 73 is used to construct a system of linear equations based on the pixel coordinates of the target fingertip feature point in the first and second images and the preset parameters of the two cameras, and solve the system of linear equations to obtain the three-dimensional coordinates of the target fingertip feature point in the stereo vision coordinate system. The motion mapping module 74 is used to transform the three-dimensional coordinates to the screen coordinate system to obtain the screen pixel coordinates of the target fingertip feature point, and control the cursor to move to the screen pixel coordinates. The click control module 75 is used to calculate the pixel distance between each of the at least two sets of reference feature points, and trigger or not trigger the cursor click command based on the comparison result of the pixel distance.
[0111] The dual-camera module 71 includes two monocular cameras. The field of view of each monocular camera covers at least three corners of the screen. The field of view of the two monocular cameras overlaps, covering the entire space of the user's gesture operation in front of the screen. It can also simultaneously capture images of any fingertip in the overlapping area, adapting to the palm recognition needs of left-handed and right-handed users.
[0112] Two monocular cameras are symmetrically arranged along the horizontal edge of the screen, and in one of the following two layouts:
[0113] Layout 1: Two monocular cameras are respectively installed in the upper left and upper right corners of the screen;
[0114] Layout 2: Two monocular cameras are installed in the lower left and lower right corners of the screen, respectively.
[0115] In some embodiments, the device interaction system further includes a calibration module. This calibration module is used to calibrate the two cameras to obtain preset parameters, including performing single-target calibration on each camera using a calibration board to obtain the intrinsic parameter matrices of each camera; performing dual-target calibration on the two cameras to obtain the extrinsic parameter matrices; and calibrating the positional relationship between the two cameras and the screen plane to obtain the transformation matrix from the stereo vision coordinate system to the screen coordinate system.
[0116] In some embodiments, the device interaction system further includes an image calibration module. This image calibration module is used to extract a first distortion coefficient from the intrinsic parameter matrix of the camera capturing the first image, and to perform distortion correction on the first image using the first distortion coefficient; and to extract a second distortion coefficient from the intrinsic parameter matrix of the camera capturing the second image, and to perform distortion correction on the second image using the second distortion coefficient.
[0117] The cursor remote control method and device interaction system for fingertip projection onto the screen in this application have the following beneficial effects:
[0118] 1) The entire process of contactless operation, which involves "controlling the cursor with fingertips and triggering clicks with specific gestures," completely replaces traditional contact-based touchscreen control. In high-frequency, multi-user scenarios such as restaurant ordering and medical consultations, it can completely avoid users' hands directly touching the screen surface, solving the problem of cross-infection of bacteria and viruses caused by repeated touches at the source of interaction, and reducing public health and safety risks.
[0119] 2) To address the core deficiency of monocular cameras in being unable to determine the physical location of the fingertip, this application uses dual cameras to simultaneously acquire images. Based on the principle of binocular visual triangulation, the least squares method is used to solve the three-dimensional coordinates of the fingertip. This can accurately obtain the spatial location of the fingertip within the operating area in front of the screen, ensuring the synchronization and accuracy of cursor movement and fingertip actions, and overcoming the pain point of monocular cameras being "susceptible to distance interference in two-dimensional recognition".
[0120] 3) Compared with existing 3D camera technology, this solution uses dual monocular cameras to build a vision system, eliminating the need for additional 3D sensing modules, significantly reducing hardware costs, and solving the application limitations of 3D camera technology that is "precise but expensive".
[0121] 4) This solution can be directly integrated into touchscreen-controlled tablet devices, requiring only the addition of dual monocular cameras at the corners of the screen. No major modifications to the main device structure are needed, lowering the upgrade threshold for existing touchscreen devices. For existing equipment such as restaurant self-service ordering machines and medical bedside monitors, modular installation can upgrade to contactless interaction functions, extending the device's lifespan.
[0122] This application also provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a network device or stored on any usable medium. When the computer program product runs on at least one network device, it causes the at least one network device to perform a cursor remote control method involving fingertip projection onto a screen.
[0123] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a network device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the network device to execute a cursor remote control method involving fingertip projection onto a screen.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0126] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. In addition, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of this description. Moreover, this application should be understood not only through the actual terms used, but also through the meaning implied by each term.
[0127] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.
[0128] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for remotely controlling a cursor projected from a fingertip onto a screen, characterized in that, include: The first and second images, containing gestures, are captured simultaneously using two cameras. Target fingertip feature points and at least two sets of reference feature points are detected from the first image and the second image, respectively; Obtain the pixel coordinates of the target fingertip feature points in the first image and the second image, respectively, as well as the pixel coordinates of the at least two sets of reference feature points; Based on the pixel coordinates of the target fingertip feature point in the first image and the second image, and the preset parameters of the two cameras, a system of linear equations is constructed. The system of linear equations is solved to obtain the three-dimensional coordinates of the target fingertip feature point in the stereo vision coordinate system. The three-dimensional coordinates are transformed into the screen coordinate system to obtain the screen pixel coordinates of the target fingertip feature point, and the cursor is controlled to move to the screen pixel coordinates. Calculate the pixel distance between each of the at least two sets of reference feature points, and trigger or not trigger the cursor click command based on the comparison result of the pixel distance.
2. The method as described in claim 1, characterized in that, The two cameras are designated as camera 1 and camera 2. The system of linear equations constructed includes: A first collinearity equation is constructed based on the pixel coordinates of the target fingertip feature points in the first image and the preset parameters of the first camera; A second collinearity equation is constructed based on the pixel coordinates of the target fingertip feature point in the second image and the preset parameters of the second camera; The first collinear equation and the second collinear equation are combined to obtain the linear equation system.
3. The method as described in claim 2, characterized in that, The system of linear equations is as follows: Where (u1, v1) are the pixel coordinates of the target fingertip feature point in the first image. (u2, v2) are the pixel coordinates of the target fingertip feature point in the second image, and (X, Y, Z) are the three-dimensional coordinates of the target fingertip feature point in the stereo vision coordinate system. These are the preset parameters for the first camera. These are the preset parameters for the second camera.
4. The method as described in claim 3, characterized in that, The preset parameters of the first camera are obtained by the dot product of the intrinsic parameter matrix and the extrinsic parameter matrix of the first camera, and the preset parameters of the second camera are obtained by the dot product of the intrinsic parameter matrix and the extrinsic parameter matrix of the second camera.
5. The method as described in claim 1, characterized in that, Transforming the three-dimensional coordinates to the screen coordinate system to obtain the screen pixel coordinates of the target fingertip feature point includes: Transform the 3D coordinates (X, Y, Z) in the stereo vision coordinate system to the 3D coordinates (Xs, Ys, Zs) in the screen coordinate system; Extract the two-dimensional coordinates (Xs, Ys) from the three-dimensional coordinates (Xs, Ys, Zs) in the screen coordinate system, and use the two-dimensional coordinates (Xs, Ys) as the screen pixel coordinates of the target fingertip feature point.
6. The method as described in claim 5, characterized in that, The three-dimensional coordinates are transformed to the screen coordinate system using the following formula: Where (X, Y, Z) are the three-dimensional coordinates in the stereo vision coordinate system, and (Xs, Ys, Zs) are the three-dimensional coordinates in the screen coordinate system. and This is the transformation matrix from the stereo vision coordinate system to the screen coordinate system.
7. The method as described in claim 1, characterized in that, It also includes calibrating the two cameras to obtain the preset parameters, including: The two cameras were individually calibrated using a calibration board to obtain the intrinsic parameter matrices of each camera. Perform dual-target positioning to obtain the extrinsic parameter matrices of each of the two cameras; The positional relationship between the two cameras and the screen plane is determined, and the transformation matrix from the stereo vision coordinate system to the screen coordinate system is obtained.
8. The method as described in claim 1, characterized in that, Also includes: Obtain the first distortion coefficient of the camera that captured the first image, and perform distortion correction on the first image using the first distortion coefficient; Obtain the second distortion coefficient of the camera that captured the second image, and perform distortion correction on the second image using the second distortion coefficient.
9. The method as described in claim 1, characterized in that, The at least two sets of reference feature points include a first reference point pair and a second reference point pair. The pixel Euclidean distance between the first reference point pair and the second reference point pair is calculated. When the pixel Euclidean distance between the first reference point pair is less than the pixel Euclidean distance between the second reference point pair, a cursor click command is triggered.
10. A device interaction system, characterized in that, include: A dual-camera module is used to simultaneously capture a first image and a second image that includes gestures; The key point detection module is used to detect target fingertip feature points and at least two sets of reference feature points from the first image and the second image respectively, and to obtain the pixel coordinates of the target fingertip feature points in the first image and the second image respectively, as well as the pixel coordinates of the at least two sets of reference feature points; The coordinate calculation module is used to construct a system of linear equations based on the pixel coordinates of the target fingertip feature point in the first image and the second image, as well as the preset parameters of the two cameras, and solve the system of linear equations to obtain the three-dimensional coordinates of the target fingertip feature point in the stereo vision coordinate system. The motion mapping module is used to transform the three-dimensional coordinates to the screen coordinate system to obtain the screen pixel coordinates of the target fingertip feature point, and control the cursor to move to the screen pixel coordinates. as well as The click control module is used to calculate the pixel distance between each of the at least two sets of reference feature points, and to trigger or not trigger the cursor click command based on the comparison result of the pixel distance.
11. The device interaction system as described in claim 10, characterized in that, The dual-camera module includes two monocular cameras. The field of view of each monocular camera covers at least three corners of the screen. The field of view of the two monocular cameras overlaps, and the overlapping area covers the entire space of the user's gesture operation in front of the screen. It can also simultaneously capture images of any fingertip resting in the overlapping area, adapting to the palm recognition needs of left-handed and right-handed users.
12. The device interaction system as described in claim 11, characterized in that, Two monocular cameras are symmetrically arranged along the horizontal edge of the screen, and in one of the following two layouts: Layout 1: Two monocular cameras are respectively installed in the upper left and upper right corners of the screen; Layout 2: Two monocular cameras are installed in the lower left and lower right corners of the screen, respectively.
13. The device interaction system as described in claim 10, characterized in that, Also includes: The calibration module is used to calibrate the two cameras to obtain the preset parameters, including using a calibration board to perform single-target calibration on the two cameras to obtain the intrinsic parameter matrices of the two cameras respectively; performing dual-target calibration on the two cameras to obtain the extrinsic parameter matrices; and calibrating the positional relationship between the two cameras and the screen plane to obtain the transformation matrix from the stereo vision coordinate system to the screen coordinate system.
14. The device interaction system as described in claim 13, characterized in that, Also includes: An image calibration module is used to obtain the first distortion coefficient of the camera that captured the first image, and to perform distortion correction on the first image using the first distortion coefficient; Obtain the second distortion coefficient of the camera that captured the second image, and perform distortion correction on the second image using the second distortion coefficient.
15. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-9.
16. A computer program product containing instructions, characterized in that, When the instruction is executed by the processor, the processor performs the method as described in any one of claims 1-9.