Computer-implemented method, user input system and computer program product

Abstract

A computer-implemented method is provided. The computer-implemented method includes determining a reference surface; determining a target object of the target user; determining a relative displacement between the target object and the reference surface; and determining a user-machine interaction instruction based on the relative displacement.

Description

Technical Field

[0001] The present invention relates to display technology, and more particularly, to a computer-implemented method, a user input system, and a computer program product. Background Art

[0002] The display device generally includes a display panel for displaying images. Sometimes, the display device also includes a touch device such as a touch layer on the display panel. The touch device allows the user to input information by touch writing. Summary of the invention

[0003] In one aspect, the present disclosure provides a computer-implemented method comprising: determining a reference surface; determining a target object of a target user; determining a relative displacement between the target object and the reference surface; and determining a user-machine interaction instruction based on the relative displacement.

[0004] Optionally, determining the reference surface includes: determining coordinates of a plurality of calibration points in a three-dimensional coordinate system; and deriving the reference surface based on the plurality of calibration points.

[0005] Optionally, determining the coordinates of the multiple calibration points in the three-dimensional coordinate system includes: capturing an image of the real-world object using a camera; and based on parameters of the camera, converting the coordinates of the multiple calibration points in the image of the real-world object in the image coordinate system into coordinates of the multiple calibration points in the three-dimensional coordinate system.

[0006] Optionally, the user-machine interaction instruction includes displaying a trajectory image; wherein the computer-implemented method further includes: detecting and tracking an object motion trajectory of the target object relative to the reference surface; and displaying the trajectory image in a target area based on the object motion trajectory relative to the reference surface.

[0007] Optionally, the computer-implemented method further comprises: determining whether the object motion trajectory is transformed into a pause gesture; and after determining that the object motion trajectory is transformed into a pause gesture, pausing display of the trajectory image.

[0008] Optionally, the computer-implemented method further includes: converting the object motion trajectory relative to the reference surface into the trajectory image in real time; pausing the conversion of the object motion trajectory relative to the reference surface into the trajectory image when a pause gesture is detected; and continuing to detect and track the object motion trajectory of the target object relative to the reference surface after pausing the conversion of the object motion trajectory relative to the reference surface into the trajectory image.

[0009] Optionally, the computer-implemented method further comprises: recovering, at a new reference point, converting the object motion trajectory relative to the reference surface into a new trajectory image in real time, and starting the new trajectory image at the new reference point relative to the previous trajectory image; wherein the new reference point is determined based on the result of continuing to detect and track the object motion trajectory relative to the reference surface.

[0010] Optionally, the pause gesture is a gesture in which the target object moves toward a pause direction relative to the reference surface; and if the object making the object motion trajectory moves a distance exceeding a second threshold distance along the pause direction relative to the reference surface, the gesture is recognized as the pause gesture.

[0011] Optionally, the computer-implemented method further includes: determining whether the object motion trajectory is transformed into a display gesture; and when it is determined that the object motion trajectory is transformed into a display gesture, displaying the trajectory image in a target area based on the object motion trajectory relative to the reference surface.

[0012] Optionally, the display gesture is a gesture of making a finger of the target user contact the reference surface.

[0013] Optionally, the computer-implemented method further comprises: identifying an initiation gesture indicating an initiation of providing the user-machine interaction instruction; wherein providing the user-machine interaction instruction is performed after identifying the initiation gesture.

[0014] Optionally, the user-machine interaction instruction includes executing a user command; wherein the computer-implemented method further includes: detecting and tracking the object motion trajectory of the target object relative to the reference surface; and executing the user command based on the object motion trajectory relative to the reference surface.

[0015] Optionally, the user-machine interaction instruction includes adjusting a reference point; wherein adjusting the reference point includes recognizing an adjustment gesture of the target user.

[0016] Optionally, the adjustment gesture is a gesture that causes the target object of the target user to be out of contact with the reference surface and move along an adjustment direction; wherein the computer-implemented method further comprises adjusting the reference point along the adjustment direction of the movement of the target object; wherein, when the movement of the target object is within a distance relative to the reference surface, the gesture is recognized as the adjustment gesture; and the distance is less than a first threshold distance.

[0017] Optionally, the reference point is a point at which display of the trajectory image begins.

[0018] Optionally, the computer-implemented method further comprises: determining a target area on the display panel that is designated for the target user to provide the user-machine interaction instruction.

[0019] Optionally, the user-machine interaction instruction includes displaying a trajectory image in the target area; wherein determining the target area includes: determining a gaze point on the display panel; and determining the target area based on the gaze point.

[0020] Optionally, determining the gaze point includes determining whether K gaze points of the target user determined respectively from K image frames of the target user are located at substantially the same position.

[0021] Optionally, determining the target area includes: determining the coordinates of multiple calibration points in a three-dimensional coordinate system; establishing a spatial equation of a real-world object based on the coordinates of the multiple calibration points in the three-dimensional coordinate system; determining the position coordinates of the target object in the three-dimensional coordinate system; and mapping the position coordinates of the target object in the three-dimensional coordinate system to the input position coordinates in the display panel coordinate system, thereby determining the target area.

[0022] Optionally, the display panel is virtually divided into a plurality of candidate target areas; wherein the computer-implemented method further comprises designating at least one candidate target area among the plurality of candidate target areas as the target area for the target user.

[0023] Optionally, the computer-implemented method also includes: activating the target area; wherein activating the target area includes: identifying a corresponding selection gesture that matches one of multiple candidate selection gestures; and activating a corresponding candidate target area among the multiple candidate target areas to designate it as the target area of ​​the target user.

[0024] Optionally, the user-machine interaction instruction further includes adjusting the target area to obtain an adjusted target area.

[0025] Optionally, the target object includes a palm of the target user; and adjusting the target area includes identifying a gesture for adjusting a position of the palm of the target user.

[0026] Optionally, the computer-implemented method further comprises: determining whether the adjusted target area overlaps with any of the activated target areas in the display panel.

[0027] Optionally, the computer-implemented method further includes: when it is determined that the adjusted target area overlaps with any one of the activated target areas in the display panel, providing a notification to the target user that the adjusted target area overlaps with any one of the activated target areas in the display panel.

[0028] Optionally, the computer-implemented method further comprises: when it is determined that the adjusted target area does not overlap with any of the activated target areas in the display panel, replacing the target area with the adjusted target area for inputting information or providing user commands.

[0029] Optionally, the computer-implemented method also includes: identifying a confirmation gesture of the target user; and when the confirmation gesture is identified, when the target area is subject to adjustment, accepting the adjusted target area as the confirmation area for inputting information or providing user commands, or when the target area is not subject to adjustment, accepting the target area as the confirmation area for inputting information or providing user commands.

[0030] Optionally, identifying the target user includes: detecting and tracking a gesture of the user; and comparing the gesture of the user with a target gesture; wherein, when it is determined that the gesture of the user is the target gesture, the user is identified as the target user.

[0031] On the other hand, the present disclosure provides a user input system, comprising: one or more cameras; a display panel; a memory; and one or more processors; wherein the memory and the one or more processors are connected to each other; and the memory stores computer executable instructions for controlling the one or more processors to perform the following operations: determining a reference surface; determining a target object of a target user; determining a relative displacement between the target object and the reference surface; and determining a user-machine interaction instruction based on the relative displacement.

[0032] In another aspect, the present disclosure provides a computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to: determine a reference surface; determine a target object of a target user; determine a relative displacement between the target object and the reference surface; and based on the relative displacement, determine a user-machine interaction instruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] According to various disclosed embodiments, the following drawings are examples only for illustration purposes and are not intended to limit the scope of the present invention.

[0034] Figure 1is a flow chart illustrating a computer-implemented method in some embodiments according to the present disclosure.

[0035] Figure 2 A top-down image depicting a user's hand.

[0036] Figure 3 A side view image of a user's hand is depicted.

[0037] Figure 4 is a flow chart illustrating a computer-implemented method in some embodiments according to the present disclosure.

[0038] Figure 5 The bounding box of the target user's hand in the corresponding image in the image of the target user's hand is shown.

[0039] Figure 6 is a flow chart illustrating a computer-implemented method in some embodiments according to the present disclosure.

[0040] Figure 7 The process of obtaining the intrinsic and extrinsic parameters of the camera is shown.

[0041] Figure 8 A plurality of candidate selection gestures in accordance with some embodiments of the present disclosure are shown.

[0042] Fig. 9 is a flow chart illustrating a computer-implemented method in some embodiments according to the present disclosure.

[0043] Fig.10 is a flow chart illustrating a computer-implemented method in some embodiments according to the present disclosure.

[0044] Fig.11 is a flow chart illustrating a computer-implemented method in some embodiments according to the present disclosure.

[0045] Fig.12 is a schematic diagram showing a user input system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0046] The present disclosure will now be described in more detail with reference to the following examples. It should be noted that the following description of some of the embodiments presented herein is for illustration and description purposes only. It is not intended to be exhaustive or limited to the precise form disclosed.

[0047] Related input methods usually require a touchpad or a laser pen to input information. The inventors of the present disclosure have found that related input methods are usually not suitable for certain scenarios, especially scenarios where multiple users share the same display panel for displaying user input. For example, when multiple users want to write on the display panel, each of them will have to approach the same display panel.

[0048] Therefore, the present disclosure provides, among other things, a computer-implemented method, a user input system, and a computer program product that substantially obviates one or more of the problems due to limitations and disadvantages of the prior art. In one aspect, the present disclosure provides a computer-implemented method. In some embodiments, the computer-implemented method includes determining a reference surface; determining a target object of a target user; determining a relative displacement between the target object and the reference surface; and determining a user-machine interaction instruction based on the relative displacement.

[0049] In some embodiments, the computer-implemented method includes identifying a target user, the target user being a user who wants to remotely input information or provide a user command on the display panel. Various suitable methods can be used to identify the target user. In some embodiments, identifying the target user includes identifying a target gesture of the user. Examples of target gestures include raising a hand, and a palm staying on a table for a fixed time.

[0050] In some embodiments, a computer-implemented method includes determining a target area on a display panel designated for a target user for inputting information or providing user commands. Various suitable methods may be used to determine the target area. In some embodiments, determining the target area includes determining a gaze point on the display panel; and determining the target area using the gaze point. Optionally, the computer-implemented method includes determining a gaze direction of the target user, and determining a point on the display panel where the gaze direction intersects the plane of the display panel, thereby determining the gaze point. Optionally, the computer-implemented method includes determining the gaze point using a neural network.

[0051] In some embodiments, the gaze point may be used as a reference point. For example, the target area may be an area having the gaze point as a center point.

[0052] In some embodiments, the gaze point can be used as a reference point. In one example, the reference point is the point at which information input by the target user or a user command input by the target user is started to be displayed. Optionally, the reference point is the point at which the trajectory image is started to be displayed. Optionally, the reference point is the point at which the strokes of a character (e.g., a Chinese character) or a letter (e.g., a, b, c, d) are started to be displayed.

[0053] In some embodiments, the computer-implemented method also includes adjusting the reference point. Optionally, adjusting the reference point includes recognizing an adjustment gesture of the target user. In one example, the adjustment gesture is a gesture in which the target object (e.g., finger) of the target user is not in contact with the table surface and moves along an adjustment direction. Optionally, the computer-implemented method also includes adjusting the reference point along the adjustment direction of the movement of the target object (e.g., finger movement). Optionally, the gesture is recognized as an adjustment gesture when the movement of the target object (e.g., finger movement) is within a distance relative to the reference surface; the distance is less than a first threshold distance. Various appropriate threshold distances can be used as the first threshold distance, for example, 1mm, 2mm, 3mm, 4mm, 5mm, 6mm, 7mm, 8mm, 9mm, or 10mm.

[0054] In some embodiments, the computer-implemented method further comprises identifying an initiation gesture indicating the initiation of inputting information or providing a user command. When the initiation gesture is identified, the user is allowed to input information or provide a user command on the display panel. An example of an initiation gesture includes an L-shaped gesture.

[0055] Various appropriate initiation gestures may be used for various appropriate purposes. In some embodiments, the initiation gesture is a first type of initiation gesture that indicates that a trajectory image is to be displayed based on the user's object motion trajectory. For example, after recognizing the first type of initiation gesture, the computer-implemented method further includes determining whether the user's object motion trajectory is transformed into a display gesture; and when it is determined that the user's object motion trajectory is transformed into a display gesture, displaying a trajectory image in a target area based on the user's object motion trajectory relative to a reference surface. In one example, displaying the trajectory image includes displaying strokes of characters or letters based on the object motion trajectory.

[0056] In some embodiments, the initiation gesture is a second type of initiation gesture indicating initiation of providing a user-machine interaction instruction. For example, after identifying the second type of initiation gesture, the computer-implemented method further includes determining whether the user's object motion trajectory is transformed into a page-turning gesture; and when it is determined that the user's object motion trajectory is transformed into a page-turning gesture, changing the image displayed on the display panel to a different image (e.g., the next page).

[0057] In some embodiments, the computer-implemented method further comprises displaying a trajectory image in a target area based on an object motion trajectory of a user relative to a reference surface, such as a finger motion trajectory on a table. Optionally, displaying the trajectory image is performed after recognizing the initiation gesture. Optionally, displaying the trajectory image is performed after determining the target area. The user's object motion trajectory can be a motion trajectory of any suitable object, such as a user's finger (e.g., the user's index finger) or any suitable object held by the user, such as a stylus, pen, pencil, or laser pointer.

[0058] In some embodiments, the computer-implemented method further comprises executing a user command based on an object motion trajectory of the user relative to a reference surface, such as a finger motion trajectory on a table. Optionally, executing the user command is performed after identifying the initiating gesture. Optionally, executing the user command is performed after determining the target area.

[0059] In some embodiments, the computer-implemented method further comprises obtaining an image of the user's hand at one or more different angles. Optionally, the image of the user's hand at one or more different angles comprises an image of a top view of the user's hand and an image of a side view of the user's hand. The term "top view" generally refers to a view along a substantially vertical direction (e.g., substantially perpendicular to the direction of the desktop on which the hand is placed). The term side view generally refers to a view along a substantially horizontal direction (e.g., substantially parallel to the direction of the desktop on which the hand is placed). As used herein, the terms "substantially vertical", "substantially horizontal", "substantially vertical" and "substantially parallel" are not limited to the situations of "completely vertical", "completely horizontal", "completely vertical" and "completely parallel", but allow a certain degree of deviation. For example, the deviation may be a deviation of less than 40 degrees, such as less than 35 degrees, less than 30 degrees, less than 25 degrees, less than 20 degrees, less than 15 degrees, less than 10 degrees, less than 5 degrees, less than 1 degree or less than 0.5 degrees.

[0060] Figure 1 is a flowchart illustrating a computer-implemented method in some embodiments according to the present disclosure. Figure 1 In some embodiments, a computer-implemented method includes using one or more cameras (e.g., Figure 1 In some embodiments, the one or more cameras include: Figure 1 . In some embodiments, the first camera Camera1 and the second camera Camera2 are configured to capture images of the user's hand at two different angles. In one example, the first camera Camera1 is configured to capture a side view image of the user's hand. In another example, the second camera Camera2 is configured to capture a top view image of the user's hand.

[0061] Figure 2 A top-down image depicting a user's hand. Figure 3A side view image of a user's hand is depicted. In some embodiments, each user or each group of one or more users is assigned groups of cameras, including a first camera Camera1 and a second camera Camera2. The computer-implemented method can be used in various appropriate scenarios. In an exemplary scenario, the computer-implemented method is implemented in a conference room setting, where multiple users or groups of users occupy the conference room. The conference room is equipped with one or more groups of first cameras Camera1 and second cameras Camera2, for example, each seat in the conference room is equipped with groups of first cameras Camera1 and second cameras Camera2. In an alternative scenario, at least one user can participate in the meeting remotely, for example, at least one user can join a virtual meeting from their home office. The home office can be equipped with groups of first cameras Camera1 and second cameras Camera2. Various alternative embodiments can be practiced in the present disclosure.

[0062] In some embodiments, the computer-implemented method further comprises detecting and tracking the object motion trajectory of the user, and obtaining data of the object motion trajectory relative to the reference surface. In some embodiments, the computer-implemented method further comprises displaying a trajectory image in the target area based on the object motion trajectory of the user relative to the reference surface, such as the motion trajectory of a finger on a table. The object motion trajectory of the user can be the motion trajectory of any suitable object, such as a finger of the user (e.g., an index finger of the user) or any suitable object held by the user, such as a stylus, pen, pencil, or laser pointer.

[0063] In some embodiments, displaying a trajectory image in a target area based on a user's object motion trajectory relative to a reference surface includes determining a relative displacement between the target object and the reference surface on a plane parallel to the reference surface; and converting the relative displacement on the plane parallel to the reference surface into a trajectory image.

[0064] In some embodiments, the computer-implemented method further includes determining whether the user's object motion trajectory is transformed into a pause gesture. In some embodiments, the pause gesture is a gesture in which the target object (e.g., a finger) moves relative to the reference surface toward a pause direction. In one example, the pause direction is an upward vertical direction. Optionally, if the object making the user's object motion trajectory moves a distance exceeding a second threshold distance in the upward vertical direction, the gesture is identified as a pause gesture.

[0065] When it is determined that the pause gesture is not detected, the computer-implemented method includes continuing to detect and track the user's object motion trajectory; and obtaining data of the object motion trajectory. For example, the object continues to move the stroke of a character or letter or graphic.

[0066] When it is determined that the user's object motion trajectory is transformed into a pause gesture, the computer-implemented method includes pausing detection and tracking of the user's object motion trajectory; pausing acquisition of data of the object motion trajectory. In one example, the pause is performed when the user completes the movement of the stroke of a character or letter or graphic.

[0067] In some embodiments, after pausing the conversion of the object motion trajectory relative to the reference surface into a trajectory image, the computer-implemented method includes continuing to detect and track the object motion trajectory relative to the reference surface. When a pause gesture is not detected, the computer-implemented method includes converting the object motion trajectory relative to the reference surface into a trajectory image in real time. When a pause gesture is detected, the computer-implemented method includes pausing the conversion of the object motion trajectory relative to the reference surface into the trajectory image. Although the conversion of the object motion trajectory relative to the reference surface into a trajectory image is stopped, the computer-implemented method continues to detect and track the object motion trajectory relative to the reference surface. By continuing to detect and track the object motion trajectory relative to the reference surface, once a display gesture is detected, the computer-implemented method resumes converting the object motion trajectory relative to the reference surface into a trajectory image in real time. When resuming the conversion of the object motion trajectory relative to the reference surface into a trajectory image in real time, a reference point for starting to display the trajectory image is required. The inventors of the present disclosure have found that by continuing to detect and track the object motion trajectory relative to the reference surface while pausing the conversion of the object motion trajectory relative to the reference surface into a trajectory image in real time, a new trajectory image (e.g., a trajectory image of a new stroke of a character or letter) can be accurately displayed at an appropriate position relative to a previous trajectory image (e.g., a trajectory image of a previous stroke of a character or letter).

[0068] The above process can be repeated until all strokes are completed. The combination of the trajectory images of all strokes produces a combined image of a character or letter, sentence or paragraph, etc., in which all strokes are arranged in appropriate positions. An image of clear text can be obtained.

[0069] In some embodiments, a computer-implemented method includes capturing an image of a user's hand using a first camera Camera1 and a second camera Camera2; detecting and tracking the user's object motion trajectory, and acquiring data of the object motion trajectory. In one example, in a previous image frame, the data of the object motion trajectory includes a first series of positions F1 of a user's fingertip; in a current image frame, the data of the object motion trajectory includes a second series of positions F2 of the user's fingertip. In the second image frame, the user's fingertip has a y-axis coordinate y1 at time point t1, a y-axis coordinate y2 at time point t2, a y-axis coordinate y3 at time point t3, and a y-axis coordinate y4 at time point t4. Time points t1, t2, t3, and t4 are sequential time points, where t1 is a time point earlier than t4. A pause gesture is recognized when y1 is greater than each of y2, y3, and y4 by a difference value, and the difference value is greater than a second threshold distance θ. Various appropriate threshold distances may be used as the second threshold distance θ, for example, 1 mm, 2 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, or 10 mm. In another example, θ is 5 mm.

[0070] In some embodiments, the computer-implemented method further comprises displaying a trajectory image based on the object motion trajectory of the user relative to the reference surface. In one example, a pause gesture is recognized in the second image frame, and the computer-implemented method further comprises inputting the first series of positions F1 of the user's fingertip into a pre-trained neural network (e.g., a recurrent neural network) to recognize information input by the user (e.g., strokes of a character, strokes of a letter, strokes of a figure, etc.); and displaying it as a trajectory image.

[0071] In some embodiments, the computer-implemented method further comprises identifying a line break gesture. Upon identifying the line break gesture, the computer-implemented method further comprises starting a new line for inputting information or providing a user command. In one example, the line break gesture is a fist gesture.

[0072] In some embodiments, one or more cameras are rotatable cameras, for example, four-way rotatable cameras (e.g., up, down, left, right). Optionally, one or more cameras are zoomable cameras, for example, cameras with adjustable focal lengths. By having these features, gesture recognition and object motion trajectory detection can be performed with improved accuracy.

[0073] Figure 4 is a flowchart illustrating a computer-implemented method in some embodiments according to the present disclosure. Figure 4 In some embodiments, the computer-implemented method includes capturing an image of a user's hand. For example, Figure 1The first camera Camera1 depicted in FIG. 1 is used to capture an image of a user's hand. Once the image of the user's hand is obtained, in some embodiments, the computer-implemented method further includes detecting and tracking a gesture of the user. In some embodiments, detecting and tracking a gesture of the user includes recognizing a hand image from each of the images of the user's hand; extracting a plurality of features from the hand image; and recognizing a gesture based on the plurality of features extracted from the hand image. In one example, the plurality of features includes 21 features important for recognizing a gesture.

[0074] As described above, in some embodiments, a computer-implemented method includes identifying a target user, which is a user who wants to remotely input information or provide user commands on a display panel. In some embodiments, identifying the target user includes identifying a target gesture of the user. Therefore, after identifying a gesture based on multiple features extracted from a hand image, in some embodiments, the computer-implemented method also includes comparing the gesture with a target gesture. When it is determined that the gesture is a target gesture, the user is identified as a target user. Examples of target gestures include raising a hand and keeping the palm on the table for a fixed time. In a specific example, the target gesture is to keep the palm on the table for more than 1 second. The hand movement of the target user controls at least a portion of the display panel, for example, the target user is able to remotely input information or provide user commands on the display panel, while non-target users cannot do so.

[0075] As described above, in some embodiments, the computer-implemented method includes a target area on the display panel designated to a target user for inputting information or providing a user command. In one example, the display panel can be virtually divided into a plurality of candidate target areas, each candidate target area being designated to a corresponding candidate target user. The display panel is configured to allow a plurality of candidate target users to input information or provide a user command in a plurality of candidate target areas simultaneously. In one example, the display panel is virtually divided into eight candidate target areas designated to eight candidate target users. Therefore, the display panel allows a total of eight candidate target users to input information or provide a user command in a total of eight candidate target areas simultaneously.

[0076] In some embodiments, when a target user is identified, a target area is assigned to the target user. Figure 4 In some embodiments, the computer-implemented method further comprises activating the target area, for example, displaying the target area with a border such as a dashed line on the display panel. For example, an initial point for inputting information in the target area may be assigned by the display panel at the upper left corner of the target area. In one example, the initial point in the coordinate system of the target area has coordinates (10, 10).

[0077] Once the target area is assigned to the target user, in some embodiments, the computer-implemented method further comprises adjusting the target area to obtain an adjusted target area. Optionally, the target object comprises a palm of the target user; and adjusting the target area comprises adjusting the position of the palm of the user. For illustration, in one example, each image in the image of the user's hand has a size of W1×H1, and the display area of ​​the display panel has a size of W2×H2. In two adjacent corresponding images in the image of the user's hand, the center of the palm portion image in the first adjacent corresponding image in the image of the user's hand moves a distance Dx along the x-axis in the coordinate system of the image of the user's hand relative to the second adjacent corresponding image in the image of the user's hand, and moves a distance Dy along the y-axis in the coordinate system of the image of the user's hand. When the movement of the center of the palm portion image is mapped to the display area of ​​the display panel, the adjusted target area moves a distance (Dx×W2) / W1 along the x-axis in the coordinate system of the display area of ​​the display panel relative to the target area, and moves a distance (Dy×H2) / H1 along the y-axis in the coordinate system of the display area of ​​the display panel.

[0078] In some embodiments, the computer-implemented method further comprises determining whether the adjusted target area overlaps with any activated target area in the display panel. When it is determined that the adjusted target area overlaps with any activated target area in the display panel, the adjusted target area is not used to input information or provide user commands. Optionally, the computer-implemented method further comprises providing a user with a notification that the adjusted target area overlaps with any activated target area in the display panel. Optionally, the computer-implemented method further comprises repeating the step of adjusting the target area until an updated adjusted target area that does not overlap with any activated target area in the display panel is identified.

[0079] When it is determined that the adjusted target area does not overlap with any activated target area in the display panel, the computer-implemented method further includes replacing the target area with the adjusted target area for inputting information or providing a user command.

[0080] In some embodiments, the computer-implemented method further includes recognizing a confirmation gesture of the user. Upon recognizing the confirmation gesture, the computer-implemented method includes accepting the adjusted target area as the confirmation area for inputting information or providing user commands when the target area is subject to adjustment, or accepting the target area as the confirmation area for inputting information or providing user commands when the target area is not subject to adjustment. In one example, the confirmation gesture is a fist gesture.

[0081] Various suitable alternative embodiments may be practiced in the present disclosure. For example, various alternative methods may be used to determine a target area on a display panel designated for a target user to input information or provide a user command. In some embodiments, the display panel is configured to allow multiple candidate target users to input information or provide user commands in multiple candidate target areas. In one example, the display panel is virtually divided into four candidate target areas designated for four candidate target users. For example, the display panel may have four candidate input positions (e.g., four candidate cursor positions), including [0.25×W screen ,0.25×H screen ], [0.25×W screen ,0.75×H screen ]、[0.75×W screen ,0.25×H screen ] and [0.75×W screen ,0.75×H screen ]; among them, W screen represents the width of the display area of ​​the display panel; and H screen In some embodiments, the computer-implemented method includes identifying a corresponding selection gesture that matches one of a plurality of candidate selection gestures; and activating a corresponding candidate target area from a plurality of candidate target areas to be designated as a target area for a target user. Figure 8 A plurality of candidate selection gestures in accordance with some embodiments of the present disclosure are shown.

[0082] Various suitable alternative embodiments may be practiced in the present disclosure. Figure 6 is a flowchart illustrating a computer-implemented method in some embodiments according to the present disclosure. Figure 6 The computer-implemented method includes determining coordinates of a plurality of calibration points in a world coordinate system. Optionally, the plurality of calibration points are a plurality of points of a real-world object. In one example, the real-world object is a table, such as a table in a conference room; and the plurality of calibration points are a plurality of corner points of the table.

[0083] In some embodiments, determining the coordinates of the multiple calibration points in the world coordinate system includes capturing an image of the real-world object using a third camera, such as a monocular camera attached to the display panel; and converting the coordinates of the multiple calibration points in the image of the real-world object in the image coordinate system (e.g., a two-dimensional coordinate system) into coordinates of the multiple calibration points in the world coordinate system (e.g., a three-dimensional coordinate system) based on parameters of the third camera. Optionally, the parameters of the third camera include internal parameters and external parameters of the third camera. In one example, for the four corner points of a table, the coordinates of the multiple calibration points in the world coordinate system can be represented as (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4).

[0084] In some embodiments, the computer-implemented method further comprises establishing a spatial equation of the real world object based on the coordinates of the plurality of calibration points in the world coordinate system. In one example, any three of the coordinates of the plurality of calibration points in the world coordinate system may be used to establish the spatial equation.

[0085] In one example, (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3) can be used to establish a spatial equation. The spatial equation can be expressed as ax+by+cz+d=0;

[0086] Among them, a=((y2-y1)×(z3-z1))-((y3-y1)×(z2-z1));

[0087] b = ((z2-z1) × (x3-x1)) - ((z3-z1) × (x2-x1));

[0088] c = ((x2-x1) × (y3-y1)) - ((x3-x1) × (y2-y1)); and

[0089] d=-ax1-by1-(c×z1).

[0090] In some embodiments, the computer-implemented method further includes identifying a target user, the target user being a user who wants to remotely input information or provide a user command on the display panel. In some embodiments, identifying the target user includes identifying a target gesture of the user. Optionally, the computer-implemented method further includes comparing the gesture with a target gesture. When it is determined that the gesture is a target gesture, the user is identified as a target user. Examples of target gestures include raising a hand, and the palm of a hand staying on a table for a fixed time. In a specific example, the target gesture is to keep the palm of a hand on the table for more than 1 second.

[0091] In some embodiments, the computer-implemented method further includes determining a bounding box of the target user's hand in each of the images of the target user's hand. Figure 5 The bounding box of the target user's hand in the corresponding image in the image of the target user's hand is shown. The computer-implemented method also includes determining the coordinates of the midpoint of the bottom MPB in the image coordinate system; converting the coordinates of the midpoint of the bottom MPB to the hand position coordinates in the world coordinate system. In one example, the hand position coordinates in the world coordinate system are (x0, y0, z0).

[0092] In some embodiments, the computer-implemented method further includes determining the distance between the hand position coordinates in the world coordinate system and the surface of the real-world object (e.g., a desktop). Determine this distance.

[0093] Optionally, a, b, c and d are parameters of the spatial equations of the real-world object.

[0094] In some embodiments, the computer-implemented method further includes determining whether the distance is less than a third threshold distance; when determining that the distance is less than the third threshold distance, determining that the hand of the target user is on the surface of the real-world object (e.g., a desktop); and when determining that the hand of the target user is on the surface of the real-world object, mapping the hand position coordinates in the world coordinate system to the input position coordinates in the display panel coordinate system. Various suitable threshold distances can be used as the third threshold distance, for example, 1 mm, 2 mm, 3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, 9 mm, or 10 mm. In one example, the input position coordinates (x in the display panel coordinate system) screen ,y screen ) can be calculated according to the following formula:

[0095] as well as

[0096]

[0097] Among them, W screen represents the width of the display area of ​​the display panel; and H screen Indicates the height of the display area of ​​the display panel.

[0098] In some embodiments, the input position coordinates in the display panel coordinate system are the coordinates of the cursor in the display panel coordinate system. In some embodiments, the input position coordinates in the display panel coordinate system are the coordinates of a reference point at which information input by a target user or a user command input by a target user begins to be displayed.

[0099] In some embodiments, the computer-implemented method further comprises adjusting the input position coordinates by adjusting the hand position coordinates in the world coordinate system. In one example, adjusting the hand position coordinates in the world coordinate system is performed by moving the target user's hand.

[0100] In some embodiments, the computer-implemented method further comprises identifying an initiation gesture indicating the initiation of inputting information or providing a user command. When the initiation gesture is identified, the user is allowed to input information or provide a user command on the display panel. An example of an initiation gesture includes an L-shaped gesture.

[0101] As described above, based on the internal parameters and external parameters of the third camera, the coordinates of multiple calibration points in the image of the real world object in the image coordinate system can be converted into the coordinates of multiple calibration points in the world coordinate system. Various appropriate methods can be used to obtain the internal parameters and external parameters of the third camera. Figure 7 The process of obtaining the internal and external parameters of the camera is shown. Figure 7 , [X, Y, Z, 1] represents the coordinates in the world coordinate system, and [x, y, 1] represents the coordinates in the display panel coordinate system. The internal parameters are determined by the intrinsic parameters of the camera. Figure 7 The internal parameter matrix shown is represented as:

[0102]

[0103] In this internal parameter matrix,

[0104] Where f is the focal length of the camera, d x Represents the width of the photosensitive pixel along the x-axis of the camera coordinate system, d y It represents the width of the photosensitive pixel along the y-axis of the camera coordinate system, and (u0, v0) represents the coordinates of the center point of the photosensitive plate in the camera coordinate system.

[0105] The extrinsic parameters are determined at least in part by the relative position between the world coordinate system and the camera coordinate system. Figure 7 The external parameter matrix shown is represented as:

[0106]

[0107] Where R represents the rotation matrix and T represents the translation vector.

[0108] As used herein, a world coordinate system refers to a user-defined three-dimensional world coordinate system that takes a specific point as the origin and is introduced to describe the position of a target object in the real world. As used herein, a camera coordinate system refers to a coordinate system established with a camera as the origin. It is defined as the connection between the world coordinate system and the display panel coordinate system in order to describe the position of an object from the perspective of the camera. The origin of the camera coordinate system is different from the origin of the world coordinate system.

[0109] Various suitable calibration methods can be used to obtain the intrinsic and extrinsic parameters of the camera. Examples of suitable calibration methods include the Zhang Zhengyou calibration method and the chessboard calibration method.

[0110] As previously described, the computer-implemented method discussed in the present disclosure may use a monocular camera to determine the coordinates of multiple calibration points in a world coordinate system. In an alternative embodiment, the computer-implemented method discussed in the present disclosure may use a binocular camera to determine the coordinates of the multiple calibration points in a world coordinate system. In some embodiments, the computer-implemented method includes determining the internal parameters and external parameters of a left eye camera of a binocular camera, and determining the internal parameters and external parameters of a right eye camera of a binocular camera. Once the internal parameters and external parameters of the binocular camera are determined, the computer-implemented method also includes determining the coordinates of multiple calibration points in a world coordinate system; establishing a spatial equation of the real-world object based on the coordinates of the multiple calibration points in the world coordinate system; identifying a target user by identifying the user's target gesture; determining the hand position coordinates in the world coordinate system; and mapping the hand position coordinates in the world coordinate system to the input position coordinates in the display panel coordinate system, such as Figure 6 and related texts discussed in the text.

[0111] Various suitable alternative embodiments may be practiced in the present disclosure. Fig. 9 is a flowchart illustrating a computer-implemented method in some embodiments according to the present disclosure. Fig. 9 In some embodiments, a computer-implemented method includes using one or more cameras (e.g., Fig. 9 ) to capture an image of the user's hand, and using one or more cameras (e.g., Fig. 9The first camera Camera1 and the second camera Camera2 are used to capture the facial image of the user. The first camera Camera1 and the second camera Camera2 can be attached to the display panel. Each first image captured by the first camera Camera1 is divided into a plurality of first sub-areas, each containing a hand image of a plurality of users. Each second image captured by the second camera Camera2 is divided into a plurality of second sub-areas, each containing a facial image of a plurality of users. Each first sub-area in the plurality of first sub-areas and each second sub-area in the plurality of second sub-areas are identified as containing a hand image and a facial image of the same user, respectively. By obtaining the hand images and the facial images of the plurality of users, a correlation between the hand images and the facial images of the plurality of users can be established. Based on the correlation, when a hand or an object held by a hand is identified as a target object, each user can be easily identified.

[0112] In some embodiments, the computer-implemented method includes identifying a target user, the target user being a user who wants to remotely input information or provide a user command on a display panel. In some embodiments, identifying the target user includes identifying a target gesture of the user. When it is determined that the user's gesture is a target gesture, the user is identified as a target user. Examples of target gestures include raising a hand, and holding a palm on a table for a fixed time. In a specific example, the target gesture is holding a palm on a table for more than 1 second.

[0113] When the target user is determined, in some embodiments, the computer-implemented method further includes determining a gaze point on the display panel; and determining the target area using the gaze point. Optionally, the computer-implemented method includes determining a gaze direction of the target user, and determining a point on the display panel where the gaze direction intersects the plane of the display panel, thereby determining the gaze point. Various suitable methods can be used to determine the gaze direction. In a specific example, the gaze direction is determined using a GazeOnce algorithm.

[0114] In some embodiments, the gaze point may be used as a reference point. For example, the target area may be an area having the gaze point as a center point.

[0115] In some embodiments, the gaze point can be used as a reference point. In one example, the reference point is the point at which information input by the target user or a user command input by the target user is started to be displayed. Optionally, the reference point is the point at which the trajectory image is started to be displayed. Optionally, the reference point is the point at which the strokes of a character (e.g., a Chinese character) or a letter (e.g., a, b, c, d) are started to be displayed.

[0116] In some embodiments, the computer-implemented method further includes determining whether K gaze points of the target user determined from the K image frames of the target user are located at substantially the same position, where K is a positive integer. In one example, K=5. As used herein, the term "substantially the same position" refers to K gaze points within a circle whose radius is less than a threshold value (e.g., less than 10 cm, less than 9 cm, less than 8 cm, less than 7 cm, less than 6 cm, less than 5 cm, less than 4 cm, less than 3 cm, less than 2 cm, or less than 1 cm).

[0117] In some embodiments, when it is determined that the K gaze points of the target user determined from the K image frames of the target user are located at substantially the same position, the computer-implemented method further includes determining a target area using the gaze points as reference points. For example, the target area may be an area having the gaze points as a center point. In a specific example, the target area is an area having a size of 50 cm×50 cm and having the gaze points as its center point.

[0118] In some embodiments, the computer-implemented method further comprises recognizing a confirmation gesture of the user. When recognizing the confirmation gesture, the computer-implemented method comprises accepting the target area as a confirmation area for inputting information or providing a user command. In one example, the confirmation gesture is an OK gesture.

[0119] The methods described in this disclosure may also be applied to providing user commands. Fig.10 is a flowchart illustrating a computer-implemented method in some embodiments according to the present disclosure. Fig.10 In some embodiments, a computer-implemented method includes using one or more cameras (e.g., Fig.10 ) to capture an image of the user's hand, and using one or more cameras (e.g., Fig.10 The first camera Camera1 and the second camera Camera2 are used to capture the facial image of the user. The first camera Camera1 and the second camera Camera2 can be attached to the display panel. Each first image captured by the first camera Camera1 is divided into a plurality of first sub-areas, each including a plurality of user's hand images. Each second image captured by the second camera Camera2 is divided into a plurality of second sub-areas, each including a plurality of user's facial images. Each first sub-area in the plurality of first sub-areas and each second sub-area in the plurality of second sub-areas are identified as respectively including a hand image and a facial image of the same user.

[0120] In some embodiments, the computer-implemented method includes identifying a target user, the target user being a user who wants to remotely input information or provide a user command on a display panel. In some embodiments, identifying the target user includes identifying a target gesture of the user. When it is determined that the user's gesture is a target gesture, the user is identified as a target user. Examples of target gestures include raising a hand, and holding a palm on a table for a fixed time. In a specific example, the target gesture is holding a palm on a table for more than 1 second.

[0121] When determining the target user, in some embodiments, the computer-implemented method further includes determining a gaze point on the display panel; and using the gaze point as a reference point to determine the command area. Optionally, the computer-implemented method includes determining a gaze direction of the target user, and determining a point on the display panel where the gaze direction intersects the plane of the display panel, thereby determining the gaze point. Various suitable methods can be used to determine the gaze direction. In a specific example, the gaze direction is determined using a GazeOnce algorithm.

[0122] In some embodiments, in some embodiments, the computer-implemented method further comprises determining whether K gaze points of the target user determined from K image frames of the target user are located at substantially the same position, K being a positive integer. In one example, K=5.

[0123] In some embodiments, when it is determined that the K gaze points of the target user determined from the K image frames of the target user are located at substantially the same position, the computer-implemented method further comprises determining a command area using the gaze points as reference points. For example, the command area may be an area having the gaze points as a center point. In a specific example, the target area is an area having a size of 50 cm×50 cm and having the gaze points as its center point. Optionally, the computer-implemented method further comprises highlighting the command area.

[0124] In some embodiments, the computer-implemented method further includes recognizing a command gesture of a user. Upon recognizing the command gesture, the computer-implemented method includes executing a user command based on the command gesture. In one example, the first command gesture is a gesture of transitioning from a palm to a fist, and the computer-implemented method includes executing a first user command to shrink the command area. In another example, the second command gesture is a gesture of transitioning from a fist to a palm, and the computer-implemented method includes executing a second user command to enlarge the command area. In another example, the third command gesture is raising and lowering the target user's finger twice relative to a reference surface, and the computer-implemented method includes executing a third user command of double-clicking in the command area.

[0125] The inventors of the present disclosure have discovered a novel and unique method of providing user-machine interaction instructions. Figures 1 to 10 In some embodiments, the computer-implemented method is based on a relative displacement between a target object and a reference surface (eg, a desktop). Fig.11 is a flowchart illustrating a computer-implemented method in some embodiments according to the present disclosure. Fig.11 In some embodiments, a computer-implemented method includes determining a reference surface; determining a target object of a target user; determining a relative displacement between the target object and the reference surface; and determining a user-machine interaction instruction based on the relative displacement. The reference surface can be a surface of any suitable real-world object. Optionally, the reference surface is a substantially horizontal surface. As used herein, the term "substantially horizontal" includes a completely horizontal surface, as well as a surface extending generally in a horizontal direction with a tolerance of -20 degrees to 20 degrees (e.g., -20 degrees to -15 degrees, -15 degrees to -10 degrees, -10 degrees to -5 degrees, -5 degrees to 0 degrees, 0 degrees to 5 degrees, 5 degrees to 10 degrees, 10 degrees to 15 degrees, or 15 degrees to 20 degrees). In one example, the reference surface is a desktop, such as a desktop in a conference room.

[0126] In some embodiments, determining the reference surface includes determining the coordinates of a plurality of calibration points in a three-dimensional coordinate system (e.g., a world coordinate system or a camera coordinate system); and deriving the reference surface based on the plurality of calibration points. In some embodiments, determining the coordinates of the plurality of calibration points in the three-dimensional coordinate system includes capturing an image of the real-world object using a third camera, such as a monocular camera attached to the display panel; and converting the coordinates of the plurality of calibration points in the image of the real-world object in the image coordinate system (e.g., a two-dimensional coordinate system) into the coordinates of the plurality of calibration points in the three-dimensional coordinate system (e.g., a three-dimensional coordinate system) based on the parameters of the third camera. Optionally, the parameters of the third camera include internal parameters and external parameters of the third camera. In one example, for the four corner points of a table, the coordinates of the plurality of calibration points in the three-dimensional coordinate system may be represented as (x1, y1, z1), (x2, y2, z2), (x3, y3, z3) and (x4, y4, z4). In another example, the reference surface is a surface having (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4) as four corner points.

[0127] In some embodiments, the computer-implemented method further includes identifying a user who desires to remotely provide user-machine interaction instructions on the display panel as a target user.

[0128] In some embodiments, the computer-implemented method further includes identifying an object making the object motion trajectory as a target object.

[0129] In some embodiments, determining the user-machine interaction instruction based on the relative displacement includes displaying a trajectory image in the target area based on the object motion trajectory of the user relative to the reference surface. In some embodiments, determining the user-machine interaction instruction based on the relative displacement includes executing a user command based on the object motion trajectory of the user relative to the reference surface.

[0130] In some embodiments, the user-machine interaction instruction includes displaying a trajectory image. In some embodiments, the computer-implemented method further includes detecting and tracking an object motion trajectory relative to a reference surface; displaying a trajectory image in a target area based on the object motion trajectory of the user relative to the reference surface; determining whether the object motion trajectory of the user is converted into a pause gesture; and pausing the display of the trajectory image when it is determined that the object motion trajectory of the user is converted into a pause gesture. Optionally, the pause gesture is a gesture in which the target object moves in a pause direction relative to the reference surface. Optionally, if the object making the object motion trajectory moves a distance exceeding a second threshold distance along the pause direction relative to the reference surface, the gesture is recognized as a pause gesture.

[0131] In some embodiments, the user-machine interaction instruction includes displaying a trajectory image. In some embodiments, the computer-implemented method further includes determining whether the user's object motion trajectory is converted into a display gesture; and when it is determined that the user's object motion trajectory is converted into a display gesture, based on the user's object motion trajectory relative to a reference surface, displaying the trajectory image in the target area. In one example, when it is determined that the user's object motion trajectory is converted into a display gesture, the display panel begins to display the strokes of the character or letter or graphic based on the user's object motion trajectory relative to the reference surface. In another example, the display gesture is detected after a pause gesture is detected. When it is determined that a display gesture is detected after a pause gesture is detected, the display panel resumes displaying new strokes of the character or letter or graphic based on the user's object motion trajectory relative to the reference surface. Optionally, the display gesture is a gesture that brings the target user's finger into contact with the reference surface.

[0132] When it is determined that the display gesture is not detected, the computer-implemented method includes continuing to suspend detection and tracking of the user's object motion trajectory; and suspending acquisition of data of the object motion trajectory.

[0133] In some embodiments, when a display gesture is not detected, the computer-implemented method continues to detect and track the object motion trajectory relative to the reference surface without converting the object motion trajectory relative to the reference surface into a trajectory image in real time. Although the conversion of the object motion trajectory relative to the reference surface into a trajectory image is stopped, the computer-implemented method continues to detect and track the object motion trajectory relative to the reference surface. By continuing to detect and track the object motion trajectory relative to the reference surface, once a display gesture is detected, the computer-implemented method resumes converting the object motion trajectory relative to the reference surface into a trajectory image in real time. When resuming the conversion of the object motion trajectory relative to the reference surface into a trajectory image in real time, a reference point for starting to display the trajectory image is required. The inventors of the present disclosure have found that by continuing to detect and track the object motion trajectory relative to the reference surface while pausing the conversion of the object motion trajectory relative to the reference surface into a trajectory image, a new trajectory image (e.g., a trajectory image of a new stroke of a character or letter) can be accurately displayed at an appropriate position relative to a previous trajectory image (e.g., a trajectory image of a previous stroke of a character or letter).

[0134] The above process can be repeated until all strokes are completed. The combination of the trajectory images of all strokes produces a combined image of a character or letter, sentence or paragraph, etc., in which all strokes are arranged in appropriate positions. An image of clear text can be obtained.

[0135] In some embodiments, the user-machine interaction instruction includes identifying an initiation gesture. In some embodiments, the computer-implemented method further includes identifying an initiation gesture indicating initiation of providing the user-machine interaction instruction. Optionally, providing the user-machine interaction instruction is performed after identifying the initiation gesture.

[0136] In some embodiments, the user-machine interaction instruction includes executing a user command. In some embodiments, the computer-implemented method further includes detecting and tracking an object motion trajectory of a user; and executing the user command based on the object motion trajectory of the user relative to a reference surface.

[0137] In some embodiments, the user-machine interaction instruction includes adjusting a reference point (e.g., a reference point for starting display of a trajectory image). Optionally, adjusting the reference point includes recognizing an adjustment gesture of a target user. Optionally, the adjustment gesture is a gesture in which a target object (e.g., a finger) of the target user is not in contact with a reference surface and moves along an adjustment direction. Optionally, the computer-implemented method further includes adjusting the reference point along the adjustment direction of the movement of the target object (e.g., finger movement). Optionally, the gesture is recognized as an adjustment gesture when the movement of the target object (e.g., finger movement) is within a distance relative to the reference surface; the distance is less than a first threshold distance. In a specific example, the adjustment gesture is a gesture in which the target object (e.g., finger) moves less than 5 cm on a plane parallel to the reference surface.

[0138] In some embodiments, the computer-implemented method further includes determining a target area on the display panel designated for a target user for providing user-machine interaction instructions.

[0139] In another aspect, the present disclosure provides a user input system. In some embodiments, the user input system includes one or more cameras; a display panel; a memory; and one or more processors. In some embodiments, the memory and the one or more processors are connected to each other; and the memory stores computer executable instructions to control the one or more processors to: determine a reference surface; determine a target object of a target user; determine a relative displacement between the target object and the reference surface; and determine a user-machine interaction instruction based on the relative displacement.

[0140] Fig.12 is a schematic diagram showing a user input system according to some embodiments of the present disclosure. Fig.12 A user input system for implementing the method according to the present disclosure is shown. Fig.12 , the user input system 1000 may include any appropriate type of interactive whiteboard, such as a plasma interactive whiteboard, a liquid crystal display (LCD) interactive whiteboard, a touch screen interactive whiteboard, a projection interactive whiteboard, a non-intelligent interactive whiteboard, an intelligent interactive whiteboard, etc. The user input system 1000 may also include other computing systems, such as a personal computer (PC), a tablet computer or a mobile computer or a smart phone, etc. In addition, the user input system 1000 may be any appropriate content presentation device capable of presenting any appropriate content. The user may interact with the user input system 1000 to perform other activities of interest.

[0141] like Fig.12As shown, user input system 1000 may include processor 1002, storage medium 1004, display 1006, communication module 1008, database 1010, peripheral device 1012, and camera 1014. Certain devices may be omitted and other devices may be included to better describe relevant embodiments.

[0142] The processor 1002 may include any appropriate one or more processors. The processor 1002 may include multiple cores for multithreading or parallel processing. The processor 1002 may execute a sequence of computer program instructions to perform various processes. The storage medium 1004 may include a memory module (e.g., ROM, RAM, flash memory module) and a large capacity memory (e.g., CD-ROM and hard disk), etc. The storage medium 1004 may store a computer program to implement various processes when the computer program is executed by the processor 1002. For example, the storage medium 1004 may store a computer program to implement various algorithms when the computer program is executed by the processor 1002.

[0143] Further, the communication module 1008 may include some network interface devices for establishing a connection through a communication network (e.g., a TV cable network, a wireless network, the Internet). The database 1010 may include one or more databases for storing some data and performing some operations on some stored data, such as database search.

[0144] Display 1006 can provide information to a user. Display 1006 can include any suitable type of computer display device or electronic device display, such as an LCD or OLED based device. Peripheral devices 1012 can include various sensors or other I / O devices, such as a keyboard or mouse.

[0145] In some embodiments, the memory also stores computer executable instructions to control one or more processors to: determine the coordinates of multiple calibration points in a three-dimensional coordinate system (e.g., a world coordinate system); and derive the reference surface based on the multiple calibration points, thereby determining the reference surface.

[0146] In some embodiments, the memory also stores computer executable instructions to control one or more processors to: capture an image of a real-world object using a camera; and based on parameters of the camera, convert the coordinates of the multiple calibration points in the image of the real-world object in the image coordinate system into the coordinates of the multiple calibration points in the three-dimensional coordinate system, thereby determining the coordinates of the multiple calibration points in the three-dimensional coordinate system.

[0147] In some embodiments, the memory further stores computer executable instructions to control the one or more processors to: identify a user who wants to remotely provide user-machine interaction instructions on the display panel as a target user.

[0148] In some embodiments, the memory further stores computer executable instructions to control the one or more processors to: identify an object making an object motion trajectory as a target object.

[0149] In some embodiments, the user-machine interaction instructions include displaying a trajectory image. In some embodiments, the memory also stores computer executable instructions to control one or more processors to: detect and track the user's object motion trajectory; based on the object motion trajectory of the user relative to the reference surface, display the trajectory image in the target area; determine whether the object motion trajectory of the user is transformed into a pause gesture; and when it is determined that the user's object motion trajectory is transformed into a pause gesture, pause the display of the trajectory image. Optionally, the pause gesture is a gesture in which the target object moves toward the pause direction relative to the reference surface. Optionally, if the object making the object motion trajectory moves a distance exceeding a second threshold distance along the pause direction relative to the reference surface, the gesture is recognized as a pause gesture. In one example, the pause direction is an upward vertical direction. Optionally, if the object making the object motion trajectory moves a distance exceeding a second threshold distance along the upward vertical direction, the gesture is recognized as a pause gesture.

[0150] In some embodiments, the user-machine interaction instructions include displaying a trajectory image. In some embodiments, the memory further stores computer executable instructions to control one or more processors to: determine whether the user's object motion trajectory is converted into a display gesture; and when it is determined that the user's object motion trajectory is converted into a display gesture, display a trajectory image in a target area based on the user's object motion trajectory relative to a reference surface. Optionally, the display gesture is a gesture that brings the target user's finger into contact with the reference surface.

[0151] In some embodiments, the user-machine interaction instructions include identifying an initiation gesture. In some embodiments, the memory further stores computer executable instructions to control one or more processors to: identify an initiation gesture indicating initiation of providing the user-machine interaction instructions. Optionally, the providing the user-machine interaction instructions is performed after the initiation gesture is identified.

[0152] In some embodiments, the user-machine interaction instructions include executing user commands. In some embodiments, the memory further stores computer executable instructions to control one or more processors to: detect and track the object motion trajectory of the user relative to the reference surface; and execute the user command based on the object motion trajectory of the user relative to the reference surface.

[0153] In some embodiments, the memory further stores computer executable instructions to control one or more processors to: identify a target user, the target user being a user who wants to remotely input information or provide a user command on the display panel; and determine a target area on the display panel designated for the target user to input information or provide a user command. The display panel is configured to display a trajectory image in the target area or execute a user command based on the object motion trajectory of the user relative to the reference surface.

[0154] In some embodiments, the memory further stores computer executable instructions to control one or more processors to: determine a gaze point on the display panel; and determine a target area using the gaze point. Optionally, the reference point is a point at which information input by the target user or a user command input by the target user is started to be displayed.

[0155] In some embodiments, the user-machine interaction instruction includes adjusting a reference point (eg, a reference point for starting to display the trajectory image). Optionally, the memory further stores computer executable instructions to control one or more processors to: recognize an adjustment gesture of the target user.

[0156] In some embodiments, the adjustment gesture is a gesture in which the target object (e.g., finger) of the target user is not in contact with the desktop and moves along the adjustment direction. In some embodiments, the memory also stores computer executable instructions to control one or more processors to: adjust the reference point along the adjustment direction of the movement of the target object (e.g., finger movement). Optionally, when the movement of the target object (e.g., finger movement) is within a distance relative to the reference surface, the gesture is recognized as an adjustment gesture; the distance is less than a first threshold distance.

[0157] In some embodiments, the memory further stores computer executable instructions to control the one or more processors to: determine whether K gaze points of the target user determined respectively from the K image frames of the target user are located at substantially the same position.

[0158] In some embodiments, the memory also stores computer executable instructions to control one or more processors to: determine the coordinates of multiple calibration points in a three-dimensional coordinate system (e.g., a world coordinate system or a camera coordinate system); establish the spatial equation of the real-world object based on the coordinates of the multiple calibration points in the three-dimensional coordinate system; determine the position coordinates of the target object in the three-dimensional coordinate system; and map the position coordinates of the target object in the three-dimensional coordinate system to the input position coordinates in the display panel coordinate system, thereby determining the target area.

[0159] In some embodiments, the display panel is virtually divided into a plurality of candidate target areas. In some embodiments, the memory further stores computer executable instructions to control one or more processors to: designate at least one candidate target area of ​​the plurality of candidate target areas as the target area for the target user.

[0160] In some embodiments, the memory further stores computer executable instructions to control one or more processors to: activate a target area. Optionally, the memory further stores computer executable instructions to control one or more processors to: identify a corresponding selection gesture that matches one of a plurality of candidate selection gestures; and activate a corresponding candidate target area in the plurality of candidate target areas to designate it as the target area for the target user.

[0161] In some embodiments, the memory also stores computer executable instructions to control one or more processors to: detect and track the object motion trajectory of the user; determine whether the object motion trajectory of the user turns into a pause gesture; and pause displaying the trajectory image when it is determined that the object motion trajectory of the user turns into a pause gesture.

[0162] In some embodiments, the memory also stores computer executable instructions to control one or more processors to: convert the object motion trajectory relative to a reference surface into a trajectory image in real time; pause converting the object motion trajectory relative to the reference surface into the trajectory image when a pause gesture is detected; and continue detecting and tracking the object motion trajectory relative to the reference surface after pausing converting the object motion trajectory relative to the reference surface into the trajectory image.

[0163] In some embodiments, the pause gesture is a gesture in which the target object moves toward the pause direction relative to the reference surface. Optionally, if the object making the user's object motion trajectory moves along the pause direction for a distance exceeding a second threshold distance, the gesture is recognized as a pause gesture.

[0164] In some embodiments, the user-machine interaction instructions include identifying an initiation gesture. In some embodiments, the memory further stores computer executable instructions to control one or more processors to: identify an initiation gesture indicating an initiation of inputting information or providing a user command. Optionally, after identifying the initiation gesture, displaying the trajectory image in the target area or executing the user command is performed.

[0165] In some embodiments, the user-machine interaction instruction further comprises adjusting the target area to obtain an adjusted target area. Optionally, the target object comprises a palm of the target user; and the target area is adjusted in response to adjusting the position of the palm of the user.

[0166] In some embodiments, the memory further stores computer executable instructions to control one or more processors to: determine whether the adjusted target area overlaps with any of the activated target areas in the display panel. Optionally, when it is determined that the adjusted target area overlaps with any of the activated target areas in the display panel, the memory further stores computer executable instructions to control one or more processors to: cause the display panel to provide the user with a notification that the adjusted target area overlaps with any of the activated target areas in the display panel. Optionally, when it is determined that the adjusted target area does not overlap with any of the activated target areas in the display panel, the memory further stores computer executable instructions to control one or more processors to: replace the target area with the adjusted target area for inputting information or providing user commands.

[0167] In some embodiments, the memory also stores computer executable instructions to control one or more processors to: recognize a user's confirmation gesture; and when a confirmation gesture is recognized, accept the adjusted target area when the target area is subject to adjustment, or accept the target area as a confirmation area for inputting information or providing user commands when the target area is not subject to adjustment.

[0168] In some embodiments, the memory further stores computer executable instructions to control one or more processors to: detect and track a user's gesture; and compare the user's gesture with a target gesture. Optionally, when it is determined that the user's gesture is a target gesture, the user is identified as a target user.

[0169] In another aspect, the present disclosure provides a computer program product including a non-transitory tangible computer-readable medium having computer-readable instructions thereon. In some embodiments, the computer-readable instructions are executable by one or more processors to cause the one or more processors to perform: determining a reference surface; determining a target object of a target user; determining a relative displacement between the target object and the reference surface; and determining a user-machine interaction instruction based on the relative displacement.

[0170] In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: determining the coordinates of multiple calibration points in a three-dimensional coordinate system (e.g., a world coordinate system or a camera coordinate system); and deriving the reference surface based on the multiple calibration points, thereby determining the reference surface.

[0171] In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: capturing an image of a real-world object using a camera; and based on parameters of the camera, converting the coordinates of the multiple calibration points in the image of the real-world object in the image coordinate system into the coordinates of the multiple calibration points in the three-dimensional coordinate system, thereby determining the coordinates of the multiple calibration points in the three-dimensional coordinate system.

[0172] In some embodiments, the computer readable instructions are executable by one or more processors to cause the one or more processors to further perform: identifying a user who wants to remotely provide user-machine interaction instructions on the display panel as a target user.

[0173] In some embodiments, the computer readable instructions may be executed by one or more processors to cause the one or more processors to further perform: identifying an object making an object motion trajectory as a target object.

[0174] In some embodiments, the user-machine interaction instructions include displaying a trajectory image. In some embodiments, the computer-readable instructions are executable by one or more processors to cause the one or more processors to further perform: detecting and tracking the object motion trajectory of the user; displaying the trajectory image in a target area based on the object motion trajectory of the user relative to the reference surface; determining whether the object motion trajectory of the user is transformed into a pause gesture; and pausing the display of the trajectory image when it is determined that the object motion trajectory of the user is transformed into a pause gesture. Optionally, the pause gesture is a gesture in which the target object moves toward a pause direction relative to the reference surface. Optionally, if the object making the object motion trajectory moves a distance exceeding a second threshold distance along the pause direction relative to the reference surface, the gesture is recognized as a pause gesture.

[0175] In some embodiments, the user-machine interaction instructions include displaying a trajectory image. In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: determining whether the user's object motion trajectory is converted into a display gesture; and when it is determined that the user's object motion trajectory is converted into a display gesture, displaying a trajectory image in a target area based on the user's object motion trajectory relative to a reference surface. Optionally, the display gesture is a gesture that causes the target user's finger to contact the reference surface.

[0176] In some embodiments, the user-machine interaction instructions include identifying an initiation gesture. In some embodiments, the computer-readable instructions are executable by one or more processors to cause the one or more processors to further perform: identifying an initiation gesture indicating initiation of providing the user-machine interaction instructions. Optionally, performing the providing the user-machine interaction instructions after identifying the initiation gesture.

[0177] In some embodiments, the user-machine interaction instructions include executing user commands. In some embodiments, the computer readable instructions are executable by one or more processors to cause the one or more processors to further perform: detecting and tracking the object motion trajectory of the user; and executing the user command based on the object motion trajectory of the user relative to the reference surface.

[0178] In some embodiments, the computer readable instructions can be executed by one or more processors to cause the one or more processors to further perform: determining a gaze point on the display panel; and determining a target area using the gaze point. Optionally, the reference point is a point at which the information input by the target user or the user command input by the target user begins to be displayed.

[0179] In some embodiments, the user-machine interaction instructions include adjusting a reference point. In some embodiments, the computer-readable instructions are executable by one or more processors to cause the one or more processors to further perform: adjusting the reference point. Optionally, the computer-readable instructions are executable by one or more processors to cause the one or more processors to further perform: identifying an adjustment gesture of the target user.

[0180] In some embodiments, the adjustment gesture is a gesture in which the target object (e.g., finger) of the target user is not in contact with the desktop and moves along the adjustment direction. In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: adjusting the reference point along the adjustment direction of the movement of the target object (e.g., finger movement). Optionally, the gesture is recognized as an adjustment gesture when the movement of the target object (e.g., finger movement) is within a distance relative to the reference surface; the distance is less than a first threshold distance. In a specific example, the adjustment gesture is a gesture in which the target object (e.g., finger) moves less than 5 cm on a plane parallel to the reference surface.

[0181] In some embodiments, the computer readable instructions may be executed by one or more processors to cause the one or more processors to further perform: determining whether K gaze points of the target user determined respectively from K image frames of the target user are located at substantially the same position.

[0182] In some embodiments, computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: determining the coordinates of multiple calibration points in a three-dimensional coordinate system (e.g., a world coordinate system or a camera coordinate system); establishing a spatial equation of the real-world object based on the coordinates of the multiple calibration points in the three-dimensional coordinate system; determining the position coordinates of the target object in the three-dimensional coordinate system; and mapping the position coordinates of the target object in the three-dimensional coordinate system to the input position coordinates in the display panel coordinate system, thereby determining the target area.

[0183] In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further execute: virtually dividing the display panel into a plurality of candidate target areas. In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further execute: designating at least one candidate target area of ​​the plurality of candidate target areas as the target area for the target user.

[0184] In some embodiments, the computer-readable instructions are executable by one or more processors to cause the one or more processors to further perform: activating a target area. In some embodiments, the computer-readable instructions are executable by one or more processors to cause the one or more processors to further perform: identifying a corresponding selection gesture that matches one of a plurality of candidate selection gestures; and activating a corresponding candidate target area of ​​the plurality of candidate target areas to designate it as the target area for the target user.

[0185] In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: detecting and tracking the object motion trajectory of the user; determining whether the object motion trajectory of the user has turned into a pause gesture; and pausing display of the trajectory image when it is determined that the object motion trajectory of the user has turned into a pause gesture.

[0186] In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: converting the object motion trajectory relative to a reference surface into a trajectory image in real time; pausing the conversion of the object motion trajectory relative to the reference surface into the trajectory image when a pause gesture is detected; and continuing to detect and track the object motion trajectory relative to the reference surface after pausing the conversion of the object motion trajectory relative to the reference surface into the trajectory image.

[0187] In some embodiments, the pause gesture is a gesture in which the target object moves relative to the reference surface toward a pause direction. Optionally, if the object making the object motion trajectory moves along the pause direction for a distance exceeding a second threshold distance, the gesture is recognized as a pause gesture.

[0188] In some embodiments, the user-machine interaction instructions include identifying an initiation gesture. In some embodiments, the computer-readable instructions are executable by one or more processors to cause the one or more processors to further perform: identifying an initiation gesture indicating an initiation of inputting information or providing a user command. Optionally, after identifying the initiation gesture, performing displaying the trajectory image in the target area or executing the user command.

[0189] In some embodiments, the user-machine interaction instructions further include adjusting the target area. In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: adjusting the target area to obtain an adjusted target area. Optionally, the target object includes a palm of the target user; adjusting the target area is performed in response to adjusting the position of the palm of the user.

[0190] In some embodiments, the computer readable instructions may be executed by one or more processors to cause the one or more processors to further perform: determining whether the adjusted target area overlaps with any of the activated target areas in the display panel.

[0191] In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: when it is determined that the adjusted target area overlaps with any of the activated target areas in the display panel, providing a notification to the user that the adjusted target area overlaps with any of the activated target areas in the display panel.

[0192] In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: when it is determined that the adjusted target area does not overlap with any of the activated target areas in the display panel, replacing the target area with the adjusted target area for inputting information or providing user commands.

[0193] In some embodiments, the computer-readable instructions may be executed by one or more processors to cause the one or more processors to further perform: identifying a user's confirmation gesture; and when the confirmation gesture is identified, accepting the adjusted target area as a confirmation area for inputting information or providing user commands when the target area is subject to adjustment, or accepting the target area as a confirmation area for inputting information or providing user commands when the target area is not subject to adjustment.

[0194] In some embodiments, the computer readable instructions may be executed by one or more processors to cause the one or more processors to further perform: detecting and tracking a user's gesture; and comparing the user's gesture to a target gesture. Optionally, when it is determined that the user's gesture is a target gesture, the user is identified as a target user.

[0195] The functional modules / units in all or some steps, systems, and devices disclosed in the above method may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically include computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0196] For the purpose of illustration and description, the above description of the embodiments of the present invention has been given. It is not exhaustive, nor is it intended to limit the present invention to the precise form or exemplary embodiments disclosed. Therefore, the foregoing description should be considered illustrative rather than restrictive. Obviously, many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described to explain the principles of the present invention and its best mode practical application, so that those skilled in the art can understand the various embodiments of the present invention and the various modifications suitable for the specific use or implementation under consideration. The scope of the present invention is intended to be defined by the appended claims and their equivalents, wherein all terms are meant to have the broadest reasonable meaning unless otherwise stated. Therefore, the term "the present invention" and the like do not necessarily limit the scope of the claims to a specific embodiment, and the reference to the exemplary embodiments of the present invention does not mean a limitation of the present invention, and such limitation should not be inferred. The present invention is limited only by the spirit and scope of the appended claims. In addition, these claims may involve the use of "first", "second", etc., followed by a noun or element. These terms should be understood as nomenclature, and should not be interpreted as limiting the number of elements modified by these nomenclatures, unless a specific number has been given. Any advantages and benefits described may not apply to all embodiments of the present invention. It should be understood that those skilled in the art may make changes to the described embodiments without departing from the scope of the present invention as defined by the appended claims. In addition, the elements and assemblies in this disclosure are not intended to be contributed to the public, regardless of whether the element or assembly is clearly described in the appended claims.

Claims

1. A computer-implemented method comprising: Determine the reference surface; Determine the target audience of target users; determining a relative displacement between the target object and the reference surface; as well as Based on the relative displacement, a user-machine interaction instruction is determined.

2. The computer-implemented method of claim 1 , wherein: Determining the reference surface comprises: Determining coordinates of a plurality of calibration points in a three-dimensional coordinate system; and Based on the plurality of calibration points, the reference surface is derived.

3. The computer-implemented method of claim 2, wherein: Determining the coordinates of the plurality of calibration points in the three-dimensional coordinate system comprises: Capturing images of real-world objects using a camera; and Based on the parameters of the camera, the coordinates of the multiple calibration points in the image of the real-world object in the image coordinate system are converted into the coordinates of the multiple calibration points in the three-dimensional coordinate system.

4. A computer-implemented method according to any one of claims 1 to 3, wherein: The user-machine interaction instructions include displaying a trajectory image; The computer-implemented method further comprises: detecting and tracking an object motion trajectory of the target object relative to the reference surface; and The trajectory image is displayed in a target area based on the motion trajectory of the object relative to the reference surface.

5. The computer-implemented method of claim 4, further comprising: Determining whether the object motion trajectory is transformed into a pause gesture; as well as After determining that the object motion trajectory is transformed into a pause gesture, the trajectory image is temporarily displayed.

6. The computer-implemented method of claim 4, further comprising: converting the object motion trajectory relative to the reference surface into the trajectory image in real time; When a pause gesture is detected, pausing converting the object motion trajectory relative to the reference surface into the trajectory image; as well as After pausing the conversion of the object motion trajectory relative to the reference surface into the trajectory image, continuing to detect and track the object motion trajectory of the target object relative to the reference surface.

7. The computer-implemented method of claim 6, further comprising: Restoring the object motion trajectory relative to the reference surface into a new trajectory image in real time at a new reference point, where the new trajectory image is started relative to a previous trajectory image; Wherein, the new reference point is determined based on the result of continuing to detect and track the motion trajectory of the object relative to the reference surface.

8. The computer-implemented method of claim 5, wherein: The pause gesture is a gesture in which the target object moves toward a pause direction relative to the reference surface; as well as If the object making the object motion trajectory moves a distance exceeding a second threshold distance along the pause direction relative to the reference surface, the gesture is recognized as the pause gesture.

9. The computer-implemented method of claim 4, wherein: The computer-implemented method further comprises: Determining whether the object motion trajectory is transformed into a display gesture; and When it is determined that the object motion trajectory is transformed into a display gesture, the trajectory image is displayed in a target area based on the object motion trajectory relative to the reference surface.

10. The computer-implemented method of claim 9, wherein: The display gesture is a gesture that brings the target user's finger into contact with the reference surface.

11. The computer-implemented method of any one of claims 1 to 10, further comprising: identifying an initiation gesture indicating an initiation of providing said user-machine interaction instruction; After the initiating gesture is recognized, providing the user-machine interaction instruction is executed.

12. The computer-implemented method of any one of claims 1 to 3, wherein: The user-machine interaction instructions include executing user commands; The computer-implemented method further comprises: detecting and tracking the object motion trajectory of the target object relative to the reference surface; and The user command is executed based on the object motion trajectory relative to the reference surface.

13. A computer-implemented method according to any one of claims 1 to 12, wherein: The user-machine interaction instructions include adjusting a reference point; Wherein, adjusting the reference point includes recognizing an adjustment gesture of the target user.

14. The computer-implemented method of claim 13, wherein: The adjustment gesture is a gesture that causes the target object of the target user to be out of contact with the reference surface and to move along an adjustment direction; Wherein, the computer-implemented method further comprises: adjusting the reference point along the adjustment direction of the movement of the target object; Wherein, when the movement of the target object is within a distance relative to the reference surface, the gesture is recognized as the adjustment gesture; and the distance is less than a first threshold distance.

15. The computer-implemented method of claim 13, wherein: The reference point is the point at which the trajectory image starts to be displayed.

16. The computer-implemented method of any one of claims 1 to 11 and 13 to 15, further comprising: A target area on the display panel is determined that is designated for the target user to provide the user-machine interaction instruction.

17. The computer-implemented method of claim 16, wherein: The user-machine interaction instructions include displaying a trajectory image in the target area; Wherein, determining the target area includes: determining a gaze point on the display panel; and Based on the gaze point, the target area is determined.

18. The computer-implemented method of claim 17, wherein: Determining the gaze points includes: determining whether K gaze points of the target user, respectively determined from K image frames of the target user, are located at substantially the same position.

19. The computer-implemented method of claim 16, wherein: Determining the target area includes: Determine the coordinates of multiple calibration points in a three-dimensional coordinate system; Establishing a spatial equation of a real-world object based on the coordinates of the plurality of calibration points in the three-dimensional coordinate system; Determining the position coordinates of the target object in the three-dimensional coordinate system; and The position coordinates of the target object in the three-dimensional coordinate system are mapped to the input position coordinates in the display panel coordinate system, thereby determining the target area.

20. The computer-implemented method of claim 16, wherein: The display panel is virtually divided into a plurality of candidate target areas; The computer-implemented method further includes: specifying at least one candidate target area among the plurality of candidate target areas as the target area for the target user.

21. The computer-implemented method of claim 20, further comprising activating the target area; in, Activating the target area includes: identifying a corresponding selection gesture that matches one of the plurality of candidate selection gestures; and A corresponding candidate target area among the plurality of candidate target areas is activated to be designated as the target area of ​​the target user.

22. A computer-implemented method according to any one of claims 16 to 21, wherein: The user-machine interaction instruction further includes adjusting the target area to obtain an adjusted target area.

23. The computer-implemented method of claim 22, wherein: The target object includes a palm of the target user; and adjusting the target area includes identifying a gesture for adjusting a position of the palm of the target user.

24. The computer-implemented method of claim 22, further comprising: It is determined whether the adjusted target area overlaps with any of the activated target areas in the display panel.

25. The computer-implemented method of claim 24, further comprising: When it is determined that the adjusted target region overlaps with any one of the activated target regions in the display panel, a notification is provided to the target user that the adjusted target region overlaps with any one of the activated target regions in the display panel.

26. The computer-implemented method of claim 24, further comprising: When it is determined that the adjusted target area does not overlap with any of the activated target areas in the display panel, the target area is replaced with the adjusted target area for inputting information or providing a user command.

27. The computer-implemented method of any one of claims 22 to 26, further comprising: Recognizing a confirmation gesture of the target user; as well as When the confirmation gesture is recognized, when the target area is subject to adjustment, the adjusted target area is accepted as the confirmation area for inputting information or providing user commands, or when the target area is not subject to adjustment, the target area is accepted as the confirmation area for inputting information or providing user commands.

28. A computer-implemented method according to any one of claims 1 to 27, wherein: Identifying the target user includes: detecting and tracking gestures of the user; and comparing the gesture of the user to a target gesture; When it is determined that the gesture of the user is the target gesture, the user is identified as the target user.

29. A user input system comprising: one or more cameras; Display panel; Memory; as well as one or more processors; wherein the memory and the one or more processors are connected to each other; and The memory stores computer executable instructions for controlling the one or more processors to perform the following operations: Determine the reference surface; Determine the target audience of target users; determining a relative displacement between the target object and the reference surface; and Based on the relative displacement, a user-machine interaction instruction is determined.

30. A computer program product comprising a non-transitory tangible computer readable medium having computer readable instructions thereon, the computer readable instructions executable by a processor to cause the processor to: Determine the reference surface; Determine the target audience of target users; determining a relative displacement between the target object and the reference surface; and Based on the relative displacement, a user-machine interaction instruction is determined.

Images