Cursor positioning method and electronic equipment

Abstract

The invention discloses a cursor positioning method and electronic equipment. The method comprises the following steps: tracking the sight line of a user; determining position point information of a fixation point of the sight line of the user in a three-dimensional scene output by the naked eye three-dimensional display device in response to the situation that the sight line of the user meets the change condition; based on the position point information, performing coarse positioning on a cursor in the three-dimensional scene to obtain an initial position point of the cursor; and performing positioning adjustment on the initial position point of the cursor based on the input operation of the user to obtain a target position point of the cursor.

Description

Technical Field

[0001] This application relates to the field of display control technology, and in particular to a cursor positioning method and an electronic device. Background Technology

[0002] When displaying 3D scenes using a glasses-free 3D display device, users can directly view 3D images without wearing polarized glasses or other auxiliary equipment. However, it is currently difficult to accurately determine the required 3D coordinates within a 3D scene using a mouse. Summary of the Invention

[0003] On the one hand, this application provides a cursor positioning method, including:

[0004] Track the user's gaze;

[0005] In response to the user's gaze meeting the change condition, the position information of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device is determined;

[0006] Based on the location point information, the cursor in the three-dimensional scene is coarsely located to obtain the initial position point of the cursor;

[0007] The initial position of the cursor is adjusted based on the user's input to obtain the target position of the cursor.

[0008] In one possible implementation, determining the position information of the user's gaze point in the 3D scene output by the glasses-free 3D display device includes:

[0009] Determine the range of three-dimensional coordinates of the user's gaze point in the three-dimensional scene output by the naked-eye 3D display device;

[0010] The step of coarsely locating the cursor in the 3D scene based on the location point information to obtain the initial position point of the cursor includes:

[0011] Using the three-dimensional coordinate range as the coarse positioning area for the cursor, the cursor in the three-dimensional scene is moved into the coarse positioning area to obtain the initial position point of the cursor;

[0012] or,

[0013] The determination of the position information of the user's gaze point in the 3D scene output by the naked-eye 3D display device includes:

[0014] Determine the target's 3D coordinates in the 3D scene output by the naked-eye 3D display device, based on the user's gaze point;

[0015] The step of coarsely locating the cursor in the 3D scene based on the location point information to obtain the initial position point of the cursor includes:

[0016] The cursor in the three-dimensional scene is moved to the position point corresponding to the three-dimensional coordinates of the target to obtain the initial position point of the cursor.

[0017] In another possible implementation, adjusting the initial position of the cursor based on user input includes at least one of the following:

[0018] Based on the input operation and the initial position of the cursor, adjust the position of the cursor;

[0019] Based on the input operation and the initial position of the cursor, adjust the position of the cursor within the coarse positioning area;

[0020] Based on the input operation and the initial position of the cursor, the position of the cursor is adjusted within the set range of the target's three-dimensional coordinates.

[0021] In yet another possible implementation, the tracking of the user's gaze includes:

[0022] When the electronic device is in its first positioning state, it tracks the user's gaze;

[0023] The cursor positioning method further includes: detecting that the user's operation meets the switching conditions, and switching the positioning state of the electronic device from the first positioning state to the second positioning state;

[0024] The step of adjusting the initial position of the cursor based on user input includes:

[0025] When the electronic device is in the second positioning state, the initial position of the cursor is adjusted based on the user's input.

[0026] In another possible implementation, the operation action satisfies the switching conditions, including: the operation amplitude of the operation action is greater than a set threshold and / or the duration of the operation action exceeds a duration threshold.

[0027] The cursor positioning method further includes: when the electronic device is in a second positioning state, if no input operation from the user is detected within a specified period of time, switching the positioning state of the electronic device from the second positioning state to the first positioning state.

[0028] In another possible implementation, determining the position information of the user's gaze point in the 3D scene output by the glasses-free 3D display device includes:

[0029] Determine the two-dimensional coordinates of the user's gaze point on the naked-eye 3D display device;

[0030] Determine the target three-dimensional mapping function based on the three-dimensional mapping function corresponding to at least one depth coordinate interval;

[0031] Based on the two-dimensional coordinates, the position information of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device is determined using the target three-dimensional mapping function.

[0032] In another possible implementation, determining the target 3D mapping function based on a 3D mapping function corresponding to at least one depth coordinate interval includes:

[0033] From the three-dimensional mapping functions corresponding to at least one depth coordinate interval, determine the target three-dimensional mapping function that is currently available as a mapping function;

[0034] The cursor positioning method also includes:

[0035] If the depth value in the target three-dimensional coordinates or the depth range in the three-dimensional coordinate range exceeds the target depth coordinate interval corresponding to the target three-dimensional mapping function, determine the target interval boundary in the target depth coordinate interval that is closest to the three-dimensional coordinates or the three-dimensional coordinate range, and increment the number of times the target interval boundary has exceeded the boundary by one;

[0036] If the number of times the boundary of the target interval is crossed exceeds a set threshold, a candidate depth coordinate interval that is outside the target depth coordinate interval and adjacent to the boundary of the target interval is determined.

[0037] The three-dimensional mapping function corresponding to the candidate depth coordinate interval is determined as the currently available mapping function.

[0038] Another possible implementation includes:

[0039] The position information of the gaze point is calibrated using a calibration function to obtain calibrated position information. The calibration function is a function fitted based on at least one calibration coordinate pair. The calibration coordinate pair includes: the historical three-dimensional coordinates corresponding to the historical gaze point of the user's gaze, and the cursor three-dimensional coordinates corresponding to the historical three-dimensional coordinates. The cursor three-dimensional coordinates are the historical target position point of the cursor corresponding to the historical gaze point.

[0040] The cursor positioning method further includes:

[0041] In response to a target event indicating the presence of a target location point of the cursor, at least one calibration coordinate pair is obtained, wherein the target event is used to indicate that the user confirms that the target location point of the cursor is accurately positioned.

[0042] Based on at least one of the calibration coordinate pairs obtained within the most recently set time period, a calibration function is fitted to convert the historical three-dimensional coordinates in the calibration coordinate pairs into the cursor three-dimensional coordinates.

[0043] In another possible implementation, obtaining at least one calibration coordinate pair in response to a target location event of the cursor includes:

[0044] In response to a target location event of the cursor, a first coordinate sequence and a second coordinate sequence are obtained. The first coordinate sequence includes at least one gaze point three-dimensional coordinates, and the second coordinate sequence includes cursor three-dimensional coordinates corresponding to the at least one gaze point three-dimensional coordinates. The at least one gaze point three-dimensional coordinates are the three-dimensional coordinates of at least one historical gaze point of the user's line of sight in the three-dimensional scene within the target duration range corresponding to the current moment.

[0045] Determine the trajectory similarity between the first coordinate trajectory corresponding to the first coordinate sequence and the second coordinate trajectory corresponding to the second coordinate sequence;

[0046] In response to the trajectory similarity exceeding a first threshold, a trajectory transformation function is determined for converting the first coordinate trajectory into a corresponding second coordinate trajectory;

[0047] The trajectory conversion function is used to convert the three-dimensional coordinates of the gaze point to obtain the reference three-dimensional coordinates corresponding to the three-dimensional coordinates of the gaze point;

[0048] If the distance between the reference 3D coordinates corresponding to the gaze point 3D coordinates and the cursor 3D coordinates corresponding to the gaze point 3D coordinates is less than a second threshold, the gaze point 3D coordinates are used as historical 3D coordinates and form a calibration coordinate pair with the cursor 3D coordinates corresponding to the gaze point 3D coordinates.

[0049] On the other hand, this application provides an electronic device, including: a naked-eye 3D display device, a data acquisition device, and a processor;

[0050] The acquisition device is used to track the user's line of sight;

[0051] The processor is configured to, in response to a change in the user's gaze, determine the position information of the user's gaze point in the three-dimensional scene output by the naked-eye 3D display device; based on the position information, perform coarse positioning of the cursor in the three-dimensional scene to obtain the initial position point of the cursor; and adjust the initial position point of the cursor based on the user's input operation to obtain the target position point of the cursor. Attached Figure Description

[0052] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0053] Figure 1 A flowchart illustrating the cursor positioning method provided in this application;

[0054] Figure 2 Another flowchart illustrating the cursor positioning method provided in this application;

[0055] Figure 3 Another flowchart illustrating the cursor positioning method provided in this application;

[0056] Figure 4 Another flowchart illustrating the cursor positioning method provided in this application;

[0057] Figure 5 This is a schematic diagram illustrating the switching process between the two positioning states of the electronic device in this application;

[0058] Figure 6 Another flowchart illustrating the cursor positioning method provided in this application;

[0059] Figure 7 Another flowchart illustrating the cursor positioning method provided in this application;

[0060] Figure 8 This application presents a schematic diagram illustrating an implementation process for constructing a calibration function;

[0061] Figure 9 This is a schematic diagram of one implementation process for obtaining calibration coordinate pairs in this application;

[0062] Figure 10 Example diagram of the process for selecting calibration coordinate pairs for this application;

[0063] Figure 11 A schematic diagram of the component architecture of the electronic device provided in this application. Detailed Implementation

[0064] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0066] like Figure 1 The diagram illustrates a flowchart of a cursor positioning method provided in this application. The method of this embodiment can be applied to an electronic device that has or is connected to a naked-eye 3D display device.

[0067] The method in this embodiment may include:

[0068] S101 tracks the user's gaze.

[0069] In this application, the purpose of tracking the user's gaze is to determine the user's gaze point in the naked-eye 3D display device.

[0070] For example, eye trackers or infrared cameras can be used to continuously collect images of the user's eyes, and based on these images, the direction of the user's gaze and the point where the gaze lands can be determined without any restrictions.

[0071] S102, in response to the user's gaze meeting the change condition, determines the position information of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device.

[0072] Among them, the condition for redetermining the user's gaze point is the trigger condition for re-determining the user's gaze point. The condition for redetermining the user's gaze point can include one or more of the following: the user's gaze point changes, the magnitude of the change in the user's gaze point exceeds a set magnitude threshold, and the duration of the change in the user's gaze point exceeds a set target duration.

[0073] It is understood that a glasses-free 3D display device can output different images to each of the user's eyes according to the position of the user's eyes, and enable the user to view a 3D image through the interlacing of images. In this application, the 3D scene output by the glasses-free 3D display device is either a virtual 3D scene composed of images output by the glasses-free 3D display device, or a scene corresponding to a 3D image output by the glasses-free 3D display device.

[0074] In this application, the location information of the user's gaze point in the 3D scene refers to the location information of the point where the user's gaze falls in the 3D scene. This location information can be a single 3D coordinate or a range of 3D coordinates, without limitation.

[0075] It is understandable that there are multiple ways to determine the location information corresponding to the user's gaze point based on the user's gaze, and this application does not impose any restrictions on this.

[0076] S103, based on the position point information, performs coarse positioning of the cursor in the 3D scene to obtain the initial position point of the cursor.

[0077] Coarse positioning of the cursor refers to determining the cursor's position based on the location point information to obtain the cursor's initial position point. For example, coarse positioning of the cursor can be achieved by adjusting the cursor's position based on the location point, so that the cursor moves to the coordinate position, coordinate range, or a certain range of the coordinate position corresponding to the location point information.

[0078] In this application, the final positioning of the cursor is completed in two stages. The first positioning stage, or coarse positioning stage, is the positioning of the cursor based on the position information of the user's gaze point. In the first positioning stage, the accuracy requirement for the initial position of the cursor is relatively low, and adjusting the position of the cursor based on the position information is actually a coarse positioning operation.

[0079] It should be noted that the user's gaze may move multiple times, which may cause the user's gaze to meet the change condition multiple times. Each time the user's gaze meets the change condition, the position information corresponding to the user's gaze point will be re-determined. Therefore, the above steps can be performed multiple times.

[0080] S104, based on the user's input operation, adjust the initial position of the cursor to obtain the target position of the cursor.

[0081] The user's input operations can be operations performed using a mouse, controller (such as a gamepad), or finger, and these operations are used to control cursor movement. For example, the input operation could be moving the mouse or scrolling the mouse wheel; adjusting the position of the cursor via a controller; or moving the user's finger in physical space.

[0082] In this application, adjusting the initial position of the cursor based on input operations can be achieved by first positioning the cursor at that initial position, then adjusting its position based on the input operation to obtain the final position of the cursor, i.e., the target position. This adjustment of the cursor's position based on input operations constitutes the second positioning stage, the fine-tuning stage. This second positioning stage is a precise positioning stage; therefore, by moving the cursor in conjunction with user input operations, the initial position of the cursor is adjusted to accurately obtain the target position.

[0083] It's understandable that the user's gaze point in a 3D scene can accurately reflect their focus. Therefore, after coarsely locating the cursor based on the user's gaze point in the 3D scene, the initial cursor position is relatively close to the user's desired position. Based on this, a second stage of cursor positioning adjustment is performed based on the user's input. This adjustment is relatively minor, and consequently, the duration, magnitude, and number of input operations required to adjust the cursor position can be reduced, lowering the overall complexity of the operation.

[0084] As can be seen from the above, in this application, after determining the position information of the user's gaze point in the 3D scene output by the naked-eye 3D display device, the cursor in the 3D scene is first coarsely located based on this position information. Since the initial position of the cursor determined based on the coarse positioning will be relatively close to the cursor position expected by the user, the cursor is then repositioned and adjusted based on the initial position obtained from the coarse positioning, combined with the user's input operation. This not only improves the accuracy of cursor positioning but also reduces the need for users to perform large input movements, multiple manual adjustments, or long input operations to adjust the cursor, thus reducing the difficulty and complexity of cursor positioning.

[0085] In this application, the location information of the user's gaze point in the 3D scene can be specific 3D coordinates, a range of 3D coordinates, or other feature information that can characterize the annotation area of ​​the gaze point in the 3D scene; there are no restrictions on this. The implementation of coarse cursor positioning based on the location information will also differ depending on the specific content of the location information. Several possible scenarios are explained below.

[0086] In the first possible scenario, the location information of the user's gaze point in the 3D scene can be the 3D coordinate range of that gaze point within the 3D scene. The following will combine... Figure 2 Please provide an explanation.

[0087] like Figure 2This illustration shows another implementation flowchart of a cursor positioning method provided in this application. The method in this embodiment may include:

[0088] S201, tracks the user's gaze.

[0089] S202, in response to the user's gaze meeting the changing conditions, determine the range of three-dimensional coordinates of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device.

[0090] In this application, the location information of the gaze point is the three-dimensional coordinate range of the gaze point, and this three-dimensional coordinate range is the gaze area of ​​the user's line of sight, that is, the scene space area that the user focuses on in the three-dimensional scene.

[0091] S203, using the three-dimensional coordinate range as the coarse positioning area of ​​the cursor, move the cursor in the three-dimensional scene to the coarse positioning area to obtain the initial position point of the cursor.

[0092] In this embodiment, during the coarse positioning stage of the cursor, it is only necessary to move the cursor in the 3D scene to a position within the 3D coordinate range of the user's visual focus. The specific location of the cursor within this 3D coordinate range is not restricted. For example, the cursor can be moved to the center point of the 3D coordinate range, or it can be randomly moved to any point within that range.

[0093] Once the cursor is moved into the range of these three-dimensional coordinates, the current position of the cursor is its initial position.

[0094] S204, based on the user's input operation, adjust the initial position of the cursor to obtain the target position of the cursor.

[0095] Understandably, determining the 3D coordinate range of the user's gaze point within a 3D scene does not require precisely pinpointing the exact location of the gaze point, thus reducing the difficulty and time required to locate the user's gaze position within the 3D scene. Based on this, using this 3D coordinate range as the coarse positioning area for the cursor, moving the cursor to this area allows for convenient and efficient coarse cursor positioning. Furthermore, after coarsely positioning the initial cursor position, user input can still be used to adjust the cursor's position, thereby accurately locating the cursor's target position.

[0096] In this application, there are several possibilities for adjusting the initial position of the cursor based on input operations.

[0097] For example, in one possible scenario, the cursor position can be adjusted based on the input operation and the cursor's initial position. In this implementation, the adjusted cursor position depends only on the cursor's initial position and the magnitude of the input operation; the range of cursor position adjustment is unrestricted, enabling more flexible cursor position adjustment.

[0098] In another possible scenario, the cursor position can be adjusted within the coarse positioning area based on the input operation and the cursor's initial position. In this case, the cursor position can be adjusted via input, but only within the coarse positioning area, ensuring the target cursor position remains within this area. In this scenario, after determining the coarse positioning area based on the user's line of sight, the user can only control the cursor's movement within this area. This reduces the likelihood of accidentally making excessive input movements that could cause the cursor position to deviate further from the desired location, thus improving cursor positioning efficiency.

[0099] In the first possible scenario, the location information of the user's gaze point in the 3D scene can be the 3D coordinates of that gaze point. The following will combine... Figure 3 Please provide an explanation. For example... Figure 3 This illustration shows another flowchart of the cursor positioning method provided in this application. The method in this embodiment may include:

[0100] S301 tracks the user's gaze.

[0101] S302, in response to the user's gaze meeting the changing conditions, determines the target's three-dimensional coordinates in the three-dimensional scene output by the naked-eye three-dimensional display device, based on the user's gaze point.

[0102] For ease of distinction, the three-dimensional coordinates of the gaze point in the three-dimensional scene are referred to as the target three-dimensional coordinates. This application does not impose restrictions on the specific implementation method for determining the three-dimensional coordinates of the user's gaze point in the three-dimensional scene.

[0103] S303, move the cursor in the 3D scene to the position point corresponding to the 3D coordinates of the target, and obtain the initial position point of the cursor.

[0104] In this embodiment, the cursor moves to the target's 3D coordinates in the 3D scene where the user's gaze point is located. Therefore, the cursor's initial position is also the user's gaze point in the 3D scene. Based on this, given the accurate location of the gaze point in the target's 3D coordinates, coarse positioning can also accurately pinpoint the cursor's location, making the cursor's initial position closer to or at the user's desired position. This further reduces the complexity of subsequent manual fine-tuning of the cursor by the user.

[0105] S304, based on the user's input operation, adjust the initial position of the cursor to obtain the target position of the cursor.

[0106] In this embodiment, there are no restrictions on the specific implementation of adjusting the initial position of the cursor based on the user's input.

[0107] For example, in one possible implementation, the cursor position can be adjusted based on the input operation and the cursor's initial position. In this possible implementation, the specific range of the cursor position adjustment based on the input operation is not limited; please refer to the previous related introduction for details.

[0108] In another possible implementation, the cursor position can be adjusted within a set range of the target's three-dimensional coordinates based on the input operation and the cursor's initial position. In this implementation, the cursor position can only be adjusted within a set range of the three-dimensional coordinates corresponding to the user's gaze point, provided the user's gaze point's three-dimensional coordinates can be determined relatively accurately. Controlling the cursor movement within this coarse positioning area based on the user's input operation reduces the possibility of the cursor deviating further from the desired position due to accidentally excessive input movement, thus improving cursor positioning efficiency.

[0109] The setting range can be set according to actual needs.

[0110] In one alternative approach, to further improve the accuracy of cursor positioning and reduce the complexity of manual cursor positioning by the user, the set range can be the error range of the target's three-dimensional coordinates. This error range is the inherent error range of the calculation algorithm or model used to calculate the target's three-dimensional coordinates. For example, the error range of the calculation algorithm or model used to determine the three-dimensional coordinates of the gaze point in this application under the target confidence level can be tested. For instance, the error range can be the errors on the three-dimensional coordinate axes, respectively... , , Then the error range can be set to this preset range. The target confidence level can be set according to actual needs, such as a target confidence level of 95%.

[0111] It is understandable that after detecting user input, the user's gaze may change due to focusing on the mouse or other objects, or other reasons. However, the purpose of this change in gaze is not to adjust the initial cursor position. Therefore, if the initial cursor position is adjusted at this time, it will also lead to cursor positioning deviation or even error.

[0112] Furthermore, during the process of locating the position of the user's gaze point, there may be situations where the electronic device detects input operations due to erroneous operations. Both the user's gaze and the input operation can trigger the adjustment of the cursor position, which will inevitably lead to cursor positioning confusion, and may result in the inability to accurately locate the cursor position.

[0113] Therefore, in order to improve positioning accuracy, this application can set two positioning states for the electronic device: a first positioning state and a second positioning state. The first positioning state is a state stage where cursor positioning is based on the user's line of sight, while the second positioning state is a state node where cursor positioning is based on the user's input operation. That is, when the electronic device is in the first positioning state, it enters the coarse positioning stage of the cursor (the first positioning stage); and when the electronic device is in the second positioning state, it enters the fine positioning stage of the cursor (the second positioning stage).

[0114] The following is combined Figure 4 Please provide an explanation, such as Figure 4 This illustration shows another flowchart of the cursor positioning method provided in this application. The method in this embodiment may include:

[0115] S401 tracks the user's gaze when the electronic device is in its first positioning state.

[0116] S402, in response to the user's gaze meeting the change condition, determines the position information of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device.

[0117] S403, Based on the location information, perform coarse positioning of the cursor in the 3D scene to obtain the initial position of the cursor.

[0118] In this application, the electronic device tracks the user's line of sight only when the electronic device is in the first positioning state, and performs coarse positioning of the cursor to determine the initial position point of the cursor when the user's line of sight meets the change conditions.

[0119] In practical applications, after the naked-eye 3D display device of an electronic device is activated, the electronic device can default to the first positioning state.

[0120] S404, The user's operation action is detected to meet the switching conditions, and the positioning state of the electronic device is switched from the first positioning state to the second positioning state.

[0121] In this context, user actions refer to actions performed by the user using a mouse, controller, or finger. For example, detecting a user action can involve detecting a change in the state of the action, including but not limited to changes such as switching from a static to a moving state, a change in the action's position, or a switch from a non-input state to an input state. Taking a mouse as an example, if mouse movement or scrolling of the mouse wheel is detected, then a user action on the mouse is confirmed.

[0122] In this application, the user's operation satisfying the switching condition indicates that the user needs to manually adjust the cursor position; that is, the user has a need to adjust the cursor through input. Based on this, when the user's operation satisfies the switching condition, the electronic device can be switched to a second positioning state, enabling the electronic device to perform fine positioning of the cursor based on the user's input.

[0123] The switching conditions for the operation can be set as needed. For example, to reduce false positives, the switching conditions can include at least one of the following: the amplitude of the operation exceeds a set threshold, and the duration of the operation exceeds a set duration threshold. For instance, if the operator is a finger, the amplitude of the operation could be the distance the finger moves exceeding a set distance threshold; if the operator is a mouse, the amplitude of the operation could be the distance the mouse moves, or the amount of scrolling of the mouse wheel (e.g., scroll angle, number of scroll teeth, or scroll amplitude). The duration threshold can be set based on actual needs, such as 2 milliseconds.

[0124] S405, when the electronic device is in the second positioning state, the initial position of the cursor is adjusted based on the user's input operation to obtain the target position of the cursor.

[0125] In this application, when the electronic device enters the second positioning state, the electronic device will adjust the initial position of the cursor based on the user's input operation. Even if the user's line of sight changes, the electronic device will not adjust the initial position of the cursor again.

[0126] As can be seen, in this embodiment, the electronic device has two positioning states. In the first positioning state, the electronic device performs coarse positioning of the cursor in the 3D scene based on the user's line of sight to determine the cursor's initial position. Only when the user's operation meets the switching conditions will the electronic device switch from the first positioning state to the second positioning state. In the second positioning state, the electronic device can adjust the cursor's initial position solely based on the user's input, effectively distinguishing between the coarse and fine positioning stages. This reduces cursor positioning anomalies caused by interference between the user's line of sight and input, improving the accuracy of cursor positioning.

[0127] Understandably, after the user completes fine cursor positioning, the user can end the input operation. However, to allow the user to still achieve coarse cursor positioning through their line of sight, this application can switch the electronic device from a second positioning state to a first positioning state. Specifically, when the electronic device is in the second positioning state, if no input operation from the user is detected within a specified time period, the positioning state of the electronic device is switched from the second positioning state to the first positioning state.

[0128] For ease of understanding, combined with Figure 5 Let's take the example of a user precisely positioning a cursor in a 3D scene by manipulating the mouse. Figure 5 The diagram illustrates the switching process between two positioning states of an electronic device.

[0129] exist Figure 5 In this context, eye-tracking positioning indicates that the electronic device is in a first positioning state, while mouse adjustment represents that the electronic device is in a second positioning state. For ease of understanding, we will use the 3D coordinates of the gaze point in the 3D scene as an example to illustrate this.

[0130] Depend on Figure 5 It can be seen that when the electronic device is in eye-tracking positioning state (first positioning state), the electronic device will determine the three-dimensional coordinates of the user's gaze point in the three-dimensional scene. The cursor is moved within the 3D scene based on these 3D coordinates to obtain the cursor's initial positioning point. In eye-tracking mode, the cursor's initial positioning point changes as the user's gaze shifts.

[0131] If the electronic device detects mouse movement while in eye-tracking mode, it can determine whether the mouse movement distance exceeds a distance threshold m and the continuous duration of the mouse movement exceeds a set duration threshold n. If the mouse movement distance does not exceed m and the continuous duration of the mouse movement does not exceed n, the electronic device will remain in eye-tracking mode.

[0132] Of course, it's also possible to detect only whether the mouse movement distance exceeds m or whether the continuous duration of mouse movement exceeds n. Additionally, in practical applications, it's possible to monitor whether the mouse wheel scrolling amount exceeds a threshold or whether the continuous scrolling duration exceeds n, etc., details of which will not be elaborated further.

[0133] If the mouse moves a distance greater than m and the continuous duration of the mouse movement exceeds n, the electronic device will still switch from eye-tracking positioning state to mouse adjustment state (i.e., second positioning state).

[0134] When an electronic device is in mouse positioning mode, the cursor position can be adjusted within the error range corresponding to the three-dimensional coordinates of the gaze point, based on the left-right movement distance, the up-down movement distance of the mouse, and the scroll amount of the scroll wheel, to obtain the target cursor position. For example, left-right mouse movement can be used to adjust the cursor position... The amount of movement on the coordinate axis, such as the vertical movement of the mouse, can be used to adjust the cursor position. The amount of movement on the coordinate axis, and scrolling the mouse wheel can adjust the cursor position. The amount of movement on the coordinate axis.

[0135] like Figure 5 It can be seen that the error ranges of the three-dimensional coordinates of the fixation point on the three coordinate axes are as follows: , and ,in, , and Represents respectively in , and The maximum error on these three coordinate axes.

[0136] When an electronic device is in mouse adjustment mode, the cursor position can be continuously adjusted if the mouse is continuously operated. If the mouse stops moving, this application can monitor whether no mouse operation (such as mouse movement or scroll wheel scrolling) is detected for a set duration T seconds, at which point the electronic device will switch from mouse adjustment mode to eye-tracking positioning mode.

[0137] In the above embodiments of this application, there are various possible implementations for determining the location information of the user's gaze point.

[0138] It is understandable that determining the two-dimensional coordinates of the gaze point on a naked-eye 3D display device is a relatively mature technology. In order to more accurately determine the position information of the gaze point, this application can first determine the two-dimensional coordinates of the gaze point on the naked-eye 3D display device, and then map the two-dimensional coordinates to the position information in the 3D scene.

[0139] Furthermore, considering that the conversion of 2D to 3D positional information is not a simple linear mapping, if the same 2D-to-3D mapping function is used across the entire depth range of a 3D scene, the accuracy of the converted positional information will inevitably be poor. Therefore, this application employs a novel method for converting 2D coordinates into 3D positional information to improve the accuracy of the determined gaze point's positional information in the 3D scene. The following section combines... Figure 6 Please provide an explanation.

[0140] like Figure 6 This illustrates another flowchart of the cursor positioning method provided in this application. The method in this embodiment may include:

[0141] S601 tracks the user's gaze.

[0142] S602, in response to the user's gaze meeting the changing conditions, determines the two-dimensional coordinates of the user's gaze point on the naked-eye 3D display device.

[0143] In this application, there are no restrictions on the specific implementation of determining the two-dimensional coordinates of the user's gaze point on the naked-eye 3D display device.

[0144] S603, determine the target three-dimensional mapping function based on the three-dimensional mapping function corresponding to at least one depth coordinate interval.

[0145] It is understandable that the depth range of a 3D scene output by a naked-eye 3D display at different times is fixed and deterministic. However, the degree of convergence and divergence of the human eye varies when observing 3D objects at different distances. The degree of convergence and divergence is relatively greater when viewing closer objects, and relatively smaller when viewing distant objects. Therefore, the relationship between the distance between the human eye and objects in the 3D scene and the degree of convergence of the human eye cannot be represented by a simple function. Based on this theory, within the depth range of a 3D scene, it is impossible to accurately represent the conversion relationship from 2D coordinates to 3D coordinates or the range of 3D coordinates using a single function.

[0146] Based on this, in order to more accurately determine the 3D position information that the line of sight on the naked-eye 3D display device can be mapped to at different depths, this application segments the depth range that the 3D scene output by the naked-eye 3D display device can cover, obtaining at least one depth coordinate interval. For each depth coordinate interval, a corresponding 3D mapping function is constructed, so that the 3D mapping function corresponding to each depth coordinate interval can more accurately express the conversion relationship between depth and 2D coordinates within that depth coordinate interval.

[0147] There are many possible implementations for constructing the 3D mapping function corresponding to the depth coordinate interval, and no specific restrictions are imposed.

[0148] For example, in one possible implementation, for each depth coordinate interval, this application uses multiple sets of sample gaze point pairs corresponding to that depth coordinate interval to fit a 3D mapping function corresponding to that depth coordinate interval. This 3D mapping function can be a linear multivariate function, or of course, other types of functions.

[0149] Each sample gaze point pair includes the two-dimensional coordinates of the user's sample gaze point on the naked-eye 3D display device and the three-dimensional coordinates of that sample gaze point in the sample 3D scene output by the naked-eye 3D display device. For ease of distinction, the gaze point of the sample user's line of sight used in constructing the 3D mapping function and determining other subsequent functions is called the sample gaze point, and the 3D scene output by the naked-eye 3D display device used to collect sample points is called the sample 3D scene.

[0150] There are no restrictions on how the gaze pairs of the sample are obtained.

[0151] For example, when outputting a sample 3D scene through a naked-eye 3D display device, multiple reference points can be manually selected from the sample 3D scene to determine the 3D coordinates of each reference point within the sample 3D scene. Then, for each reference point... When the sample user gazes at the reference point, the two-dimensional coordinates of the gazing point on the naked-eye 3D display device are determined using eye-tracking devices such as eye trackers. ,in, The two-dimensional coordinates of the sample user's left eye gaze on the naked-eye 3D display device. The two-dimensional coordinates of the sample user's right eye gaze on the naked-eye 3D display device. Based on this, each reference point... The three-dimensional coordinates of the reference point and the corresponding two-dimensional coordinates This constitutes a sample gaze pair.

[0152] After obtaining multiple sets of sample gaze point pairs corresponding to each depth coordinate interval, this application can specify the reference point in each set of sample gaze point pairs. The three-dimensional coordinates are used as the supervised ground truth, with the reference point as the reference. Corresponding two-dimensional coordinates As input, a function is fitted. For ease of understanding, we will use the example of fitting a linear multivariate function, and how this fitted linear multivariate function can be used to transform the 3D representation of the gaze point, as an example. Based on multiple sets of sample gaze points, this application can fit a linear multivariate function represented by the following formula:

[0153] (Formula 1)

[0154] Formula 1 represents the first... depth coordinate range The corresponding three-dimensional mapping function. ,Right now In order to be with the first The two-dimensional coordinates corresponding to a depth in a depth coordinate interval For a piecewise linear mapping matrix, The offset is used to fit these two parameters based on the multiple sets of sample gaze point pairs, thereby determining the fitted linear multivariate function as the three-dimensional mapping function corresponding to the depth coordinate interval.

[0155] Based on this, the two-dimensional coordinates of the user's line of sight on the naked-eye 3D display device are determined. Substituting these values ​​into Formula 1 above, we can obtain the 3D coordinates of the target in the 3D scene from the user's gaze point. ,Right now .

[0156] In this application, the target 3D mapping function is one of the 3D mapping functions corresponding to the at least one depth coordinate interval. Furthermore, the target 3D mapping function is a 3D mapping function suitable for converting the 2D coordinates into depth coordinates or depth coordinate ranges.

[0157] In this application, there are multiple ways to determine the target 3D mapping function. For example, the target 3D mapping function can be determined from the 3D mapping functions corresponding to the at least one depth coordinate interval. In this case, after each calculation of position point information such as 3D coordinates or 3D coordinate range using the target 3D mapping function, it is necessary to evaluate whether the target 3D mapping function is suitable to continue as the 3D mapping function for calculating 3D coordinates. This will be described in step S604 later and will not be repeated here.

[0158] In another possible scenario, this application may also obtain at least one target historical 3D coordinate, which is the 3D coordinate of at least one historical gaze point corresponding to the user's gaze in the 3D scene. For example, the at least one target historical 3D coordinate is the 3D coordinate of at least one historical gaze point of the user's gaze obtained within the most recent preset time period in the 3D scene. Accordingly, a target 3D mapping function can be determined based on the depth coordinates in each target historical 3D coordinate and the 3D mapping function corresponding to at least one depth coordinate interval.

[0159] Understandably, in practical applications, during the coarse localization stage, since the movement of the user's gaze is a continuous process, it may be necessary to repeatedly determine the three-dimensional coordinates of the gaze point. If the three-dimensional coordinates corresponding to the user's gaze point have not been determined before the current moment, that is, there are no historical three-dimensional coordinates corresponding to the user's gaze point, then the target three-dimensional mapping function can be randomly determined from the three-dimensional mapping functions corresponding to at least one depth coordinate interval.

[0160] If at least one historical 3D coordinate of a target is obtained, since the user's line of sight changes relatively little in a very short time, the 3D mapping function associated with the depth coordinate interval corresponding to the average value of the depth coordinate of the at least one historical 3D coordinate of the target can be determined as the target 3D mapping function.

[0161] Furthermore, to more accurately determine a suitable target 3D mapping function, this application can also determine the reference 3D mapping function used for calculating the historical 3D coordinates of the user's gaze point in the most recent calculation. Accordingly, the at least one target historical 3D coordinate can be at least one historical 3D coordinate corresponding to the most recent at least one historical gaze point belonging to the user's gaze, determined using the 3D mapping function. Based on this, this application can determine the number of boundary crossings corresponding to the two interval boundaries of the reference depth coordinate interval corresponding to the reference 3D mapping function, based on the depth coordinates in the at least one target historical 3D coordinate. For each interval boundary of the reference depth coordinate interval, the number of boundary crossings is the total number of target historical 3D coordinates whose boundary is the boundary crossing boundary. Wherein, if the depth interval of the target historical 3D coordinate exceeds the reference depth coordinate interval, the interval boundary closest to the depth coordinate of the target historical 3D coordinate within the reference depth coordinate interval is the boundary crossing boundary corresponding to the target historical 3D coordinate.

[0162] Based on this, if there are candidate interval boundaries in the two interval boundaries of the reference depth coordinate interval where the number of out-of-bounds exceeds a set threshold, the other depth coordinate intervals adjacent to the candidate interval boundary are determined as candidate depth coordinate intervals, and the three-dimensional mapping function corresponding to the candidate depth coordinate interval is determined as the target three-dimensional mapping function.

[0163] S604, Based on the two-dimensional coordinates, the position information of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device is determined using the target three-dimensional mapping function.

[0164] For example, based on the two-dimensional coordinates, the target three-dimensional mapping function can be used to determine the three-dimensional coordinates or range of three-dimensional coordinates of the user's gaze point in the three-dimensional scene.

[0165] In particular, considering that the sample 3D coordinates or sample 3D coordinate ranges in the sample gaze point pairs used to construct the target 3D mapping function are based on the fixed and standard position of the sample user's head relative to the naked-eye 3D display device, in actual applications, the relative position of the user's head relative to the naked-eye 3D display screen may change.

[0166] Therefore, to make the calculated location information more accurate, this application can also obtain the relative spatial coordinates of the user's head relative to the naked-eye 3D display device. Based on this, after determining the target 3D coordinates or 3D coordinate range corresponding to the user's gaze point using a target 3D mapping function, this application can further correct the target 3D coordinates or 3D coordinate range based on the relative spatial coordinates to obtain the corrected target 3D coordinates or 3D coordinate range.

[0167] Understandably, when the target 3D mapping function is the most recently determined available mapping function, in order to ensure that the target 3D mapping function remains a suitable available mapping function, after determining the location point information, if the depth value in the target 3D coordinates or the depth range in the 3D coordinate range exceeds the target depth coordinate interval corresponding to the target 3D mapping function, this application also needs to determine the target interval boundary closest to the 3D coordinates or 3D coordinate range within the target depth coordinate interval, and increment the boundary crossing count of the target interval boundary by one. Correspondingly, if the boundary crossing count of the target interval boundary exceeds a set threshold, a candidate depth coordinate interval outside the target depth coordinate interval but adjacent to the target interval boundary can be determined; the 3D mapping function corresponding to the candidate depth coordinate interval is then determined as the currently available mapping function.

[0168] In this context, the boundary of the target interval closest to the given 3D coordinate or 3D coordinate range within the target depth coordinate interval is the boundary on one side where the 3D coordinate or 3D coordinate range exceeds the target depth coordinate interval. For example, if the depth coordinate interval is [5, 10], and the currently determined 3D coordinate depth value is 12, then it means that the 3D coordinate exceeds the boundary corresponding to "10" in the depth coordinate interval. Therefore, the depth coordinate interval [10, 15] can be determined as a candidate depth coordinate interval.

[0169] Specifically, to more accurately determine the available mapping function, this application can also count the number of consecutive boundary violations for each interval boundary in the target depth coordinate interval. Based on this, for the target 3D mapping function, after determining the target interval boundary, if the number of consecutive boundary violations for another interval boundary in the target depth coordinate interval is not 0, the number of consecutive boundary violations for that other interval boundary is reset to zero, and the number of consecutive boundary violations for the target interval boundary is incremented by one. Of course, if the target 3D coordinates or the range of 3D coordinates do not exceed the target depth coordinate interval corresponding to the target 3D mapping function, the number of consecutive boundary violations for both interval boundaries of the target depth coordinate interval can be set to zero, and the target 3D mapping function can continue to be used as a usable mapping function.

[0170] S605, based on this position information, coarsely locate the cursor in the 3D scene to obtain the initial position of the cursor.

[0171] S606, based on the user's input operation, adjusts the initial position of the cursor to obtain the target position of the cursor.

[0172] The above steps S605 and S606 can be described in relation to other embodiments of this application, and will not be repeated here.

[0173] In the embodiments, at least one depth coordinate range can be set according to actual needs or in combination with experience.

[0174] In one possible implementation, in order to more reasonably determine each depth coordinate interval and achieve a more reasonable piecewise fitting of the three-dimensional mapping function, this application can also combine the theoretical convergence angle function to reasonably determine at least one depth coordinate interval corresponding to the naked-eye three-dimensional display device.

[0175] As discussed earlier, the human eye exhibits varying degrees of convergence and divergence when observing 3D objects at different distances. In processing the depth range of a stereoscopic display, we can start with the theoretical convergence and divergence angle function and divide the display space along the depth direction into several segments. This allows the convergence and divergence angle variation within each segment to be represented by a low-error quadratic function, thus meeting the accuracy requirements of the subsequent calibration function fitting. For ease of description, we will use the 3D coordinate axes in the following sections. Taking the axis representing the depth direction as an example:

[0176] First, determine the range of scene depth that the naked-eye 3D display device can output, and use this range of scene depth as the reference depth interval to be processed.

[0177] The scene depth range is defined by the maximum and minimum depths that a glasses-free 3D display device can output, specifically determined by the display characteristics of the device and the features of the 3D scenes it supports. For example, the scene depth range can be expressed as... , This represents the minimum depth value within the depth range of this scene. This represents the maximum depth value within the scene's depth range.

[0178] Secondly, obtain at least one depth gaze coordinate of the sample user's gaze in the reference depth range and the convergence and divergence angle of the sample user's gaze in the depth gaze coordinate.

[0179] For ease of distinction, users who pre-participate in the test depth range are referred to as sample users, and the 3D scene output by the naked-eye 3D display during the test is referred to as the sample 3D scene. In this application, multiple sample points can be collected from the reference depth range within the sample 3D scene, and the depth of each sample point is used as a depth gaze coordinate for the sample user. The gaze convergence angle when the sample user's gaze is fixed on the sample point is determined as the gaze convergence angle corresponding to the depth gaze coordinate of that sample point, resulting in at least one set of data pairs. Each data pair includes a depth gaze coordinate and the gaze convergence angle corresponding to that depth gaze coordinate.

[0180] Among them, the eye gaze of the sample users is focused on the first Depth gaze coordinates of each sample point Convergence and divergence angles at time It can be obtained using the following formula two for calculating the theoretical convergence angle:

[0181] (Formula 2);

[0182] in, This represents the average interpupillary distance.

[0183] Next, based on the gaze coordinates at each depth and the gaze convergence angles corresponding to each depth gaze coordinate, an angle mapping function is fitted to characterize the relationship between depth coordinates and convergence angles.

[0184] Since the theoretical convergence angle calculation function changes rapidly in the near distance range and tends to change gradually in the far distance range, in order to accurately represent the convergence angle of the line of sight, it is necessary to use a piecewise fitting method to determine the angle mapping function of each segment (i.e., depth coordinate interval). The final suitable depth coordinate intervals are the depth coordinate intervals required for piecewise fitting of the three-dimensional mapping function.

[0185] For example, the angle mapping function to be fitted can be expressed as the following formula three:

[0186] (Formula 3);

[0187] in, Indicates the angle of convergence and divergence. These are depth coordinates. These are the coefficients of a quadratic function. Based on each set of data pairs, these three coefficients can be obtained by fitting using the least squares method, thus obtaining the fitted angle mapping function. The details will not be elaborated further.

[0188] Then, based on at least one depth gaze coordinate and gaze convergence angle, the loss function value of the angle mapping function corresponding to the reference depth interval is determined.

[0189] For example, the root mean square error of the convergence angle calculated by the angle mapping function of the reference depth interval can be calculated using Formula 4, and the root mean square error of the support for each line of sight convergence angle calculated using Formula 2. :

[0190] (Formula 4);

[0191] in, This represents the total number of sampling points sampled within the reference depth range. From 1 to The natural number.

[0192] Finally, if the loss function value exceeds a set threshold, the reference depth interval is divided into two depth intervals, using the center point as the dividing point. Each of these two depth intervals is then used as the reference depth interval. The process returns to obtain at least one depth gaze coordinate and gaze convergence / divergence angle of the sample user's gaze within the reference depth interval, in order to fit the angle mapping function corresponding to the currently determined reference depth interval. If the loss function value does not exceed the set threshold, the reference depth interval is defined as a single depth coordinate interval.

[0193] Through the steps described above, the scene depth range that the naked-eye 3D display device can output can be divided into at least one depth coordinate interval. Based on the at least one depth coordinate interval that is suitable for dividing the scene depth range, a 3D mapping function corresponding to each depth coordinate interval can be fitted, as described above, and will not be repeated here.

[0194] It is understood that, in the above embodiments of this application, in order to more accurately determine the position information of the user's gaze point in the three-dimensional scene, after obtaining the position information, this application can also use a calibration function to calibrate the position information of the gaze point to obtain calibrated position information. Accordingly, the coarse positioning of the cursor in the previous embodiments is based on the calibrated position information to perform coarse positioning of the cursor.

[0195] The calibration function is a function fitted based on at least one calibration coordinate pair. Each calibration coordinate pair includes: the historical 3D coordinates corresponding to the user's historical gaze point, and the cursor's 3D coordinates corresponding to those historical 3D coordinates. The cursor's 3D coordinates are the historical target position point of the cursor corresponding to the historical gaze point. In other words, the cursor's 3D coordinates are the historical target 3D coordinates of the cursor determined based on the historical 3D coordinates; that is, the cursor's 3D coordinates are the cursor's 3D coordinates determined based on the historical 3D coordinates after coarse positioning and positioning adjustment (fine positioning).

[0196] Understandably, since the calibration function is fitted by combining the historical three-dimensional coordinates of the user's gaze points and the cursor position finally determined based on the historical three-dimensional coordinates, the calibration function is more in line with the user's gaze characteristics, thus making the position information corrected by the calibration function more accurate.

[0197] Furthermore, as the user's gaze changes, the calibration coordinate pairs can be continuously updated and the calibration function can be refitted. By calibrating the position information of the gaze point through the calibration function, it is equivalent to correcting the three-dimensional mapping function, thereby continuously improving the accuracy of the finally determined position information.

[0198] To facilitate understanding, let's take the location of the user's gaze point in the 3D scene as the target's 3D coordinates as an example. Figure 7 This illustration shows another flowchart of the cursor positioning method provided in this application. The method in this embodiment may include:

[0199] S701 tracks the user's gaze when the electronic device is in its first positioning state.

[0200] S702, in response to the user's gaze meeting the changing conditions, determines the target's three-dimensional coordinates in the three-dimensional scene output by the naked-eye 3D display device, based on the user's gaze point.

[0201] S703 uses a calibration function to calibrate the target's three-dimensional coordinates at the gaze point, obtaining the calibrated target's three-dimensional coordinates.

[0202] It should be noted that this embodiment is based on one case. If the electronic device does not have two positioning states, then after determining the target's three-dimensional coordinates corresponding to the user's gaze point, the target's three-dimensional coordinates can also be calibrated using a calibration function. The details will not be elaborated further.

[0203] S704, move the cursor in the 3D scene to the position point corresponding to the calibrated 3D coordinates of the target, and obtain the initial position point of the cursor.

[0204] S705, the system detects that the user's operation meets the switching conditions and switches the electronic device's positioning state from the first positioning state to the second positioning state.

[0205] S706, when the electronic device is in the second positioning state, the initial position of the cursor is adjusted based on the user's input operation to obtain the target position of the cursor.

[0206] It is understood that this application obtains at least one calibration coordinate pair periodically or irregularly to update the calibration coordinate pair, and refits the calibration function based on the updated calibration coordinate pair to improve the calibration accuracy of the calibration function, thereby improving the accuracy of the determined fixation point location information.

[0207] To improve the calibration accuracy of the calibration function, this application can also... Figure 8 The calibration function is constructed in this way. Figure 8 This illustration shows a schematic diagram of one implementation process for constructing the calibration function according to this application. The method in this embodiment may include:

[0208] S801, in response to a target location event of the cursor, obtain at least one calibration coordinate pair.

[0209] The target event is used to indicate that the user confirms that the target location of the cursor is accurately positioned.

[0210] It is understandable that users will only perform click, drag, or text insertion operations on the target location if they confirm that the target location is the accurate positioning point of the cursor. Therefore, the target event can be a click or drag operation on the target location of the cursor.

[0211] The calibration coordinate pair includes: the historical 3D coordinates of the user's gaze point in the 3D scene and the corresponding cursor 3D coordinates, where the cursor 3D coordinates are the coordinates of the cursor's target position point determined based on the historical 3D coordinates.

[0212] S802, based on at least one calibration coordinate pair obtained within the most recently set time period, fits a calibration function for converting the historical three-dimensional coordinates in the calibration coordinate pair into cursor three-dimensional coordinates.

[0213] For example, step S802 can be executed when the number of obtained calibration coordinate pairs reaches a set number; or step S802 can be executed when the update cycle of the calibration function is set and the update cycle is reached, without any specific restrictions.

[0214] In this application, the form of the calibration function can also be varied, and there is no limitation on it.

[0215] For example, the calibration function can be a fitted polynomial function, and each calibration coordinate pair can include a cursor's three-dimensional coordinates. A historical 3D coordinate of a historical viewpoint in a 3D scene .in, It can be regarded as the actual cursor positioning point, As input observations, based on each calibration coordinate pair, a polynomial function can be fitted as shown in Formula 5 below:

[0216] (Formula 5);

[0217] in, The 3D coordinates of the gaze point (such as historical 3D coordinates) are calibrated using a calibration function. For polynomial basis function mapping, and These are the self-calibration coefficient matrix and the offset term, respectively. During the function fitting process... The value can be any historical three-dimensional coordinate. , The value can be Corresponding cursor 3D coordinates Based on this, the least squares fitting method can be used to obtain... and Finally, the calibration function represented by Formula 5 was determined.

[0218] Accordingly, after determining the target's three-dimensional coordinates corresponding to the user's gaze point, the target's three-dimensional function is used as the formula in Formula 5. With the input values, the calibrated three-dimensional coordinates of the target can be calculated.

[0219] If the location information of the gaze point is a three-dimensional coordinate range, Formula 5 can be used to calibrate the three-dimensional coordinates of each boundary point in the three-dimensional coordinate range to obtain the calibrated three-dimensional coordinate range.

[0220] Understandably, in order to further improve the calibration accuracy of the calibration function, this application can fit different verification functions for different depth coordinate intervals to obtain different calibration functions corresponding to different depth coordinate intervals. Accordingly, after determining the target three-dimensional coordinates or three-dimensional coordinate range of the gaze point, this application can determine the calibration function corresponding to the target depth coordinate interval based on the target depth coordinate interval corresponding to the target three-dimensional mapping function, and use the calibration function to calibrate the target three-dimensional coordinates or three-dimensional coordinate range.

[0221] The process of fitting the verification function is the same for each depth coordinate interval, except that the calibration coordinate pairs corresponding to different depth coordinate intervals are different. Specifically, the depth coordinates in the cursor's 3D coordinates and / or the historical 3D coordinates of the historical gaze point belong to that depth coordinate interval.

[0222] In this application, there are several possible specific implementations for obtaining the at least one calibration coordinate pair.

[0223] For example, in one possible scenario, upon detecting the target event, the current target position of the cursor is determined as the cursor's three-dimensional coordinates, and the cursor's three-dimensional coordinates and their corresponding historical three-dimensional coordinates are determined as a calibration coordinate pair. Here, the historical three-dimensional coordinates corresponding to the cursor's three-dimensional coordinates are the target three-dimensional coordinates of the gaze point upon which the target position was determined.

[0224] Understandably, if only one calibration coordinate pair can be obtained each time a target event is detected, while accurately fitting the calibration function may require a large number of calibration coordinate pairs, then it will take a long time to obtain a sufficient number of calibration coordinate pairs. In many cases, to obtain a sufficient number of calibration coordinate pairs, it may be necessary to prompt the user to assist in calibrating the accurate cursor position point to obtain the calibration coordinate pairs, which will inevitably increase the complexity of fitting or updating the calibration function.

[0225] Therefore, to reduce the complexity of fitting or updating the calibration function, this application also provides another method for obtaining calibration sample pairs, such as... Figure 9 As shown. Figure 9 The diagram illustrates a process for obtaining calibration coordinate pairs according to this application, which may include:

[0226] S901, in response to the existence of a target at the cursor's target position, obtains the first coordinate sequence and the second coordinate sequence.

[0227] The first coordinate sequence includes: at least one gaze point three-dimensional coordinate. The second coordinate sequence includes: cursor three-dimensional coordinates corresponding to the at least one gaze point three-dimensional coordinate.

[0228] Wherein, the at least one gaze point's three-dimensional coordinates are the three-dimensional coordinates of at least one historical gaze point of the user's gaze within the target duration range corresponding to the current moment in the three-dimensional scene. The cursor's three-dimensional coordinates corresponding to the gaze point's three-dimensional coordinates are the three-dimensional coordinates of the cursor's target position point determined based on the gaze point's three-dimensional coordinates.

[0229] For example, based on the time when the target event is detected. Based on this, obtain the time window. Using the target 3D coordinates of the inner fixation point as the fixation point's 3D coordinates, a first coordinate sequence consisting of at least one fixation point's 3D coordinates is obtained. The preset duration can be set according to specific needs. Simultaneously, the coordinates of each target location point within this time window, determined based on the 3D coordinates of the gaze point, are used as the cursor's 3D coordinates, thus obtaining at least one second coordinate sequence composed of the cursor's 3D coordinates.

[0230] S902, determine the trajectory similarity between the first coordinate trajectory corresponding to the first coordinate sequence and the second coordinate trajectory corresponding to the second coordinate sequence.

[0231] There are multiple ways to determine the trajectory similarity between two coordinate trajectories, and this application does not impose any restrictions on this.

[0232] For example, trajectory similarity can be the direction cosine similarity between the first coordinate sequence and the second coordinate sequence. It can be calculated using the following formula six:

[0233] (Formula 6);

[0234] in, Represents the first coordinate sequence of the first coordinate sequence Three-dimensional coordinates of a gaze point Represents the second coordinate sequence of the first... Three-dimensional coordinates of the cursor From 1 to natural numbers, This represents the total number of three-dimensional coordinates of the gaze point in the first coordinate sequence, and also the total number of three-dimensional coordinates of the cursor in the second coordinate sequence. Indicates the trajectory of the first coordinate at the th position. Segment directional increment, Indicates the second coordinate trajectory at the th The direction increment of the segment.

[0235] For example, trajectory similarity can also be a measure of the curvature difference between the first coordinate trajectory and the second coordinate trajectory. This can be expressed as Formula Seven:

[0236] (Formula 7);

[0237] in, and The first coordinate trajectory and the second coordinate trajectory are respectively in the th... The curvature of a point represents the degree of curvature of the trajectory. This is a scale parameter for curvature differences. The larger the value, the less sensitive it is to curvature differences, and the smaller the value, the more stringent it is to local curvature differences. Its value can be preset according to actual needs.

[0238] In practical applications, trajectory similarity can also be the sum of multiple similarity measures, such as trajectory similarity. The weighted sum of the directional cosine similarity and curvature difference measures can be expressed as the following formula:

[0239] (Formula 8);

[0240] in, The weighting coefficients are set.

[0241] S903, in response to the trajectory similarity exceeding a first threshold, a trajectory transformation function for converting the first coordinate trajectory into a corresponding second coordinate trajectory is determined.

[0242] Understandably, the more similar the first coordinate trajectory is to the second coordinate trajectory, the more accurate the 3D coordinates of each gaze point on the first coordinate sequence calculated using the 3D mapping function are. Only accurate gaze point 3D coordinates can be used to fit or calibrate the function. Based on this, this application will only select gaze point 3D coordinates to form calibration coordinate pairs based on the first coordinate sequence if the trajectory similarity exceeds a first threshold.

[0243] To improve the accuracy of the constructed calibration function, this application does not use the coordinates of each gaze point in the first coordinate sequence as historical 3D coordinates in the calibration coordinate pair, but requires further filtering. Therefore, this application needs to determine a trajectory transformation function for converting the first coordinate trajectory into the second coordinate trajectory.

[0244] The trajectory transformation function can be determined by fitting the three-dimensional coordinates on the two coordinate trajectories.

[0245] For example, this application can first transform the first coordinate trajectory into the second coordinate trajectory through a linear transformation. Correspondingly, the three-dimensional coordinates of the gaze point and the three-dimensional coordinates of the cursor can have the following transformation relationship as shown in Formula Nine:

[0246] (Formula Nine);

[0247] in, This represents the scaling factor for the three coordinate axes. This represents the translation vector along the three coordinate axes. This indicates element-wise multiplication.

[0248] Solve using least squares: Based on this formula, we can respectively... Taking partial derivatives yields a closed-form solution, or iterative least squares can be used to obtain the solution. and Based on the solution and Formula 9 can then be used as a trajectory transformation function.

[0249] S904 uses a trajectory transformation function to transform the three-dimensional coordinates of the gaze point, and obtains the reference three-dimensional coordinates corresponding to the three-dimensional coordinates of the gaze point.

[0250] For example, for a gaze point's three-dimensional coordinates The corresponding reference three-dimensional coordinates Substituting into Formula 9, we get: .

[0251] S905, if the distance between the reference three-dimensional coordinates corresponding to the gaze point three-dimensional coordinates and the cursor three-dimensional coordinates corresponding to the gaze point three-dimensional coordinates is less than the second threshold, the gaze point three-dimensional coordinates are used as historical three-dimensional coordinates and form a calibration coordinate pair with the cursor three-dimensional coordinates corresponding to the gaze point three-dimensional coordinates.

[0252] For example, the three-dimensional coordinates of the gaze point. Corresponding reference 3D coordinates The cursor's three-dimensional coordinates corresponding to the gaze point's three-dimensional coordinates Distance between It can be represented as: .

[0253] If the distance is less than the set second threshold, it indicates that the accuracy of the three-dimensional coordinates of the gaze point is relatively high, and the three-dimensional coordinates of the gaze point and its corresponding cursor three-dimensional coordinates can be used to form a calibration coordinate pair.

[0254] For ease of understanding Figure 9 Implementation of the embodiments, in combination Figure 10 A brief explanation is provided.

[0255] Figure 10 An example procedure for selecting calibration coordinate pairs is shown.

[0256] exist Figure 10 The horizontal and vertical axes in the three graphs represent different coordinate axes.

[0257] exist Figure 10 The first image from left to right is an example comparing the trajectories of a first coordinate sequence composed of the three-dimensional coordinates of the gaze point and a second coordinate sequence composed of the three-dimensional coordinates of the cursor. Based on the first image, this application performs linear transformations on the three-dimensional coordinates of each gaze point in the first coordinate sequence, such as using Formula 9 for translation and scaling, to align the trajectories of the first and second coordinate sequences. Figure 10 As shown in the second figure. Based on this, this application can select the calibration coordinate pair based on this step S905, such as... Figure 10 As shown, in this application, calibration point pairs can be selected based on the distance between the reference three-dimensional coordinates after the gaze point three-dimensional coordinate transformation and the cursor three-dimensional coordinate point.

[0258] Understandably, through Figure 9 This method generates calibration coordinate pairs without requiring users to calibrate them separately, reducing user operations and improving the ease of obtaining calibration coordinate pairs.

[0259] This application also provides an electronic device in its embodiments. For example... Figure 11 As shown, a schematic diagram of the composition structure of the electronic device is presented. The electronic device includes at least a naked-eye 3D display device 1101, a data acquisition device 1102, and a processor 1103.

[0260] Among them, the data acquisition device 1102 is used to track the user's line of sight;

[0261] The processor 1103 is configured to, in response to the user's gaze meeting a change condition, determine the position information of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device 1101; based on the position information, perform coarse positioning of the cursor in the three-dimensional scene to obtain the initial position point of the cursor; and adjust the initial position point of the cursor based on the user's input operation to obtain the target position point of the cursor.

[0262] The data acquisition device can be any device that can track the user's gaze, such as an eye tracker, and there are no specific restrictions.

[0263] In this application, the electronic device may also include a memory 1104 for storing programs required for the processor to perform operations.

[0264] Understandably, the electronic device may also include input units such as a mouse or joystick.

[0265] Of course, the electronic device can also have more than Figure 11 There are no restrictions on the number of components, whether more or fewer.

[0266] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the cursor positioning methods provided in this application.

[0267] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the cursor positioning methods provided in this application.

[0268] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0269] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0270] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0271] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A cursor positioning method, comprising: Track the user's gaze; In response to the user's gaze meeting the change condition, the position information of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device is determined; Based on the location point information, the cursor in the three-dimensional scene is coarsely located to obtain the initial position point of the cursor; The initial position of the cursor is adjusted based on the user's input to obtain the target position of the cursor.

2. The cursor positioning method according to claim 1, wherein determining the position information of the user's gaze point in the three-dimensional scene output by the naked-eye 3D display device includes: Determine the range of three-dimensional coordinates of the user's gaze point in the three-dimensional scene output by the naked-eye 3D display device; The step of coarsely locating the cursor in the 3D scene based on the location point information to obtain the initial position point of the cursor includes: Using the three-dimensional coordinate range as the coarse positioning area for the cursor, the cursor in the three-dimensional scene is moved into the coarse positioning area to obtain the initial position point of the cursor; or, The determination of the position information of the user's gaze point in the 3D scene output by the naked-eye 3D display device includes: Determine the target's 3D coordinates in the 3D scene output by the naked-eye 3D display device, based on the user's gaze point; The step of coarsely locating the cursor in the 3D scene based on the location point information to obtain the initial position point of the cursor includes: The cursor in the three-dimensional scene is moved to the position point corresponding to the three-dimensional coordinates of the target to obtain the initial position point of the cursor.

3. The cursor positioning method according to claim 2, wherein adjusting the initial position of the cursor based on user input includes at least one of the following: Based on the input operation and the initial position of the cursor, adjust the position of the cursor; Based on the input operation and the initial position of the cursor, adjust the position of the cursor within the coarse positioning area; Based on the input operation and the initial position of the cursor, the position of the cursor is adjusted within the set range of the target's three-dimensional coordinates.

4. The cursor positioning method according to claim 1, wherein tracking the user's gaze includes: When the electronic device is in its first positioning state, it tracks the user's gaze; The cursor positioning method further includes: detecting that the user's operation meets the switching conditions, and switching the positioning state of the electronic device from the first positioning state to the second positioning state; The step of adjusting the initial position of the cursor based on user input includes: When the electronic device is in the second positioning state, the initial position of the cursor is adjusted based on the user's input.

5. The cursor positioning method according to claim 4, wherein the operation action satisfies the switching condition, including: The magnitude of the operation is greater than a set threshold and / or the duration of the operation exceeds a duration threshold. The cursor positioning method further includes: when the electronic device is in a second positioning state, if no input operation from the user is detected within a specified period of time, switching the positioning state of the electronic device from the second positioning state to the first positioning state.

6. The cursor positioning method according to claim 2, wherein determining the position information of the user's gaze point in the three-dimensional scene output by the naked-eye 3D display device includes: Determine the two-dimensional coordinates of the user's gaze point on the naked-eye 3D display device; Determine the target three-dimensional mapping function based on the three-dimensional mapping function corresponding to at least one depth coordinate interval; Based on the two-dimensional coordinates, the position information of the user's gaze point in the three-dimensional scene output by the naked-eye three-dimensional display device is determined using the target three-dimensional mapping function.

7. The cursor positioning method according to claim 6, wherein determining the target three-dimensional mapping function based on a three-dimensional mapping function corresponding to at least one depth coordinate interval includes: From the three-dimensional mapping functions corresponding to at least one depth coordinate interval, determine the target three-dimensional mapping function that is currently available as a mapping function; The cursor positioning method also includes: If the depth value in the target three-dimensional coordinates or the depth range in the three-dimensional coordinate range exceeds the target depth coordinate interval corresponding to the target three-dimensional mapping function, determine the target interval boundary in the target depth coordinate interval that is closest to the three-dimensional coordinates or the three-dimensional coordinate range, and increment the number of times the target interval boundary has exceeded the boundary by one; If the number of times the boundary of the target interval is crossed exceeds a set threshold, a candidate depth coordinate interval that is outside the target depth coordinate interval and adjacent to the boundary of the target interval is determined. The three-dimensional mapping function corresponding to the candidate depth coordinate interval is determined as the currently available mapping function.

8. The cursor positioning method according to claim 6 further includes: The position information of the gaze point is calibrated using a calibration function to obtain calibrated position information. The calibration function is a function fitted based on at least one calibration coordinate pair. The calibration coordinate pair includes: the historical three-dimensional coordinates corresponding to the historical gaze point of the user's gaze, and the cursor three-dimensional coordinates corresponding to the historical three-dimensional coordinates. The cursor three-dimensional coordinates are the historical target position point of the cursor corresponding to the historical gaze point. The cursor positioning method further includes: In response to a target event indicating the presence of a target location point of the cursor, at least one calibration coordinate pair is obtained, wherein the target event is used to indicate that the user confirms that the target location point of the cursor is accurately positioned. Based on at least one of the calibration coordinate pairs obtained within the most recently set time period, a calibration function is fitted to convert the historical three-dimensional coordinates in the calibration coordinate pairs into the cursor three-dimensional coordinates.

9. The cursor positioning method according to claim 8, wherein obtaining at least one calibration coordinate pair in response to a target location event of the cursor includes: In response to a target location event of the cursor, a first coordinate sequence and a second coordinate sequence are obtained. The first coordinate sequence includes at least one gaze point three-dimensional coordinates, and the second coordinate sequence includes cursor three-dimensional coordinates corresponding to the at least one gaze point three-dimensional coordinates. The at least one gaze point three-dimensional coordinates are the three-dimensional coordinates of at least one historical gaze point of the user's line of sight in the three-dimensional scene within the target duration range corresponding to the current moment. Determine the trajectory similarity between the first coordinate trajectory corresponding to the first coordinate sequence and the second coordinate trajectory corresponding to the second coordinate sequence; In response to the trajectory similarity exceeding a first threshold, a trajectory transformation function is determined for converting the first coordinate trajectory into a corresponding second coordinate trajectory; The trajectory conversion function is used to convert the three-dimensional coordinates of the gaze point to obtain the reference three-dimensional coordinates corresponding to the three-dimensional coordinates of the gaze point; If the distance between the reference 3D coordinates corresponding to the gaze point 3D coordinates and the cursor 3D coordinates corresponding to the gaze point 3D coordinates is less than a second threshold, the gaze point 3D coordinates are used as historical 3D coordinates and form a calibration coordinate pair with the cursor 3D coordinates corresponding to the gaze point 3D coordinates.

10. An electronic device, comprising: Glasses-free 3D display device, acquisition device, and processor; The acquisition device is used to track the user's line of sight; The processor is configured to, in response to a change in the user's gaze, determine the position information of the user's gaze point in the three-dimensional scene output by the naked-eye 3D display device; based on the position information, perform coarse positioning of the cursor in the three-dimensional scene to obtain the initial position point of the cursor; and adjust the initial position point of the cursor based on the user's input operation to obtain the target position point of the cursor.

Images