A target prediction method, device, apparatus, and storage medium

Abstract

The application discloses a target prediction method, device and equipment and a storage medium. The method comprises the following steps: if a target selection request of a user in a three-dimensional scene is detected, the motion amplitude of the eyes and / or the head of the user when the user selects a target is determined; the type of a target prediction model is determined, and the target prediction model is constructed according to the type, the motion amplitude, an input mechanism associated with the target selection request and the target width of a target to be selected; the probability that the target to be selected belongs to a target selected by the user is determined according to the target prediction model and the attribute information of the target to be selected; and the target selected by the user is predicted from the target to be selected according to the probability. The technical scheme of the application can effectively utilize the relevant information of the eyes and the head of the user, predict the target selected by the user and improve the accuracy of target prediction.

Description

Technical Field

[0001] This invention relates to the field of augmented reality, and more particularly to a target prediction method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of Augmented Reality (AR) technology, the most common and basic application is for users to make their own actions in augmented reality 3D scenes.

[0003] Generally, existing AR headsets, such as Magic Leap and HoloLens 2, allow users to select virtual objects using handheld controllers or mid-air gestures. However, in some situations, one or both of a user's hands are busy with other tasks, such as carrying tools or other items, making it impossible for the user to interact with the AR headset using controllers or gestures.

[0004] How to effectively utilize other physical characteristics of users to identify the targets selected by users, achieve more accurate and effective target prediction, and improve the user experience is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a target prediction method, apparatus, device, and storage medium that can effectively utilize relevant information from the user's eyes and head to predict the target object selected by the user, thereby improving the accuracy of target prediction.

[0006] According to one aspect of the present invention, a target prediction method is provided, comprising:

[0007] If a user's target selection request in a three-dimensional scene is detected, the range of motion of the user's eyes and / or head when making the target selection is determined; the range of motion represents the change in the user's line of sight and / or head direction from the start of target selection to the end of target selection.

[0008] Determine the type of target prediction model, and construct the target prediction model based on the type, the motion amplitude, the input mechanism associated with the target selection request, and the target width of the candidate target.

[0009] Based on the target prediction model and the attribute information of the candidate targets, the probability that the candidate target belongs to the target selected by the user is determined;

[0010] Based on the probability, the target object selected by the user is predicted from the candidate target objects.

[0011] According to another aspect of the present invention, a target prediction device is provided, comprising:

[0012] An amplitude determination module is used to determine the amplitude of eye and / or head movement when a user makes a target selection request in a three-dimensional scene if the user's target selection request is detected; the amplitude of movement represents the change in the user's line of sight and / or head direction from the start of target selection to the end of target selection.

[0013] A construction module is used to determine the type of the target prediction model and construct the target prediction model based on the type, the motion amplitude, the input mechanism associated with the target selection request, and the target width of the candidate target.

[0014] The probability determination module is used to determine the probability that a candidate target belongs to the target selected by the user, based on the target prediction model and the attribute information of the candidate target.

[0015] The prediction module is used to predict the target object selected by the user from the candidate target objects based on the probability.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the target prediction method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the target prediction method according to any embodiment of the present invention.

[0021] The technical solution of this invention, if a user's target selection request in a 3D scene is detected, determines the range of motion of the user's eyes and / or head when making the target selection; determines the type of the target prediction model, and constructs a target prediction model based on the type, the range of motion, the input mechanism associated with the target selection request, and the target width of the candidate target; determines the probability that the candidate target belongs to the target selected by the user based on the target prediction model and the attribute information of the candidate target; and predicts the target selected by the user from the candidate target based on the probability. By effectively utilizing relevant information of the user's eyes and head to predict the target selected by the user, the accuracy of target prediction can be improved, enhancing the user experience when operating in a VR scene.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a target prediction method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a structural block diagram of a target prediction device provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," "target," "candidate," "alternative," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It should be noted that existing AR head-mounted displays, such as Magic Leap and HoloLens 2, allow users to select virtual objects using handheld controllers or mid-air gestures. However, in some situations, one or both of a user's hands are busy with other tasks, such as carrying tools or other items, making it impossible for the user to interact with the AR head-mounted display using controllers or gestures. This invention considers that eye gaze is typically rapid, allowing for the scanning of large areas with minimal physical effort. Eye gaze is also lightweight and can indicate the user's selection intent. Simultaneously, head movements can also represent the user's selection intent to a certain extent. The technical solution of this invention can determine the user's gaze and head selection point based on interface features (such as the size of the target object and its distance from the user), and further predict the target selected by the user in the AR system, effectively improving the accuracy of target selection based on the user's eyes and / or head in AR scenarios. The specific implementation scheme for target prediction will be described in detail in subsequent embodiments.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a target prediction method provided in Embodiment 1 of the present invention. This embodiment is applicable to responding to target selection requests from users wearing AR (Augmented Reality) devices and predicting the target object selected by the user. It is particularly suitable for situations where users are playing games or performing operations in an AR scene and selecting a target object from multiple objects in a three-dimensional environment. This method can be executed by a target prediction device, which can be implemented in hardware and / or software and can be configured in electronic devices or AR-related devices. Figure 1 As shown, the target prediction method includes:

[0032] S101. If a user's target selection request in a three-dimensional scene is detected, determine the range of motion of the user's eyes and / or head when making the target selection.

[0033] Among them, the target selection request refers to the user's request to select a target object from multiple objects in the three-dimensional environment. The motion amplitude characterizes the changes in the user's line of sight and / or head direction from the start of target selection to the end of target selection.

[0034] Optionally, based on a preset input mechanism, the system can detect whether the user has a corresponding input behavior. If so, it can be determined that the user has issued a target selection request in the 3D scene. For example, if the preset input mechanism is blinking, i.e., blinking a preset number of times consecutively, then when the user blinks continuously for the preset number of times, it can be determined that the user has issued a target selection request. As another example, if the preset input mechanism is hand pinching, i.e., the user pinches their thumb and forefinger together, then when the user's thumb and forefinger are detected pinching together, it can be determined that the user has issued a target selection request.

[0035] Optionally, after detecting a user's target selection request in a 3D scene, the range of motion of the user's eyes and / or head when making target selection is determined, including: prompting the user to start target selection and acquiring the movement trajectory of the user's eyes and / or head; and determining the range of motion of the user's eyes and / or head when making target selection based on the movement trajectory of the user's eyes and / or head.

[0036] For example, after detecting a user's target selection request in a 3D scene, the system can prompt the user to start the selection process and monitor the movement trajectory of the user's eyes and / or head, and determine the range of motion of the user's eyes and / or head based on the movement trajectory.

[0037] Optionally, the positions of the eye's line of sight in the 3D scene at the start of target selection and at the end of target selection can be determined separately, and the distance between the two positions can be determined as the range of eye movement. Alternatively, the rotation angle of the eye's line of sight at the start of target selection and at the end of target selection can be determined as the range of eye movement.

[0038] Optionally, the parallel direction of the user's forehead can be determined as the head pointing direction, and the landing point positions of the head pointing direction in the 3D scene at the start of target selection and the end of target selection can be determined respectively, and the distance between the two landing point positions can be determined as the head movement amplitude.

[0039] S102. Determine the type of target prediction model, and construct the target prediction model based on the type, motion amplitude, input mechanism associated with the target selection request, and target width of the candidate target.

[0040] The type of target prediction model can characterize whether the prediction is based on a single or multiple body features of the user. The target prediction model can be a unimodal or multimodal model. The input mechanism associated with the target selection request can be blinking or hand pinching. The candidate target refers to an object in the 3D scene that can be selected by the user; this object can be virtual or a real object in the user's actual scene. The target width of each object can be preset, for example, set to the maximum width of the actual size of each object.

[0041] It should be noted that in general use cases, the target width W is known. For example, in a menu bar, it is usually composed of some UI components, which usually have a fixed size. Therefore, the target width W can be obtained in two different ways: (1) The target width of each target object needs to be set as a fixed value in advance. (2) The target width W of the target object can be detected in real time to determine the width of the target object.

[0042] Optionally, the type of the target prediction model is determined, including: if the determined motion amplitude is the motion amplitude of the eyes or head, then the type of the target prediction model is determined to be a single-modal model that predicts the target through the user's eye line of sight or head pointing direction; if the determined motion amplitude is the motion amplitude of both the eyes and the head, then the type of the target prediction model is determined to be a multi-modal model that predicts the target through the user's eye line of sight and head pointing direction.

[0043] Optionally, a target prediction model is constructed based on the type, motion amplitude, input mechanism associated with the target selection request, and target width of the candidate target, including: determining the mean and variance associated with the target prediction model based on the type, motion amplitude, input mechanism associated with the target selection request, and target width of the candidate target; and constructing a likelihood function and target prediction model based on the mean and variance.

[0044] Optionally, for each candidate target, the mean and variance of the target prediction model can be determined based on its type, motion amplitude, the input mechanism associated with the target selection request, and the target width of the candidate target.

[0045] For example, if the target prediction model is a unimodal model, the mean μ and variance ∑ associated with the target prediction model can be determined using the following formula:

[0046]

[0047] Where a, b, c, d, e, f, and g are preset constant parameters. A represents the motion amplitude, W represents the target width of the candidate target, μ represents the mean of the target prediction model association, and ∑ represents the variance of the target prediction model association.

[0048] Preferably, through verification by a large number of data experiments, the preset values ​​of the constant parameters can be: a = 0.1243, b = 0.4115, e = -0.1541, f = 0.0084 and g = -0.1702.

[0049] Furthermore, experimental verification revealed that different input mechanisms have a significant impact on the preset constant parameters c and d. Therefore, preferably, when the input mechanism is hand pinching, the preset constant parameters c and d can be: c = 0.0078, d = 0.371; when the input mechanism is blinking, the preset constant parameters c and d can be: c = 0.0064, d = 0.4904.

[0050] For example, the constructed likelihood function L(μ,Σx) can be expressed as:

[0051]

[0052] Where μ represents the mean of the target prediction model association, ∑ represents the variance of the target prediction model association, and x represents the attribute information of the candidate target. For example, it can be the location information or identification information of the candidate target.

[0053] Optionally, if the target prediction model is a unimodal model, the preset constant parameters include: the average probability P(T) of each target being selected, and the probability P(G) of the eye or head indicating direction; if the target prediction model is a multimodal model, the preset constant parameters include: the average probability P(T) of each target being selected, the probability P(G) of the eye or head indicating direction, and the conditional probability P(H|G) of the head indicating direction.

[0054] For example, if the target prediction model is a single-modal model, it can be represented as:

[0055]

[0056] Where P(T|G) represents the probability that the target object T is selected under the user's eye or head pointing direction G. P(T) is the average probability that each target object is selected; for example, if the number of target objects is n, then the value of P(T) can be set to 1 / n. P(G) represents the probability of the eye or head pointing direction. P(G|T) is the probability determined based on the likelihood function that the user's eye or head pointing direction is G given that the selected target object is T.

[0057] For example, if the target prediction model is a multimodal model, it can be represented as:

[0058]

[0059] Here, P(T|G,H) represents the probability that the target object T is selected under the common direction G indicated by the user's eyes and head. P(G,H|T) represents the probability, determined by the likelihood function, that the user's eyes and head are pointing in direction G given the selected target object T. P(T) represents the average probability of each target object being selected; for example, if the number of target objects is n, the value of P(T) can be set to 1 / n. P(G) represents the probability of the direction indicated by the eyes or head. P(H|G) represents the conditional probability of the direction indicated by the head, that is, the conditional probability that the head points in direction H given that the eyes are pointing in direction (G). Generally, within a specific three-dimensional space, the value of P(H|G) can be preset as a constant.

[0060] S103. Based on the target prediction model and the attribute information of the candidate target, determine the probability that the candidate target belongs to the target selected by the user.

[0061] Among them, the attribute information of the candidate target object can be, for example, the location information or identification information of the candidate target object.

[0062] Optionally, based on relevant VR devices, the position information of the candidate target objects in the 3D scene can be collected to determine the attribute information of the candidate target objects. Generally, the position information of each candidate target object is different. Alternatively, a uniquely identifiable code can be set for each candidate target object in the 3D scene, and the code information can be determined as the attribute information of the candidate target object.

[0063] Optionally, based on the target prediction model and the attribute information of the candidate targets, the probability that a candidate target belongs to the target selected by the user is determined, including: for each candidate target, substituting the mean and variance associated with the target prediction model, as well as the attribute information of the candidate target, into the likelihood function to determine the conditional probability value associated with the target prediction model; inputting the conditional probability value and the preset constant parameters of the target prediction model into the target prediction model, and determining the probability that a candidate target belongs to the target selected by the user based on the output results.

[0064] For example, based on the likelihood function formula described above in this embodiment, for each candidate target, the mean and variance associated with the target prediction model, as well as the attribute information of the candidate target, can be substituted into the likelihood function, and the result can be determined as the conditional probability value associated with the target prediction model.

[0065] Specifically, if the target prediction model is a unimodal model, the result of the likelihood function can be used as the conditional probability value associated with the target prediction model, that is, to determine the value of P(G|T); if the target prediction model is a multimodal model, the result of the likelihood function can be used as the conditional probability value associated with the target prediction model, that is, to determine the value of P(G,H|T).

[0066] Optionally, if the target prediction model is a single-mode model, the conditional probability value and the preset constant parameters corresponding to the single-mode model can be substituted into the single-mode model, and the output of the model can be used as the probability that the candidate target belongs to the target selected by the user.

[0067] Optionally, if the target prediction model is a multimodal model, the conditional probability value and the preset constant parameters corresponding to the multimodal model can be substituted into the multimodal model, and the output of the model can be used as the probability that the candidate target belongs to the target selected by the user.

[0068] S104. Based on probability, predict the target object selected by the user from the candidate target objects.

[0069] Optionally, based on probability, the target object selected by the user can be predicted from the candidate targets. This includes determining the probability that each candidate target belongs to the target object selected by the user, and identifying the candidate target with the highest probability as the target object selected by the user. In this way, a target prediction scheme is implemented that predicts the user's selected target based on the user's eye and / or head movements.

[0070] The technical solution of this invention, if a user's target selection request in a 3D scene is detected, determines the range of motion of the user's eyes and / or head when making the target selection; determines the type of the target prediction model, and constructs a target prediction model based on the type, the range of motion, the input mechanism associated with the target selection request, and the target width of the candidate target; determines the probability that the candidate target belongs to the target selected by the user based on the target prediction model and the attribute information of the candidate target; and predicts the target selected by the user from the candidate target based on the probability. By effectively utilizing relevant information of the user's eyes and head to predict the target selected by the user, the accuracy of target prediction can be improved, enhancing the user experience when operating in a VR scene.

[0071] It should be noted that the target prediction model provided by this invention is based on Bayesian theory, so both the unimodal and multimodal models are probabilistic formulas. Since in Bayesian theory, parameters other than the likelihood function can usually be considered constants (e.g., P(T) and P(G) in the unimodal model), a crucial part is determining the likelihood functions of these two models (e.g., P(G|T) in the unimodal model). Furthermore, based on extensive experimental verification, the distribution of eye or head landing points generated when a user selects a target can be determined, and it basically conforms to a bivariate normal distribution. Therefore, this invention can directly regard the bivariate normal distribution generated by the landing point as the likelihood function; that is, the likelihood function provided by this invention is the likelihood function of the bivariate normal distribution. However, the likelihood functions corresponding to other distributions can also be used if their probabilities can be calculated from the parameters given by the Bayesian formula; this invention does not impose any restrictions on this.

[0072] It's important to note that the pre-defined constant parameters in unimodal and multimodal models can be considered constants because: T(targets) is a set of targets containing many objects (t1, t2, ..., tn). The target prediction model needs to calculate which object is the user's desired target. Taking a unimodal model as an example, we need to calculate the probability of each object being the target using the unimodal model's formula. The object with the highest probability will be predicted as the user's desired target. When comparing the calculated probabilities using the same formula, the denominator remains unchanged in Bayesian theory, so it doesn't affect the final result; therefore, the denominator can be considered a constant. In other words, only the numerator determines the probability, and the only factor in the numerator that affects the probability change is the value of the likelihood function, i.e., P(G|T) in the unimodal model.

[0073] Example 2

[0074] Figure 2 This is a structural block diagram of a target prediction device provided in Embodiment 2 of the present invention. The target prediction device provided in this embodiment is applicable to responding to target selection requests from users wearing AR (Augmented Reality) devices and predicting the target object selected by the user. It is particularly suitable for users playing games or performing operations in AR scenes, selecting a target object from multiple objects in a three-dimensional environment. This target prediction device can be implemented in hardware and / or software and configured in AR-related devices. Figure 2 As shown, the device specifically includes: an amplitude determination module 201, a construction module 202, a probability determination module 203, and a prediction module 204, wherein,

[0075] The amplitude determination module 201 is used to determine the amplitude of eye and / or head movement when the user makes a target selection request in a three-dimensional scene if a target selection request is detected; the amplitude of movement represents the change in the user's eye line of sight and / or head pointing direction from the start of target selection to the end of target selection.

[0076] The construction module 202 is used to determine the type of the target prediction model and construct the target prediction model based on the type, the motion amplitude, the input mechanism associated with the target selection request, and the target width of the candidate target.

[0077] The probability determination module 203 is used to determine the probability that the candidate target belongs to the target selected by the user based on the target prediction model and the attribute information of the candidate target;

[0078] The prediction module 204 is used to predict the target object selected by the user from the candidate target objects based on the probability.

[0079] The technical solution of this invention, if a user's target selection request in a 3D scene is detected, determines the range of motion of the user's eyes and / or head when making the target selection; determines the type of the target prediction model, and constructs a target prediction model based on the type, the range of motion, the input mechanism associated with the target selection request, and the target width of the candidate target; determines the probability that the candidate target belongs to the target selected by the user based on the target prediction model and the attribute information of the candidate target; and predicts the target selected by the user from the candidate target based on the probability. By effectively utilizing relevant information of the user's eyes and head to predict the target selected by the user, the accuracy of target prediction can be improved, enhancing the user experience when operating in a VR scene.

[0080] Furthermore, the amplitude determination module 201 is specifically used for:

[0081] Prompt the user to begin target selection and acquire the movement trajectory of the user's eyes and / or head;

[0082] Based on the movement trajectory of the user's eyes and / or head, determine the range of eye and / or head movement when the user selects a target.

[0083] Furthermore, module 202 is specifically used for:

[0084] If the determined motion amplitude is the motion amplitude of the eyes or head, then the type of the target prediction model is determined to be a single-modal model that predicts the target based on the user's line of sight or the direction indicated by the head.

[0085] If the determined range of motion is the range of motion of the eyes and head, then the type of target prediction model is determined to be a multimodal model that predicts targets based on the user's line of sight and head direction.

[0086] Furthermore, building module 202 is also used for:

[0087] Based on the type, the amplitude of motion, the input mechanism associated with the target selection request, and the target width of the candidate target, determine the mean and variance associated with the target prediction model;

[0088] Based on the mean and variance, a likelihood function and a target prediction model are constructed.

[0089] Furthermore, the probability determination module 203 is also used for:

[0090] For each candidate target, the mean and variance of the target prediction model, along with the attribute information of the candidate target, are substituted into the likelihood function to determine the conditional probability value associated with the target prediction model.

[0091] The conditional probability value and the preset constant parameters of the target prediction model are input into the target prediction model, and the probability that the candidate target belongs to the target selected by the user is determined based on the output result.

[0092] Furthermore, among them,

[0093] If the target prediction model is a single-modal model, the preset constant parameters include: the average probability of each target being selected, and the probability of the eye or head indicating the direction;

[0094] If the target prediction model is a multimodal model, the preset constant parameters include: the average probability of each target being selected, the probability of the eye or head pointing in a direction, and the conditional probability of the head pointing in a direction.

[0095] Furthermore, the prediction module 204 is specifically used for:

[0096] Determine the probability that each candidate target belongs to the target selected by the user, and determine the candidate target corresponding to the maximum probability as the target selected by the user.

[0097] Example 3

[0098] Figure 3 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of the present invention. Figure 3A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0099] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0100] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as target prediction methods.

[0102] In some embodiments, the target prediction method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the target prediction method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the target prediction method by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0108] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0109] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A target prediction method characterized by, include: If a user's target selection request in a three-dimensional scene is detected, the range of motion of the user's eyes and / or head when making the target selection is determined; the range of motion represents the change in the user's line of sight and / or head direction from the start of target selection to the end of target selection. Determine the type of target prediction model, and construct the target prediction model based on the type, the motion amplitude, the input mechanism associated with the target selection request, and the target width of the candidate target. Based on the target prediction model and the attribute information of the candidate targets, the probability that the candidate target belongs to the target selected by the user is determined; Based on the probability, predict the target object selected by the user from the candidate target objects; The determination of the target prediction model type includes: if the determined motion amplitude is the motion amplitude of the eyes or head, then the target prediction model type is determined to be a single-modal model that predicts the target through the user's eye line of sight or head indication direction; if the determined motion amplitude is the motion amplitude of both the eyes and head, then the target prediction model type is determined to be a multimodal model that predicts the target through the user's eye line of sight and head indication direction. The process of constructing a target prediction model based on the type, the amplitude of motion, the input mechanism associated with the target selection request, and the target width of the candidate target includes: determining the mean and variance associated with the target prediction model based on the type, the amplitude of motion, the input mechanism associated with the target selection request, and the target width of the candidate target; and constructing a likelihood function and a target prediction model based on the mean and variance. The determination of the probability that a candidate target belongs to the user's selected target, based on the target prediction model and the attribute information of the candidate target, includes: For each candidate target, the mean and variance of the target prediction model, along with the attribute information of the candidate target, are substituted into the likelihood function to determine the conditional probability value associated with the target prediction model. The conditional probability value and the preset constant parameters of the target prediction model are input into the target prediction model, and the probability that the candidate target belongs to the target selected by the user is determined based on the output result.

2. The method of claim 1, wherein, Determine the range of eye and / or head movements when a user selects a target, including: Prompt the user to begin target selection and acquire the movement trajectory of the user's eyes and / or head; Based on the movement trajectory of the user's eyes and / or head, determine the range of eye and / or head movement when the user selects a target.

3. The method of claim 1, wherein, Also includes: If the target prediction model is a single-modal model, the preset constant parameters include: the average probability of each target being selected, and the probability of the eye or head indicating the direction; If the target prediction model is a multimodal model, the preset constant parameters include: the average probability of each target being selected, the probability of the eye or head pointing in a direction, and the conditional probability of the head pointing in a direction.

4. The method of claim 1, wherein, Based on the aforementioned probability, predicting the target object selected by the user from the candidate targets includes: Determine the probability that each candidate target belongs to the target selected by the user, and determine the candidate target corresponding to the maximum probability as the target selected by the user.

5. A target prediction device characterized by comprising: include: An amplitude determination module is used to determine the amplitude of eye and / or head movement when a user makes a target selection request in a three-dimensional scene if the user's target selection request is detected; the amplitude of movement represents the change in the user's line of sight and / or head direction from the start of target selection to the end of target selection. A construction module is used to determine the type of the target prediction model and construct the target prediction model based on the type, the motion amplitude, the input mechanism associated with the target selection request, and the target width of the candidate target. The probability determination module is used to determine the probability that a candidate target belongs to the target selected by the user, based on the target prediction model and the attribute information of the candidate target. The prediction module is used to predict the target object selected by the user from the candidate target objects based on the probability. Specifically, the construction module is used to: if the determined motion amplitude is the motion amplitude of the eyes or head, then determine the type of the target prediction model as a single-modal model that predicts the target through the user's eye line of sight or head direction; if the determined motion amplitude is the motion amplitude of both the eyes and head, then determine the type of the target prediction model as a multi-modal model that predicts the target through the user's eye line of sight and head direction. The construction module is further configured to: determine the mean and variance associated with the target prediction model based on the type, the amplitude of motion, the input mechanism associated with the target selection request, and the target width of the candidate target; and construct the likelihood function and the target prediction model based on the mean and variance. The probability determination module is also used for: For each candidate target, the mean and variance of the target prediction model, along with the attribute information of the candidate target, are substituted into the likelihood function to determine the conditional probability value associated with the target prediction model. The conditional probability value and the preset constant parameters of the target prediction model are input into the target prediction model, and the probability that the candidate target belongs to the target selected by the user is determined based on the output result.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the target prediction method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the target prediction method according to any one of claims 1-4.

Images