Method of estimating a three-dimensional position of an object

Abstract

The present disclosure relates to a method (300) performed by a computer for estimating a three-dimensional position (150) of at least one target object (111), the method comprising: obtaining a first image (130); detecting at least one target object (111); generating a set of images depicting a subset of a captured scene; and estimating the three-dimensional position of the at least one target object using a probability distribution (P) indicative of a likelihood that the at least one target object is at a particular position (x, y, z) in the scene.

Description

Technical Field

[0001] This application relates to estimating the three-dimensional position of an object. Specifically, this application relates to estimating the three-dimensional position of an object for use by a user gaze detection system. Background Technology

[0002] Interacting with computing devices is a fundamental behavior in today's world. Computing devices such as personal computers, tablets, and smartphones are ubiquitous in daily life. The systems and methods used to interact with these devices define how they are used and for what purposes.

[0003] Advances in eye / gaze tracking technology have enabled interaction with computers / computing devices using personal gaze information. For example, the location a user is looking at / viewing on a monitor or in a real-world scene can be used as input to a computing device. This input can be used alone or in conjunction with contact-based interaction technologies (e.g., using user input devices such as keyboards, mice, touchscreens, or other input / output interfaces). Such interaction typically requires estimating the 3D position of a target object for applications such as gaze estimation, body pose estimation, or facial feature estimation.

[0004] Estimating the location of one or more target objects can be performed by estimating the parameters of a probability distribution. These parameters typically require constraints, such as ensuring that the total probability of the event does not exceed 100%. Therefore, it is important to enforce the constraints imposed by the distribution's parameters.

[0005] Physical cameras typically capture a two-dimensional projection of a three-dimensional (3D) space. A physical camera possesses the property that points in the projected image space correspond to rays in the real-world / 3D space. In this context, the term "ray" refers to the path that electromagnetic radiation might follow when ray tracing is performed, typically from the camera to the target object, which is well-known in the art. Establishing a relationship between how points in the projected image space and rays in the real-world / 3D space are correlated for a given physical camera is called calibration. Calibration parameters can be divided into two parts: camera intrinsic (inherent) parameters and camera extrinsic parameters. The camera's extrinsic parameters describe the camera's position and orientation in 3D space. The camera's intrinsic parameters describe other camera properties, such as the camera's field of view and focal length.

[0006] Traditional 3D position estimation solutions are less accurate at estimating distances to objects compared to estimating the direction / light of a target object from an image.

[0007] Reference WO 2014041188A1 illustrates the use of a probability distribution that indicates the probability of a subject gazing at an object in an image. A drawback of this solution is its low accuracy in estimating the location of objects in the image.

[0008] Therefore, an improved method is needed to estimate the three-dimensional position of the target object.

[0009] Purpose of the invention

[0010] The purpose of the embodiments of the present invention is to provide a solution that mitigates or solves the above-mentioned disadvantages. Summary of the Invention

[0011] The above objectives are achieved through the subject matter described herein. Other advantageous embodiments of the invention are described herein.

[0012] According to a first aspect of the invention, the object of the invention is achieved by a computer-executed method for estimating the three-dimensional position of at least one target object, the method comprising: acquiring a first image depicting a scene captured by a physical camera, wherein the scene includes at least one target object; detecting at least one target object, wherein the detection is performed using an object detector and the first image, wherein the detection includes identifying at least one target object by the object detector and generating coordinates by the object detector indicating the position of at least one detected target object; generating a set of images depicting a subset of the captured scene, the subset of the scene being depicted by at least one virtual camera; estimating parameters of a probability distribution by providing the generated set of images to a trained model, the probability distribution indicating the probability that at least one target object is located at a specific position in the scene; and estimating the three-dimensional position of at least one target object using the probability distribution P.

[0013] At least one advantage of the first aspect of the present invention is that it improves the accuracy of the estimated three-dimensional position of at least one target object.

[0014] In one embodiment of the first aspect, the method further includes generating one or more target regions using predetermined camera calibration parameters of a physical camera, wherein each target region defines a corresponding subset of the scene depicted by a corresponding virtual camera, wherein the corresponding subset includes a corresponding detected target object.

[0015] In one embodiment of the first aspect, one or more target regions are configured with symmetrical geometry, such as rectangles or squares.

[0016] In one embodiment of the first aspect, the predetermined camera calibration parameters of at least one virtual camera include the selection of image sensor size, main axis through the virtual camera and at least one target object, and focal length, and a set of images is generated by applying the predetermined camera calibration parameters of at least one virtual camera to each virtual camera to generate the set of images and selecting pixels falling within the target area from the first image.

[0017] In one embodiment of the first aspect, the method further includes applying a transformation function v to the parameters of the estimated probability distribution P to transform the estimated parameters within predetermined constraints of the probability distribution P.

[0018] In one embodiment of the first aspect, the trained model is trained using training data, which includes images captured by a physical camera, ground real target areas, and ground real locations of target objects.

[0019] In one embodiment of the first aspect, the trained model is trained using a gradient-based optimizer and a loss function between the coordinates of the target object and the ground truth coordinates of the target object.

[0020] In one embodiment of the first aspect, the loss function includes a negative log-likelihood function.

[0021] In one embodiment of the first aspect, the loss function is defined as:

[0022]

[0023] Where (x, y, z) are the coordinates of the target object, (A, a, μ) are the parameters of the probability distribution, and N is the normalization constant.

[0024] In one embodiment of the first aspect, the probability distribution P is defined as:

[0025]

[0026] Where P is the probability that the target object is located at a specific position (x, y, z) in the scene, A is the 2×2 covariance matrix defining the image plane of the physical camera, a is the standard deviation of the estimated depth of the target object, μ is the mean of the estimated depth of the target object, ‖‖ is the norm and [] is a two-dimensional vector, and

[0027] Here, the function g(z) is configured to be symmetric about 0, and

[0028] Where, expression

[0029]

[0030] The z-axis is convex relative to the coordinate system of at least one virtual camera.

[0031] In one embodiment of the first aspect, the function f of the probability distribution P is selected from any of the following functions:

[0032]

[0033] f(x) = exp(-|x|)

[0034] f(x) = exp(-h1(x)).

[0035] In one embodiment of the first aspect, the function g of the probability distribution P is selected from any of the following functions:

[0036] g(x) = |x|

[0037] g(x) = x 2

[0038] g(x) = h1(x).

[0039] In one embodiment of the first aspect, the method further includes transforming the estimated three-dimensional position of at least one target object into a position in the coordinate system of the first image.

[0040] In one embodiment of the first aspect, the method further includes: obtaining a second image depicting at least a portion of a scene captured by a second physical camera, wherein the at least a portion of the scene includes at least one target object; detecting at least one target object, wherein the detection of at least one target object is performed using an object detector and the second image, wherein the detection includes identifying at least one target object by the object detector and generating second coordinates by the object detector indicating the position of at least one detected target object; generating a second set of images depicting a subset of at least a portion of the captured scene, the subset of the scene being depicted by at least one second virtual camera; estimating parameters of a second probability distribution by providing the generated set of images to a trained model, the second probability distribution indicating the probability that at least one target object is located at a specific position in the second image; generating a combined probability distribution by combining the probability distribution and the second probability distribution; and using the combined probability distribution to estimate the three-dimensional position of at least one target object.

[0041] According to a second aspect of the invention, the object of the invention is achieved by a system comprising a computer, the computer including: a processing circuit system; and a memory containing instructions executable by the processing circuit system, thereby enabling the computer to operate for performing the method according to the first aspect.

[0042] According to a third aspect of the invention, the object of the invention is achieved by a computer program comprising computer-executable instructions that, when executed on a processing circuitry system included in the computer, cause the computer to perform the method according to the first aspect.

[0043] According to a fourth aspect of the invention, the object of the invention is achieved by a computer program product comprising a computer-readable storage medium having a computer program according to the third aspect contained therein.

[0044] The scope of this invention is defined by the claims, which are incorporated herein by reference. By considering the following detailed description of one or more embodiments, those skilled in the art will gain a more complete understanding of the embodiments of the invention and will recognize additional advantages thereof. Reference will be made to the accompanying drawings, which will first be briefly described. Attached Figure Description

[0045] Figure 1 A method for estimating the three-dimensional position of a target object according to one or more embodiments of the present disclosure is shown.

[0046] Figure 2 Further details of a method for estimating the three-dimensional position of a target object according to one or more embodiments of the present disclosure are shown.

[0047] Figure 3 A flowchart of a method according to one or more embodiments of this disclosure is shown.

[0048] Figure 4 Examples of methods using two or more physical cameras according to one or more embodiments of this disclosure are shown.

[0049] Figure 5A An example of obtaining a first image comprising two target objects according to one or more embodiments of the present disclosure is shown.

[0050] Figure 5B Examples of detection of two target objects according to one or more embodiments of this disclosure are shown.

[0051] Figure 6 An example of generating a set of images according to one or more embodiments of this disclosure is shown.

[0052] Figure 7A Further details are shown regarding obtaining a first image comprising two target objects according to one or more embodiments of this disclosure.

[0053] Figure 7B Images captured by a physical camera are shown.

[0054] Figure 7C A set of images depicting a subset of the captured scene is shown.

[0055] Figure 8 Examples of estimating two three-dimensional positions of a corresponding target object according to one or more embodiments of this disclosure are shown.

[0056] Figure 9 Examples of using a combined probability distribution to estimate the three-dimensional position of a target object according to one or more embodiments of this disclosure are shown.

[0057] Figure 10 A system according to one or more embodiments of the present disclosure is shown.

[0058] By considering the following detailed description of one or more embodiments, those skilled in the art will gain a more comprehensive understanding of the embodiments of the invention and will recognize additional advantages thereof. It should be understood that the same reference numerals are used to denote the same elements illustrated in one or more figures. Detailed Implementation

[0059] In this specification and the corresponding claims, the word "or" should be understood to encompass the mathematical meaning of "and" and "or," and should not be understood as XOR (exclusive OR). The indefinite article "a" in this disclosure and the claims is not limited to "a," but can also be understood as "one or more," i.e., a plural number.

[0060] This application relates to the documents Linden 2019, Mohlin 2021, Liu 2018, Robbins 1951 and Kingma 2014, the entire contents of which are incorporated herein by reference.

[0061] Erik; SJOSTRAND, Jonas; PROUTIERE, Alexandre. Learning to personalize in appearance-based gaze tracking. From: Proceedings of the IEEE / CVF International Conference on Computer Vision Workshops. 2019. pp.0-0.

[0062] MOHLIN, David; BIANCHI, Gerald; SULLIVAN, Josephine. Probabilistic Regression with Huber Distributions. arXiv preprint arXiv:2111.10296,2021.

[0063] LIU, Katherine et al., Deep inference for covariance estimation: Learning Gaussian noise models for state estimation. From: 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 1436-1443.

[0064] ROBBINS, Herbert; MONRO, Sutton. A stochastic approximation method. The Annals of Mathematical Statistics, 1951, 400-407.

[0065] KINGMA, Diederik P.; BA, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[0066] This invention relates to estimating three-dimensional position, such as the position of a target object in an image. The main concept involves capturing images of real-world scenes using a physical camera with a specific field of view.

[0067] The captured image is then provided to a suitable object detector, which returns the coordinates of the object / target object being sought, and / or the pixel block representing the object in the captured image, and / or the bounding box surrounding the target object.

[0068] For example, a virtual camera can be created based on the detected object coordinates and predetermined calibration parameters. About Figure 2The creation of the virtual camera is further described. Predetermined calibration parameters may include, for example, camera-internal (intrinsic) parameters and camera-external parameters. The calibration parameters are selected such that the entire detected object (in the captured image) is represented within a subset of pixels in the captured image. In a sense, selecting the calibration parameters causes the virtual camera to "magnify" the detected objects in the captured image. In other words, a virtual camera is created that "magnifies" the detected objects and their associated coordinates.

[0069] In one example, the camera is configured with a fixed image resolution, a main axis passing through the estimated center of the detected object, for example, based on the coordinates of the object being searched / target, and / or representing the pixel block of the object in the captured image, and / or the bounding box surrounding the object. It should be understood that the virtual camera's main axis passes through the physical camera and the location of the detected object. The virtual camera is selected with the smallest possible focal length, provided that the entire object is captured using the selected focal length.

[0070] The resulting virtual camera is configured with a main axis that passes through the positions of the physical camera and the target object. It is then possible to find a correspondence or projection between the coordinates in the original camera image space and the virtual camera image space. The pixel values ​​of the virtual camera can be sampled from the captured image using standard image interpolation methods (such as nearest neighbor, bilinear, or bicubic interpolation) to create an image indistinguishable from an image captured by a camera with the virtual camera's calibration parameters, except for some interpolation artifacts.

[0071] Images from a virtual camera are then used as input to a trained model (e.g., a convolutional neural network) that predicts parameters of a probability distribution of the location of a target object relative to the position and orientation of the virtual camera. Optionally, the output of the trained model is then sent via a function N that maps / transforms the parameters to a standard form and enforces constraints on the distribution parameters.

[0072] Then, the mapped parameters and probability distribution can be mapped back to the coordinate system of the original physical camera using the previously determined correspondence or projection between the coordinates in the original camera image space and the coordinates in the "virtual camera" image space.

[0073] First, the trained model is trained using training data, which includes images captured by a physical camera and the corresponding known coordinates of the target object within the captured images. The trained model, or its parameters, are fitted to the training data by iteratively optimizing a fitness function (also called a loss or loss function) to output a probability distribution for at least one target object and estimates of its estimated 3D location, consistent with the training data. A standard approach for designing this loss or loss function is to use a negative log-likelihood function on the probability distribution in question.

[0074] At least one objective of this disclosure is to improve the accuracy of the generated probability distribution, and thereby improve the accuracy of estimating the three-dimensional position / coordinates of at least one target object.

[0075] definition

[0076] In this disclosure, the term "calibration parameter" refers to parameters that map from the captured image space to the real-world scene space. Calibration parameters typically include camera intrinsic parameters and / or camera extrinsic parameters. Camera extrinsic parameters describe the camera's position and orientation in 3D space. Camera intrinsic parameters describe other camera properties such as principal axis, field of view, and focal length.

[0077] In this disclosure, the terms “location” or “location of at least one target object” are used interchangeably.

[0078] The term “scene” as used in this article refers to an observed part or segment of the real world or a simulated world.

[0079] As used herein, the term "physical camera" refers to an imaging device, such as a video recorder or still camera, and includes at least an optical system, lenses, and / or an image sensor with an array of image pixel sensors. The imaging device is configured to capture digital images of real-world scenes.

[0080] As used herein, the terms "video" or "image" refer to electronic media used for recording, copying, playing back, broadcasting, and displaying moving or static visual media. A video consists of one video frame or multiple subsequent video frames. Each video frame includes a representation of the field of view of a scene. This representation comprises multiple pixels arranged in a specific aspect ratio and resolution.

[0081] Figure 1 A method for estimating the three-dimensional position of at least one target object according to one or more embodiments of the present disclosure is shown.

[0082] exist Figure 1 The image shows a real-world scene 110. Scene 110 includes at least one (target) object 111, such as a face in front of a screen. Scene 110 is captured as an image 130 by a physical camera 120. The physical camera 120 may be a video camera or a still camera, and includes at least an optical system, lenses, etc., or an image sensor with an array of image pixel sensors.

[0083] Parameter estimator 140 estimates the parameters (A, a, μ) of probability distribution P, which indicates the probability that at least one target object 111 is at or located at a specific position 150 (x, y, z) in the scene captured in image 130. Figure 1In the image 130, the probability distribution P is represented by an increased point density, which corresponds to an increased probability that at least one target object 111 is at a specific location 150 (x, y, z) in the scene captured by camera 120. Figure 1 The image is presented in two dimensions (x-axis, y-axis) as captured in image 130; however, it should be understood that the probability distribution P indicates the likelihood of at least one target object. In one example, the three-degree-of-freedom probability distribution P may be associated with the estimated three-dimensional location, and the four-degree-of-freedom probability distribution P may be associated with the estimated uncertainty.

[0084] Then, the three-dimensional position 150 of at least one target object 111 is estimated using the probability distribution P. This estimation can be, for example, derived from... Figure 10 The processor / processing circuitry system shown is used to execute this.

[0085] Figure 2 Further details of a method for estimating the three-dimensional position of at least one target object according to one or more embodiments of the present disclosure are shown.

[0086] In the first step, a first image 130 depicting scene 110 is obtained, such as... Figure 1 As shown.

[0087] like Figure 1 As shown, the first image 130 is typically obtained by capturing image 130 by a physical camera 120. For example... Figure 2 As shown, scene 110 and the captured image depicting scene 130 include at least one target object 111.

[0088] In subsequent steps, at least one object is detected, wherein at least one target object 111 is detected using object detector 210 and the first image 130. In an embodiment, detection includes identifying at least one target object 111 by object detector 210 and generating coordinates 150(x", y", z") by object detector 210, which indicate the position of at least one detected target object 111 in the first image. The coordinates may be defined, for example, as the coordinates and size of the target object, as a pixel patch representing the target object in the captured image, and / or as a bounding box surrounding the object.

[0089] Object detector 210 can be any suitable object detector, such as those based on Haar features, Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG) features, Region Candidate (R-CNN), Fast R-CNN, Faster R-CNN, or the Viola–Jones object detection framework with cascaded R-CNN. Other suitable detectors include Single-Side Detector (SSD), You Only Look Once (YOLO), RefineDet for Object Detection, Retina Networks, and Deformable Convolutional Networks.

[0090] Optionally, the method further includes generating a virtual camera 220 configured with a main axis that passes through the physical camera and the location of the detected object. Additionally or alternatively, the method further includes generating one or more target regions 305, 306 using predetermined camera calibration parameters of the physical camera, such as... Figure 6 As shown, each target region 305, 306 defines a corresponding subset of the scene depicted by the corresponding virtual camera, wherein the corresponding subset includes the corresponding detected target object 111. It should be understood that the target region can be defined by a set of generated coordinates. In one example, the generated coordinates would be the corners of the bounding box surrounding the target object. In other examples, the generated coordinates can have other forms to ensure that the target object is geometrically enclosed. Such coordinates can be generated, for example, based on an object detector. The object detector could be a face detector. Target regions with a predetermined aspect ratio are displayed as boxes that depict the outline of a set of images 230.

[0091] In a subsequent step, a set of images 230 depicting a subset of the captured scene 110 is generated. This subset of scene 110 is depicted by at least one virtual camera. In other words, the virtual camera extracts pixels from images captured by the physical camera, for example, by selecting pixel / pixel values ​​or by interpolating pixel / pixel values.

[0092] The detected coordinates can optionally be used to create a virtual camera with predetermined calibration parameters. The virtual camera is generated such that the entire detected target object 111 in the captured image 130 is represented in a subset of the pixels obtained from the captured image 130. In a sense, the virtual camera is generated, and the calibration parameters are selected such that the virtual camera "magnifies" the detected object in the captured image. In other words, the calibration parameters are selected to ensure that the field of view of the virtual camera includes the entire detected object 111.

[0093] In one example, this involves generating a 220 virtual camera configured with a main axis that passes through the physical camera and the location of the detected object. The virtual camera's focal length is then selected such that its field of view completely surrounds the detected object. Figure 2 In this case, it is achieved through "magnification". Figure 1 The faces in scene 110 shown are used to generate a subset of the first image 230 for display.

[0094] The pixel values ​​of the “virtual camera” can be sampled from images captured by a physical camera using standard image interpolation methods (such as nearest neighbor, bilinear, or bicubic interpolation) to create an image that is indistinguishable from an image captured by a camera with calibration parameters for the virtual camera, except for some possible interpolation artifacts.

[0095] In subsequent steps, a set of generated images 230 (such as...) is provided to the trained model (e.g., a convolutional neural network). Figure 2 (As depicted in 240) to estimate the parameters of the probability distribution P, which indicates the probability that at least one target object 111 is at a specific location 150 (x, y, z) in the scene.

[0096] In the intermediate steps, the intermediate output of the trained model is transformed by applying a transformation function N to the estimated parameters / vector v. Figure 2 The transformation function (N) is represented by the vector v(A', a', μ') to transform the estimated parameters (A', a', μ') into transformed and constrained parameters (A, a, μ) within predetermined constraints of the probability distribution P. In other words, the transformation function N forces the intermediate output parameters (A', a', μ') of the trained model to output parameters (A, a, μ) that satisfy the constraints. These constraints are boundary conditions, such as the positive definiteness of A and μ. z The constraints are >0 and the parameter a is large enough to guarantee the convexity of the negative log-likelihood.

[0097] The probability distribution P with density function f can then be derived using parameters (A, a, μ) that satisfy the constraints.

[0098] In subsequent steps, the probability distribution P is used to estimate the three-dimensional position (x, y, z) 150 of at least one target object 111 based on the transformed and constrained parameters (A, a, μ).

[0099] At least one advantage of this embodiment is that accuracy is increased and computational complexity is reduced when estimating the distance to the target object from the image and the direction / light of the target object.

[0100] The disclosed methods can be further illustrated in the flowchart.

[0101] Figure 3 A flowchart of a method 300 according to one or more embodiments of the present disclosure is shown. Method 300 estimates the three-dimensional position 150 of at least one target object 111 and is performed by a computer. The method includes:

[0102] Step 310: Obtain a first image 130 depicting scene 110, wherein scene 110 includes at least one target object 111. The first image may be captured by a physical camera 120 or generated by another suitable image source with known calibration parameters.

[0103] The calibration parameters of the physical camera are known, and therefore the relationship between points in the image space / plane and the light rays from the camera's image sensor to objects in the real-world scene are known.

[0104] Step 320: Detect at least one target object 111, wherein at least one target object 111 is detected on the first image 130 using object detector 210. Additionally or alternatively, the detection includes identifying 321 at least one target object 111 by object detector 210, and generating 323 coordinates indicating the location of at least one detected target object by object detector 210. (About...) Figure 2 The object detector is described further.

[0105] Object detectors typically return the coordinates of the sought object / target object, and / or a pixel block representing the target object in the captured image, and / or a bounding box enclosing the object. The coordinates indicating the location of at least one detected target object can be obtained as the returned coordinates, as any pixel coordinates of the block (in the corresponding image), or as the center or corner coordinates of the bounding box. In other words, the coordinates returned from the detector can be defined, for example, as the coordinates and size of the target object, as a pixel block representing the target object in the captured image, and / or as the bounding box enclosing the object.

[0106] Step 330: Generate a set of images 230 depicting a subset of the captured scene 110. The subset of the scene is depicted by at least one virtual camera.

[0107] In one example, a single target object, such as the eye or tip of the nose of a human face, is detected. A virtual camera is generated, configured with a main axis passing through the center of the physical camera and the eye or tip of the nose. The focal length of the virtual camera is then selected such that its field of view completely surrounds the detected object, such as the human eye. For example, this could involve adapting the virtual camera's field of view to a bounding box generated by object detector 210. (About...) Figure 2 The image set 230 is further described as being generated by a virtual camera.

[0108] In another example, two target objects are detected, such as the two eyes of a human face. Two virtual cameras are then generated. A first virtual camera is generated, configured with a main axis passing through the physical camera and the center of the left pupil. A second virtual camera is generated, with its main axis passing through the physical camera and the center of the right pupil. The focal length of the first virtual camera is then selected such that its field of view completely surrounds the left eye. The focal length of the second virtual camera is then selected such that its field of view completely surrounds the right eye. For example, this could involve adapting the field of view of the virtual cameras to bounding boxes generated by object detector 210. Generating this set of images 230 includes generating an image of the first eye depicted by the first virtual camera and an image of the second eye depicted by the second virtual camera.

[0109] Figure 5B The accompanying text illustrates another example of detecting two target objects.

[0110] Step 340: Estimate the parameters of the probability distribution P by providing the generated set of images 230 to the trained model 240 (e.g., a convolutional neural network CNN), which indicates the probability that at least one target object 111 is at a specific location (x, y, z) 150 in the scene 110.

[0111] Such as about Figure 2 Further described, the intermediate output of the trained CNN model is transformed by applying a transformation function N to the estimated parameters / vector v. Figure 2 The transformation function N (represented by the vector v(A', a', μ') transforms the estimated parameters (A', a', μ') into transformed and constrained parameters (A, a, μ) or the transformed and constrained output of the trained model, within the constraints of a predetermined probability distribution P. In other words, the transformation function N forces the intermediate output parameters (A', a', μ') of the trained model to output parameters (A, a, μ) that satisfy the constraints. The probability distribution P with the density function f can then be derived using the parameters while satisfying the constraints.

[0112] Estimate or generate a probability distribution for each of the 230 images in the set, and this probability distribution is initially located in the coordinate system of the corresponding virtual camera. See [link to relevant documentation] for further details regarding the coordinate system. Figures 7A to 7C . Figures 7A to 7CIt is shown that all cameras (physical and virtual cameras respectively) have their associated coordinate systems, typically related to the principal axis and orientation of the camera's field of view. The coordinate systems in these examples are shown as having a shared origin. However, since the calibration parameters of one or more virtual cameras and the calibration parameters of physical camera 120 are known, the probability distribution of each image in the set of images 230 can be transferred / projected to the coordinate system of physical camera 120. Any suitable method in the art can be used to transform / project the probability distribution of each image in the set of images 230 from the coordinate system of the corresponding virtual camera to the coordinate system of the physical camera, such as rotation and / or translation.

[0113] The probability distribution P can be defined, for example, as follows:

[0114]

[0115] V* is the most probable position, |C| is the number of cameras, i is the index of the iteration over all cameras, and p i R is the probability of the location estimated using camera i. i It is the orientation of camera I, and t i That is the position of camera i.

[0116] R i v is R i Matrix multiplication with v.

[0117] In one example, this distribution has seven degrees of freedom. (As for...) Figure 2 As described, the intermediate output of the trained model (e.g., a neural network) is called a vector v. The mapping between the intermediate output v from the trained model and the probability distribution parameters of the probability distribution P can be constructed, for example, as follows.

[0118] An example of the output vector is given in Liu 2018. In another example in Mohlin 2021, a symmetric matrix B is formed from the output vector v using a bidirectional injective method. Eigenvalue synthesis is then performed. Furthermore, a positive definite matrix is ​​constructed by independently applying a function to each eigenvalue.

[0119] For example, by using the mappings described in Liu 2018 or Mohlin 2021, the parameters A can be derived from the first three values ​​of the vector v. For instance, in Liu 2018, they used these three values ​​to parameterize the LDL decomposition of matrix A. L is a lower unit triangular (unit triangular) matrix, and D is a diagonal matrix. The LDL decomposition is often referred to as the square-rootless Cholesky decomposition.

[0120] parameter μ x and μ yThere are no constraints, and therefore the values ​​can be directly selected from the 4th and 5th values ​​of the output vector v. The parameter μ... z There is a constraint: it must be a positive number. This constraint can be achieved, for example, by applying an exponential function to the 6th value of the output vector v. The parameter "a" needs to be greater than:

[0121]

[0122] This is to ensure that the logarithm of the probability distribution P(x,y,z) is convex.

[0123] For example, this constraint can be enforced using the following constraint:

[0124]

[0125] This probability distribution can then be mapped back to the original camera's coordinate system, since the calibration parameters reveal how the virtual and physical cameras are related.

[0126] For example, if the virtual camera is rotated by a rotation R relative to the real camera, then the probability in the original camera system is...

[0127] p 真实 (v)=p 虚拟 (Rv)

[0128] Where Rv represents standard matrix multiplication.

[0129] Step 350: Estimate the three-dimensional position 150 of at least one target object 111 using probability distribution P.

[0130] In one example, an evaluation function can be applied to a probability distribution P. The probability distribution P can be in the coordinate system of the physical camera 120, and the evaluation function can be a function that identifies the highest / maximum value of the probability distribution P and the maximum value of the relevant coordinates of the first image. For example, if g(x) = |x|, then the maximum likelihood point is at...

[0131]

[0132]

[0133]

[0134] A simpler heuristic algorithm might be x = μ x μ z ,y=μ y μ z z = μ z

[0135] Additionally or alternatively, the method further includes generating one or more target regions 305, 306 using predetermined camera calibration parameters of a physical camera, such as Figure 6 As shown, each target region 305, 306 defines a corresponding subset of the scene depicted by the corresponding virtual camera, wherein the corresponding subset includes the corresponding detected target object 111. It should be understood that the target region can be defined by a set of generated coordinates. In one example, the generated coordinates would be the corners of the bounding box surrounding the target object. In other examples, the generated coordinates can have other forms to ensure that the target object is geometrically enclosed. Such coordinates can be generated, for example, based on an object detector. The object detector could be a face detector.

[0136] In one example, generating target regions 305, 306 includes selecting or adjusting the focal length of a first virtual camera to a minimum, wherein the field of view of the virtual camera (with a given aspect ratio) completely surrounds the target object 111, for example, by surrounding the target object with geometry such as a bounding rectangle or bounding box.

[0137] Additionally or alternatively, one or more target regions 305, 306 are configured with symmetrical geometry, such as a rectangle or box shape that defines the aspect ratio of the virtual camera, for example 1.85:1, 2.39:1, 4:3, 16:9, and 3:2. The x:y aspect ratio is defined as x units wide and y units high of the image.

[0138] Additionally or alternatively, the predetermined camera calibration parameters for at least one virtual camera include selections for the image sensor size, the principal axis passing through the virtual camera and at least one target object, and the focal length. A set of images is generated by applying the predetermined camera calibration parameters of at least one virtual camera to each virtual camera to generate the set of images and selecting pixels falling within the target area from the first image.

[0139] Additionally or alternatively, method 300 further includes applying a transformation function N to the estimated parameters to transform the estimated parameters within predetermined constraints of the probability distribution P, for example, indicating that the distribution generated by the trained model integrals to the value 1.0 and is thus a probability distribution.

[0140] Additionally or alternatively, the trained model is trained using training data that includes images captured by a physical camera, ground-based target regions, and the ground-based locations of target objects. It should be understood that the term "ground-based" generally refers to data typically created through human annotation. For example, a person drawing boxes and placing points on the image in question. Ground-based data can be further created in other ways, but the standard is human annotation.

[0141] In one example, the training data includes multiple images captured by a physical camera. The training data may further include known / ground-truth coordinates of the sought object / target object in each of the multiple images. The training data may further include known / ground-truth pixel blocks representing the object in each of the multiple images. The training data may further include known / ground-truth bounding boxes surrounding the target object.

[0142] The parameters of a trained model (such as the weights of neurons in a neural network) are trained or modified to produce the output of the trained model. When compared with the training data, the output of the trained model optimizes the fitness function, also known as the loss function.

[0143] In one embodiment, the loss function is defined using the negative log-likelihood over the distribution in question.

[0144] Then the loss function can be defined as:

[0145]

[0146] Here, N is a function of the parameters (A, a, μ) of the distribution, which forces the distribution based on vector v to integrate to 1.0. That is, it ensures that P is a probability distribution.

[0147] Backpropagation of the loss or loss function through a trained model (e.g., a convolutional neural network CNN) yields the gradient with respect to the network weights.

[0148] Gradient-based optimization methods, such as SGD (Robbins 1951) or Adam (Kingma 2014), are applied to update the parameters of the trained model. For example, the optimization performed in SGD is...

[0149]

[0150] in, This indicates the calculation of the standard gradient of L.

[0151] Additionally or alternatively, the trained model is trained using a gradient-based optimizer and a loss function between the target object's coordinates and its ground truth coordinates.

[0152] Additionally or alternatively, the loss function includes the negative log-likelihood function.

[0153] Additionally or alternatively, the loss function is defined as:

[0154]

[0155] Where (x, y, z) are the coordinates of the target object, (A, a, μ) are the parameters of the probability distribution, N is a normalization constant that forces the output parameters (A', a', μ') of the trained model to output parameters (A, a, μ) that satisfy the constraints, f(x) is a function, defined later, and μ x μ is the average of the estimated probability distribution x / z, and μy is the average of the estimated probability distribution y / z. z Proportional to the estimated average depth of the object, g(x) is a function, which we will define later.

[0156] Additionally or alternatively, the probability distribution P is defined as:

[0157]

[0158] Where P is the probability that the target object is at a specific position (x, y, z) in the scene, A is the 2×2 covariance matrix defining the image plane of the physical camera, a is the standard deviation of the estimated depth of the target object, μ is the mean of the estimated depth of the target object, ‖‖ is the norm and [] is a two-dimensional vector, and

[0159] Here, the function g(z) is configured to be symmetric about 0, and

[0160] Where, expression

[0161]

[0162] The z-axis is convex relative to the coordinate system of at least one virtual camera associated with the estimate.

[0163] In different embodiments of the present invention, the function f of the probability distribution P can be selected from any of the following functions:

[0164]

[0165] f(x) = exp(-|x|).

[0166] f(x) = exp(-h1(x)).

[0167] In different embodiments of the present invention, the function g of the probability distribution P can be selected from any of the following functions:

[0168] g(x) = |x|,

[0169] g(x) = x 2 ,

[0170] g(x) = h1(x).

[0171] Additionally or alternatively, the method further includes transforming the estimated three-dimensional position of at least one target object into a position in the coordinate system of the first image. In other words, the three-dimensional position of at least one target object in the coordinate system of the virtual camera is transformed into a three-dimensional position in the coordinate system of the first image / physical camera.

[0172] In one example, the coordinate systems of the virtual and physical cameras share their origin. That is, their respective principal axes pass through a point in the physical camera. However, the physical and virtual cameras have different orientations / principal axes. The transformation between these coordinate systems is actually a rotation.

[0173] In one example, a single target object, such as the eye of a human face, is detected. A virtual camera is generated, configured with a main axis that passes through the position of the physical camera and the center of the pupil, rather than through the center of the first image. The focal length of the virtual camera is then selected such that its field of view completely surrounds the detected object, such as the human eye. For example, this could involve adapting the field of view of the virtual camera to a bounding box generated by the object detector 210. (About...) Figure 2 The generation of this set of images 230 by a virtual camera is further described. The generated set of images 230 is then fed into a trained model, which returns parameters of a probability distribution P and the relevant coordinates of the virtual camera. The 3D position of at least one target object 111 is then estimated using the probability distribution P. The estimated position of the at least one target object is then defined in the coordinate system of the virtual camera. The estimated 3D position is then transformed into the estimated position in the coordinate system of the first image.

[0174] To further improve the accuracy of the estimated three-dimensional position of the target object, a second physical camera can be used.

[0175] Figure 4 Examples of methods using two or more physical cameras according to one or more embodiments of this disclosure are shown.

[0176] In this embodiment, method 300 further includes:

[0177] Step 361: Obtain a second image depicting at least a portion of a scene captured by a second physical camera, wherein at least a portion of the scene includes at least one target object 111.

[0178] Step 362: Detect at least one target object 111, wherein at least one target object 111 is detected using an object detector and a second image, wherein the detection includes identifying at least one target object by the object detector and generating second coordinates by the object detector indicating the position of at least one detected target object.

[0179] Step 363: Generate a second set of images depicting at least a subset of the captured scene, which is depicted by at least one second virtual camera.

[0180] Step 364: Estimate the parameters of a second probability distribution P2 by providing a generated set of images to the trained model. This second probability distribution indicates the probability that at least one target object is located at a specific position (x, y, z) in the second image.

[0181] Step 365: Generate the combined probability distribution PC by combining the probability distribution P and the second probability distribution P2.

[0182]

[0183] Where |C| is the number of cameras, p i R is the estimated probability for each camera. i It is the orientation of camera I, and t i This refers to the position of camera i. In one example, generating the combined probability distribution PC involves transforming two probability distributions to the same coordinate system and then multiplying the probability distributions to form the combined probability distribution PC.

[0184] Step 366 is to find the most likely location of this distribution.

[0185] Note that in a possible implementation of method 300, steps 365 and 366 may be combined.

[0186] Step 366: Use the combined probability distribution PC to estimate the three-dimensional position of at least one target object.

[0187] Estimating the direction from the camera to the target object is generally much easier than estimating the distance using images captured by a physical camera. By using multiple cameras to observe the object from different angles, the combination of all estimates can achieve an accuracy order of magnitude higher.

[0188] In one embodiment, the combined probability distribution P(v) can be used to estimate the three-dimensional position of at least one target using the following relationship:

[0189]

[0190] Where |C| is the number of cameras, i is the index of the iteration over all cameras, and p i R is the estimated probability from each camera. i It is the orientation of each camera, and t i This refers to the position of each camera. R i v uses standard matrix multiplication to multiply R and v.

[0191] Because the probability density function p i The negative of the logarithm of (v) is convex, so a convex function remains convex after an affine transformation of the independent variable, and the sum of convex functions produces another convex function. Therefore, for this distribution, the most likely location given an estimate can be framed as a convex optimization problem. Well-known fast methods for optimizing convex functions exist, such as line search. This makes it possible to quickly find the most likely location given a sensor estimate.

[0192] Figure 5A An example of obtaining a first image 130 including two target objects 301, 302 according to one or more embodiments of the present disclosure is shown. The first image 130 is obtained by capturing a first image by a physical camera 120 (not shown). The first image 130 depicts a scene 110, which is captured by the physical camera 120. Scene 110 includes two target objects 301, 302.

[0193] Figure 5B An example of detecting two target objects 301, 302 according to one or more embodiments of the present disclosure is shown. In this example, object detection also includes generating corresponding bounding boxes 303, 304, and associated coordinates, such as center coordinates and / or angular coordinates defined in the coordinate system of the first image.

[0194] Figure 6 An example of generating a set of images according to one or more embodiments of this disclosure is shown. In this example, such as Figures 5A to 5B As shown, two virtual cameras (not shown) are generated with an aspect ratio of 4:3, and the corresponding main axis of the virtual cameras intersects with the main axis of the physical cameras at the image sensor of the physical cameras and the corresponding target objects 301 and 302.

[0195] Then, the focal length of the corresponding virtual camera is selected such that the field of view 305, 306 of the corresponding virtual camera completely surrounds the detected objects 301, 302, i.e., the face in this example. In other words, the field of view 305, 306 of the corresponding virtual camera forms the target region. In this example, the field of view 305, 306 is shown with dashed lines. For example, this could involve adapting the field of view of the virtual camera to the bounding boxes 303, 304 generated by the object detector 210. Regarding... Figure 2 The generation of this set of images 230 by the virtual camera is further described. The generated set of images 230 is then fed to a trained model 240, which returns the parameters of the probability distribution P and the relevant coordinates of the virtual camera.

[0196] Figure 7AFurther details of obtaining a first image 130 including two target objects 301, 302 according to one or more embodiments of the present disclosure are shown. In this example, a bird's-eye view shows the two target objects 301, 302, a horizontal viewpoint 705 of the physical camera indicated by dashed lines, horizontal viewpoints 702, 706 of each of the two virtual cameras, coordinate systems 703, 707 of the two virtual cameras indicated by dashed lines respectively, and coordinate system 708 of the physical camera.

[0197] For the detection of each of the target objects 301 and 302, different virtual cameras are constructed using corresponding horizontal viewpoints 702 and 706. The corresponding virtual cameras are rotated to face the target object in question and have appropriately selected fields of view to create images that primarily contain image data indicating the corresponding target object 301 or 302. Figure 7A The relative rotation can be observed because the two coordinate systems 703 and 707 rotate relative to each other. For example, coordinate system 703 is rotated counterclockwise by a certain angle relative to coordinate system 707. In the example, the relative rotation could be approximately 30 degrees.

[0198] Figure 7B Images 130 captured by a physical camera (not shown) are displayed. From Figure 7A As can be seen, an object 302 is positioned in front of the physical camera along its principal axis (shown as a vertical line of the solid type). A second object 301 is positioned approximately 30 degrees away from the principal axis of the physical camera. It can be seen that the angular difference will cause the second object 302 to appear distorted in image 130.

[0199] Figure 7C Images 709 and 710 are shown, depicting a subset of the scene captured by the virtual camera as described above. It can be seen that, after the corresponding virtual camera has depicted the corresponding target object, the effects of angular difference and distortion are now mitigated.

[0200] Figure 8 Examples of estimating two three-dimensional positions according to one or more embodiments of this disclosure are shown. Figure 8 The diagram shows two estimated positions 801 and 802 for objects 301 and 302, and the corresponding uncertainties represented by ellipses 803 and 804. These uncertainties are estimated by the trained model using output parameters A and a. This corresponds to "true" uncertainty because it is difficult to fit ambiguous inputs well during training.

[0201] In this example, the target object could be, for example, the tip of a person's nose. After estimating the probability distribution P relative to the virtual camera and then transforming that distribution to the original coordinate system of the physical camera, it can be seen that most of the uncertainty and error is mainly due to the inability to accurately determine the distance to the target object (the tip of the nose in this example). The reason is that determining the position of the tip of the nose in the image captured by the camera is relatively easy. However, estimating the distance from the virtual / physical camera to the target object is ambiguous. In one example, a face might appear smaller in the image simply because two people's faces are different sizes. In other words, a face might appear smaller in 3D but is actually closer to the camera, thus producing a visually similar appearance to the camera when two faces of the same size are located at different distances from the physical camera.

[0202] The method of the present invention alleviates this problem by applying a trained model to a subset of images depicting the captured scene, thereby improving the accuracy of the estimation of the three-dimensional position of the object.

[0203] Figure 9 An example of estimating the three-dimensional position of a target object using a combined probability distribution, according to one embodiment of this disclosure, is shown. This will be about Figure 4 Further description.

[0204] Figure 9 The horizontal viewpoints 901 and 902 of two physical cameras (not shown) are shown using dashed lines. Therefore, both physical cameras have a single target object 301 in their respective horizontal viewpoints 901 and 902.

[0205] Figure 9 The horizontal viewpoints 1001 and 1002 of the two virtual cameras, represented by dashed lines, are further illustrated. The probability distribution P of the first physical camera is shown as ellipse 1003. The probability distribution P2 of the second virtual camera is shown as ellipse 1004. The combined probability distribution PC is shown as the intersection region 1005 of ellipses 1003 and 1004.

[0206] from Figure 9 It can be seen that by generating a combined probability distribution PC, the accuracy of predicting the three-dimensional position of the target object can be significantly improved.

[0207] Figure 10 A system 1000 according to one or more embodiments of the present disclosure is illustrated. System 1000 may be an eye-tracking system. System 1000 may include a computer 1020 configured to perform the methods described herein. The system may optionally include sensors (not shown). System 1000 may further include an optional second computer 1030.

[0208] In one example, one or more sensors may be configured to capture sensor data and send the sensor data, such as gaze tracking data, to computer 1020 and / or a second computer 1030.

[0209] As previously stated, it should be understood that the method steps described herein may be distributed between computer 1020 and second computer 1030 as appropriate, without departing from the teachings of this disclosure.

[0210] The system may optionally include one or more emitters (not shown) for illuminating the user's eyes, and at least a first camera / recorder for capturing video / images of the user's eyes. The system may optionally further include, for example, a second camera / recorder (not shown) for capturing video / images of scene 110.

[0211] The at least one light emitter and the first and second cameras / video recorders (not shown) may be arranged as separate units, or included in the computer 1020.

[0212] The light emitter (not shown) can be, for example, a light-emitting diode (LED) that emits light in the infrared or near-infrared frequency band. For example, each camera can be a complementary metal-oxide-semiconductor (CMOS) camera or a charge-coupled device (CCD) camera. The camera is not limited to an IR camera, a depth camera, or a light field camera. The camera's shutter mechanism can be a rolling shutter or a global shutter.

[0213] Figure 10 The system 1000 also includes a computer 1020. The computer 1020 includes a processing circuitry system 1021 (e.g., including one or more processors) for processing images captured by the camera 1013. The circuitry system 1021 may be connected / communically coupled to the cameras 1020, 1030 and the emitter (not shown), for example, via a wired or wireless connection.

[0214] The computer 1020 can be in the form of one or more electronic control units, servers, airborne computers, digital information displays, fixed computing devices, laptop computers, tablet computers, handheld computers, wrist-worn computers, smartwatches, PDAs, smartphones, smart TVs, telephones, media players, game consoles, in-vehicle computer systems, or navigation devices.

[0215] In one example, the processing circuitry 1021 may be any of a processor and / or a central processing unit and / or a processor module and / or a plurality of processors configured to cooperate with each other. Further, the computers 1020, 1030 may further include a memory 1022.

[0216] In one example, one or more memories 1022 may include a selection of hard RAM, a disk drive, a floppy disk drive, a magnetic tape drive, an optical disc drive, a CD or DVD drive (R or RW), or other removable or fixed media drives. Memory 1022 may contain instructions executable by a processing circuitry system to perform any of the methods and / or method steps described herein.

[0217] In one or more embodiments, the computer 1020 may further include an input device (not shown) configured to receive input or indication from a user and send a user input signal instructing the user to input or indicating to the processing circuitry system 1021.

[0218] In one or more embodiments, the computer 1020 may further include a display (not shown) configured to receive display signals from the processing circuitry 1021, the display signals indicating rendered objects such as text or graphical user input objects, and to display the received signals as objects, such as text or graphical user input objects.

[0219] In one embodiment, the display (not shown) is integrated with a user input device (not shown) and is configured to receive display signals from the processing circuitry 1021, the display signals indicating rendered objects such as text or graphical user input objects, and to display the received signals as objects such as text or graphical user input objects, and / or be configured to receive input or indications from the user and send user input signals indicating user input or indications to the processing circuitry 1021.

[0220] In an embodiment, the processing circuitry 1021 is communicatively coupled to a memory 1022 and / or an optional sensor interface (not shown) and / or an optional communication interface (not shown) and / or an optional input device (not shown) and / or an optional display (not shown) and / or at least one camera (not shown). The computer 1020 may be configured to receive sensor data directly from at least one camera (not shown) or via a wired and / or wireless communication network (not shown).

[0221] The second computer 1030 described herein may include all the features described above for computer 1020 or a selection of features.

[0222] In one embodiment, the first computer 1020 is operable to send sensor data to the second computer 1030, and the second computer 1030 is operable to perform the methods described herein.

[0223] The computer may further include a processor 1021 and a memory 1022, the memory 1022 containing instructions executable by the processor 1021, thereby enabling the computer to operate for performing the methods described herein.

[0224] In one embodiment, a computer program is provided that includes computer-executable instructions for causing computer 1020, 1030 to perform the methods described herein when executed on a processing circuitry system included in the computer.

[0225] In one embodiment, a computer program product including a computer-readable storage medium having the previously described computer program is provided.

[0226] A computer-readable storage medium may be a non-transitory machine-readable medium on which instructions are stored, wherein the instructions can be executed by a processing circuit system or one or more processors to perform the methods described herein. The non-transitory machine-readable medium may be, for example, a CD, DVD, or flash memory, from which a computer program can be downloaded to memory 1022.

[0227] Furthermore, those skilled in the art will recognize that computers 1020 and 1030 may include the necessary communication capabilities for executing this solution, in the form of, for example, functions, devices, units, elements, etc. Examples of other such devices, units, elements, and functions are: processors, memories, buffers, control logic, encoders, decoders, etc., as known to those skilled in the art.

[0228] Specifically, the processing circuitry system 1021 of this disclosure may include one or more instances of: a processor and / or processing device, a processor module and multiple processors configured to cooperate with each other, a central processing unit (CPU), processing unit, processing circuitry, processor, application-specific integrated circuit (ASIC), microprocessor, field-programmable gate array (FPGA), or other processing logic capable of interpreting and executing instructions. Therefore, the expression "processing circuitry system" can refer to a processing circuitry system comprising multiple processing circuits, such as any, some, or all of the items mentioned above. The processing device may further perform data processing functions for inputting, outputting, and processing data.

[0229] Finally, it should be understood that the present invention is not limited to the embodiments described above, but also relates to and incorporates all embodiments within the scope of the appended independent claims.

Claims

1. A method (300) for estimating the three-dimensional position (150) of at least one target object (111) performed by a computer, the method comprising: A first image (130) is obtained (310), the first image depicting a scene (110) captured by a physical camera (120), wherein the scene (110) includes at least one target object (111). Detect (320) at least one target object (111), wherein the at least one target object (111) is detected on the first image (130) using an object detector (210), wherein the detection includes the object detector (210) identifying the at least one target object (111) and the object detector (210) generating coordinates indicating the position (150) of the detected at least one target object (111). Generate (330) a set of images (230) depicting a subset of the scene captured in the first image (130), the subset of the scene being depicted by at least one virtual camera, wherein the at least one virtual camera is generated based on the detected target coordinates of the at least one target object (111) and configured to have a principal axis intersecting the positions of the physical camera (120) and the at least one target object (111), and estimate (340) parameters of a probability distribution (P) by providing the generated set of images to a trained model (240), the probability distribution indicating the probability that the at least one target object (111) is located at a specific position (150) in the scene (110), and The probability distribution (P) is used to estimate (350) the three-dimensional position (150) of at least one target object.

2. The method according to claim 1, further comprising: One or more target regions (305, 306) are generated using predetermined camera calibration parameters of the physical camera, wherein each target region (305, 306) defines a corresponding subset of the scene depicted by the corresponding virtual camera, wherein the corresponding subset includes the corresponding detected target objects.

3. The method according to claim 2, wherein: The one or more target regions (305, 306) are configured with symmetrical geometry, which includes rectangles.

4. The method according to claim 2, wherein: The predetermined camera calibration parameters for the at least one virtual camera include the selection of image sensor size, the principal axis or focal length passing through the virtual camera and the at least one target object, and The generated set of images is generated by applying the predetermined camera calibration parameters of the at least one virtual camera to each virtual camera to generate the set of images and selecting pixels falling within the target area from the first image.

5. The method of claim 1, further comprising: The transformation function v is applied to the estimated parameters of the probability distribution (P) to transform the estimated parameters within predetermined constraints of the probability distribution (P).

6. The method according to claim 1, wherein, The trained model (240) is trained using training data, which includes images captured by the physical camera, ground real target areas, and the ground real location of the target object.

7. The method according to claim 6, wherein: The trained model (240) is trained using a gradient-based optimizer and a loss function between the coordinates of the target object and the ground truth coordinates of the target object (111).

8. The method according to claim 7, wherein, The loss function includes the negative log-likelihood function.

9. The method according to claim 8, wherein, The loss function is defined as: ， ， Where (x, y, z) are the coordinates of the target object, (A, a, µ) are the parameters of the probability distribution, and N is the normalization constant.

10. The method according to claim 9, wherein, The probability distribution (P) is defined as follows: ， Where P is the probability that the target object is located at a specific position in the scene, A is a 2 × 2 covariance matrix defining the image plane of the physical camera, a is the standard deviation of the estimated depth of the target object, and µ is the mean of the estimated depth of the target object. It is a norm and It is a two-dimensional vector, and Here, the function g(z) is configured to be symmetric about 0, and Where, expression ， ， The z-axis of the coordinate system relative to the at least one virtual camera is convex.

11. The method according to claim 10, wherein, The function f of the probability distribution (P) is selected from any of the following functions: ， ， 。 12. The method according to claim 10, wherein, The function g of the probability distribution (P) is selected from any of the following functions: ， ， 。 13. The method of claim 1, further comprising transforming the estimated three-dimensional position (150) of at least one target object (111) into a position in the coordinate system of the first image.

14. The method of claim 1, further comprising: A second image is obtained, the second image depicting at least a portion of the scene captured by a second physical camera, wherein the at least a portion of the scene includes the at least one target object. The at least one target object is detected using the object detector and the second image, wherein the detection includes identifying the at least one target object by the object detector and generating second coordinates by the object detector, the second coordinates indicating the position of the detected at least one target object. A second set of images is generated, the second set of images depicting at least a subset of the captured scene, the subset of the scene being depicted by at least one second virtual camera. The parameters of a second probability distribution (P2) are estimated by providing the trained model with the generated set of images. This second probability distribution indicates the probability that the at least one target object is located at a specific position in the second image. A combined probability distribution (PC) is generated by combining the first probability distribution (P) and the second probability distribution (P2). The combined probability distribution (PC) is used to estimate the three-dimensional position of the at least one target object.

15. An eye-tracking system (1000), the system (1000) comprising: The computer includes a processing circuit system; And a memory containing instructions executable by the processing circuitry system, thereby enabling the computer to operate to perform the method according to any one of claims 1 to 14.

16. A computer program product comprising a computer-readable storage medium having a computer program, the computer program including computer-executable instructions, the computer-executable instructions being configured, when executed on a processing circuitry system included in the computer, to cause the computer to perform the method according to any one of claims 1 to 14.

Images