Method and apparatus for tracking object

Abstract

The invention relates to a method for tracking an object, in particular a computer-implemented method, comprising the following steps: (i) detecting a plurality of temporally successive reference signals, each representing a body part, in particular a face or a hand, of a user by means of an electro-optical detection device, in particular a camera; (ii) detecting an object signal by means of an electro-optical detection device, the object signal representing an object, in particular a mobile device or a control handle, the object being mechanically coupled to the body part such that a substantially comparable movement of the object is effected by a movement of the body part, the detection of the object signal is implemented substantially simultaneously with the detection of one of the plurality of reference signals; (iii) determining a plurality of reference positions and / or orientations of the body part with respect to a predetermined position of the electro-optical detection device using a respective one of the plurality of reference signals; (iv) determining an object position and / or orientation of the object relative to a predetermined position of the electro-optical detection device using the object signal; (v) determining a transformed value of the geometric characteristic variable, the transformed value representing a difference between the object position and / or orientation of the object and a reference position and / or orientation determined by using a reference signal detected substantially simultaneously with the object signal; (vi) determining a plurality of further object positions and / or orientations using a respective further reference position and / or orientation of the plurality of further reference positions and / or orientations of the body part and the transformation values.

Description

Technical Field

[0001] The invention relates to a method for tracking an object, in particular to a computer-implemented method, a device and a computer program. Background Art

[0002] Modern cameras and smartphones with cameras now allow facial recognition, especially eye recognition. The algorithms used for this purpose are constantly being refined, so that it is also possible to track moving faces or other moving parts of the human body in terms of the corresponding position and orientation. In addition, objects can also be recognized and tracked by modern cameras. However, the recognition and tracking of objects may have the prerequisite that information about the corresponding object is stored in the algorithm used for this purpose, so that the algorithm can use this information. Even when information about an object is stored, this information may not be sufficient to allow reliable tracking of the object due to the lack of detail depth. For faces and other parts of the human body, extensive data has been mastered, and the corresponding algorithms can access this data in the corresponding databases. Therefore, tracking items or objects can be more difficult than tracking faces or other parts of the human body, and the computational complexity is also greater. Summary of the invention

[0003] The object of the present invention is to achieve improved tracking of objects.

[0004] This object is achieved according to the teaching of the independent claim. Various embodiments and developments of the invention are the subject matter of the dependent claims.

[0005] A first aspect of the solution relates to a method for tracking an object, in particular a computer-implemented method, comprising the following steps: (i) detecting a plurality of temporally successive reference signals by means of an electro-optical detection device, in particular a camera, the reference signals respectively representing a body part of a user, in particular a face or a hand; (ii) detecting an object signal by means of the electro-optical detection device, the object signal representing an object, in particular a mobile device or a control handle, the object being mechanically coupled to the body part so that substantially comparable movements of the object are achieved by the movement of the body part, the object signal being detected substantially simultaneously with the detection of a reference signal among the plurality of reference signals; (iii) detecting a corresponding one of the plurality of reference signals by means of the electro-optical detection device; (iv) determining the object position and / or orientation of the object relative to the predetermined position of the electro-optical detection device by using the object signal; (v) determining a transformation value of a geometric characteristic parameter, which represents the difference between the object position and / or orientation of the object and a reference position and / or an orientation, wherein the reference position and / or the orientation are respectively determined by using a reference signal detected substantially simultaneously with the object signal; and (vi) determining a plurality of additional object positions and / or orientations by using a corresponding additional reference position and / or orientation among the plurality of additional reference positions and / or orientations of the body part and the transformation value.

[0006] The concepts "comprise", "include", "have", "have", "with" or any other form thereof as may be used herein should cover a non-exclusive inclusion relationship. Thus, for example, a method or device comprising or having a list of elements is not necessarily limited to these elements, but may include other elements that are not explicitly stated or inherent to such method or such device.

[0007] In addition, unless explicitly stated to the contrary, "or" refers to an inclusive "or" rather than an exclusive "or". For example, condition A or B is satisfied by one of the following conditions: A is established (or exists) and B is not established (or does not exist); A is not established (or does not exist) and B is established (or exists); and not only A but also B is established (or exists).

[0008] As used herein, the concepts "a" and "an" are defined in the sense of "one or more." The concepts "an other" and "an additional" as well as any other forms thereof may be understood in the sense of "at least one additional."

[0009] As used herein, the concept "plurality" may be understood in the sense of "two or more".

[0010] As used herein, the terms "configured" or "built" to fulfill a specific function, and their corresponding variants, can be understood to mean that the corresponding device already exists in a scheme or setting in which the device can perform the function or the device can at least be set, i.e. configured, so that the device can perform the function after the corresponding setting. In this case, the configuration can be achieved, for example, by setting the parameters of the method flow accordingly or by switches or the like for activating or deactivating functionality or set values. In particular, the device can have a plurality of predetermined configurations or operating modes, so that the configuration can be achieved by selecting one of these configurations or operating modes.

[0011] As used herein, the term "object" may be understood to mean, in particular, a physical object that is movable and that can be moved by a movement of a user. An "object" may be, in particular, data glasses, in particular VR (virtual reality) glasses or AR (augmented reality) glasses. An "object" may also be a smartphone or a joystick (also known as a game joystick), in particular for use in flight simulators or for controlling computer games.

[0012] As used herein, the term "data glasses" may be understood to mean, in particular, a pair of glasses which, in addition to conventional glasses, have a display which can be arranged adjacent to one or both eyes of the user when the data glasses are worn. The display can comprise two sub-displays, one for each eye. Information in the form of text, graphic representations or a mixture of the two can be presented to the user on the display. The display can be, in particular, partially transparent and can therefore be designed so that the user can also recognize the environment behind the display.

[0013] As used herein, the term “signal” or “reference signal” may be understood in particular to mean an electromagnetic signal which can be detected by an electro-optical sensor and can be converted into an electrical signal.

[0014] As used herein, the term "electro-optical detection device" may be understood in particular to mean an electro-optical sensor which is designed to detect or measure electromagnetic signals and convert them into electrical signals. These sensors may in particular be charge-coupled sensors, which are also known as CCDs (charge-coupled devices); radar sensors; lidar sensors; or other sensors. In addition, such sensors may also be designed to transmit electromagnetic signals to an object and in turn measure the electromagnetic signals reflected back from the object, so that in a subsequent analysis, information about the object can be obtained from the transmitted and reflected electromagnetic signals, which is also known as TOF (time of flight) cameras.

[0015] By the method according to the first aspect, it can be achieved that an object can be tracked while adopting specific transformation values, especially spacing or angles. Here, the object is mechanically coupled to a body part of the user so that a substantially comparable movement of the object is achieved by the movement of the body part. Due to the determination of the reference position and / or orientation of the body part, the object position and / or orientation of the object can be determined by adopting the transformation value. Here, the detection of the object signal is substantially simultaneous with the detection of one reference signal among a plurality of reference signals. It can therefore be ensured that the object position and the reference position correspondingly determined by the object signal and the reference signal correspond at substantially the same time. Therefore, the transformation value thus determined also corresponds at this time. In a scenario, the reference signal and the object signal are detected at different times, in which this can result in that a movement of the body part has occurred between the detection of the reference signal and the detection of the object signal. As a result, the determined transformation value may have a significant deviation from the actual transformation value.

[0016] Assuming that the object is essentially fixed in position relative to the body part over a predetermined time period, it is possible to determine and adopt the object position and / or orientation of the object over this time period using a determined transformation value. It is thus possible to avoid continuously detecting object signals and thereby determining the corresponding object position and / or orientation. The significant computational overhead associated with this can be avoided. With the current method, this computational overhead for determining the object position and / or orientation of the object can be significantly reduced. In addition, the tracking or determination of the corresponding object position and / or orientation can be achieved more accurately. Because the object position and / or orientation of the object is indirectly determined via the reference position and / or orientation of the body part. The reference position and / or orientation of the body part can be reliably determined during this period because a wealth of already detected data can be called up for the body parts, especially for the face and hands.

[0017] Preferred embodiments of the method are described below, which can be combined as desired with one another and with other embodiments described elsewhere, unless expressly excluded or technically impossible.

[0018] In one embodiment, the method further comprises: (i) detecting a plurality of further object signals in a predetermined time interval; (ii) determining a plurality of further object positions and / or orientations of the object using a respective one of the plurality of further object signals; (iii) determining a plurality of further transformation values ​​using a respective plurality of determined object positions and / or orientations of the object and a reference position and / or orientation of the body part, the associated reference signal being detected substantially simultaneously with a respective one of the object signals; (iv) using a respective transformation value current in time from the plurality of transformation values ​​for determining the further object position. Possible changes in the position of the object relative to the body part and thus changes in the object position relative to the reference position can thus be taken into account. Due to such changes in the object position relative to the reference position, the determined transformation value can deviate from the current transformation value. Due to the determination of the transformation value using the detected object signal in the predetermined time interval, a respective updated transformation value can be determined, which can be used to determine a further, i.e. temporally subsequent, object position. The determination of the object position can thus be achieved more reliably.

[0019] In some embodiments, the predetermined time interval is adapted in relation to a predetermined criterion. Thus, an adaptation to the available computing power can be achieved. If the available computing power is graded as smaller, then the time interval can be increased, so that less computing power is required. If, on the other hand, the available computing power is graded as higher, then the time interval can be reduced, thus allowing a more precise determination of the object position, since any possible changes in the position of the object relative to the body part during this period can be taken into account when determining the object position by determining the transformation value earlier in time.

[0020] In some embodiments, the distance is determined by determining the transformation value. Thus, the distance between the object and the body part can be determined. The distance between two positions can be determined by forming the difference of the two coordinates of the determined positions and therefore requires only a small amount of effort.

[0021] In some embodiments, the angle is determined by determining the transformation value. The angle can be used to determine the rotation of the object relative to the body part and ultimately the orientation.

[0022] In some embodiments, tracking of an object within an interior of a motor vehicle is implemented, the user being a vehicle occupant.

[0023] A second aspect of the solution relates to a device for tracking an object, which is designed to carry out the method according to the first aspect.

[0024] Preferred embodiments of the device are described below, which can be combined as desired with one another and with other embodiments described elsewhere, unless expressly excluded or technically impossible.

[0025] In some embodiments, the device has: (i) an electro-optical detection device, which is constructed to detect multiple reference signals that are successive in time, and the reference signals respectively represent a body part of the user; and the electro-optical detection device is also constructed to detect an object signal, which represents the object, and the object is mechanically coupled to the body part so that basically comparable movement of the object is achieved through the movement of the body part; the detection of the object signal is basically achieved simultaneously with the detection of a reference signal among the multiple reference signals; (ii) an analysis device, which is constructed to determine multiple reference signals of a predetermined position of the body part relative to the electro-optical detection device by using a corresponding one of the multiple reference signals. position and / or orientation; and the analysis device is also constructed to determine the object position and / or orientation of the object relative to a predetermined position of the electro-optical detection device by using the object signal; and the analysis device is also constructed to determine a transformation value of a geometric characteristic parameter, which transformation value represents the difference between the object position and / or orientation of the object and a reference position and / or an orientation of a body part, wherein the reference position and / or the orientation are correspondingly determined by using a reference signal detected substantially simultaneously with the object signal; and the analysis device is also constructed to determine a plurality of other object positions and / or orientations of the object by using a corresponding one of the plurality of reference positions and / or orientations of the body part and the transformation value.

[0026] In some embodiments, the object has a mobile device, in particular data glasses, VR glasses, AR glasses or a smart phone. Such a mobile device may have a communication module, through which other devices can be communicated. Such communication can be performed in relation to the corresponding position of the mobile device.

[0027] In some embodiments, the electro-optical detection device has an interior camera of the motor vehicle, so that objects in the interior of the motor vehicle can be tracked.

[0028] A third aspect of the solution relates to a computer program having instructions which cause the device according to the second aspect to carry out the steps of the method according to the first aspect.

[0029] The computer program can be stored in particular on a non-volatile data carrier. Preferably, this is a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous when the computer program should be sold as a computer program independent of a processor platform, on which one or more programs can be executed. In other implementations, the computer program can exist as data on a data processing unit, especially on a server, and can be downloaded via a data connection, such as the Internet or a dedicated data connection, such as a private network or a local area network. In addition, the computer program can have a plurality of discrete program modules that act together. Each module can be configured in particular to be used for or at least can be used so that they are executed on different devices (computers or processor units) in the sense of distributed computing, and these different devices are geographically spaced apart from each other and are interconnected via a data network.

[0030] The features and advantages explained with respect to the first aspect of the solution also apply correspondingly to the other described aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Further advantages, features and application possibilities result from the following description of preferred embodiments in conjunction with the drawings.

[0032] The attached pictures are as follows:

[0033] Figure 1 shows a flowchart for describing a preferred embodiment of the method; and

[0034] Figure 2 A device according to one specific embodiment is shown schematically.

[0035] In the figures, the same reference symbols are always used for identical or mutually corresponding elements. DETAILED DESCRIPTION

[0036] exist Figure 1 Flowchart 100 is shown in FIG. 1 for describing a preferred embodiment of a method for tracking data glasses 230 .

[0037] In a first step 110 of the method, a plurality of temporally successive reference signals are detected by camera 210 , each of which represents head 250 of user 240 .

[0038] In another step 120 of the method, an object signal is detected by the camera 210, which object signal represents the data glasses 230, and the data glasses 230 are mechanically fixed on the head 250, so that a substantially comparable movement of the data glasses 230 is achieved by the movement of the head 250, and the object signal is detected substantially simultaneously with the detection of one reference signal among the multiple reference signals.

[0039] In a further step 130 of the method, a plurality of positions and / or orientations of head 250 relative to a predetermined position of camera 210 are determined using a respective one of the plurality of reference signals. In this case, so-called facial key points can also be used, which have fixed and predetermined points in the face, for example 68 points.

[0040] In a further step 140 of the method, the position and / or orientation of the data glasses 230 relative to a predetermined position of the camera 210 is determined using the object signal. In this case, the position or orientation of the data glasses 230 can be ascertained by the camera 210 via detection signals of active (transmitting) or passive (reflecting) infrared markers on the data glasses 230.

[0041] In a further step 150 of the method, a transformation value of the geometric characteristic variable is determined, which represents the difference between a position and / or an orientation of the data glasses 230 and a position and / or an orientation of the head 250, which position and / or the orientation are respectively determined using a reference signal that is detected essentially simultaneously with the object signal.

[0042] In a further step 160 of the method, a plurality of further positions and / or orientations of data glasses 230 are determined using a respective one of the plurality of positions and / or orientations of head 250 and the transformation value.

[0043] exist Figure 2 Schematically shows a device according to one embodiment. According to this embodiment, a camera 210 is arranged in a motor vehicle 200. Electromagnetic signals can be detected by the camera 210. In this case, the camera can have a 2D camera or also a 3D camera for detecting electromagnetic signals. Therefore, electromagnetic signals representing the head 250 of a vehicle occupant 240 can be detected by the camera 210. Electromagnetic signals representing data glasses 230 worn by the vehicle occupant 240 can also be detected by the camera 210. In this case, the data glasses 230 can be oriented and fixed on the head 250 of the vehicle occupant 240 by means of two legs and a nose bridge, just like conventional glasses for correcting vision. By means of the two legs and the nose bridge, the data glasses 230 are arranged sufficiently firmly so that during normal movements of the head 250, the position and orientation of the data glasses relative to the head 250 do not change. Therefore, a comparable movement of the data glasses 230 is achieved when the head 250 moves.

[0044] In addition, an evaluation device 220 is provided in the motor vehicle, which is connected to the camera 210 by signal technology. The evaluation device 220 is designed to determine the position and orientation or posture of the head 250 and the data glasses 230 relative to the camera 210 accordingly using the detected electromagnetic signals. The position of the camera 210 within the motor vehicle 200 can be determined by a so-called automatic calibration of the camera 210. Likewise, the position of the camera 210 can also be manually input. The orientation (head posture) of the head 250 and the orientation of the data glasses 230 can be determined by image comparison. Here, the image generated by the detected reference signal and / or object signal is compared with an image from a database, in which the corresponding orientation exists. It is also particularly advantageous that one of the determined positions and / or orientations of the data glasses 230 and the head 250 corresponds substantially at the same time. By these determined positions and / or orientations of the head 250 and the data glasses 230 corresponding at a time, the evaluation device 220 can determine the distance and / or different orientations between the head 250 and the data glasses 230. The spacing may be the spacing from the eyeglass to the face or from the temple edge to the forehead or another spacing. Likewise, the angle of the data glasses 230 relative to the head 250 can be determined. Now, when the data glasses 230 are continuously tracked, the position and / or orientation of the data glasses 230 can be determined by using the currently determined position and / or orientation of the head 250 and the spacing and / or the ascertained angle. Six degrees of freedom can be used here, i.e. three coordinates for the position and three angles, in particular Euler angles.

[0045] Since the position of the data glasses 230 is determined using the distance and / or angle of the data glasses relative to the head 250 and the position of the head 250, the required computing power can be saved. This is because it is not necessary to continuously determine the distance or angle, since this distance and the orientation of the data glasses 230 relative to the head 250 is essentially constant. In contrast, continuously determining the position of the data glasses 230 using correspondingly detected signals representing the data glasses 230 requires significantly more computing power.

[0046] As an alternative or in addition, the present disclosure may also relate to the tracking of a joystick, such as is used in simulators, in particular flight simulators or racing simulators.

[0047] Other objects are also conceivable, such as headphones or smartphones that can be tracked. The headphones can generate a spatial sound impression with a simulated sound source with fixed positions by tracking with the help of the so-called "head-related transfer function" (HRTF). By tracking the smartphone, AR results can be generated, and the fixed position elements can be supplemented in the description of the smartphone. In this case, as an alternative or in addition, other body parts other than the head can be tracked, such as the hand of the user 240, for example the hand holding the smartphone (not described here).

[0048] By tracking an object 230, such as data glasses and / or a control handle, a signal, in particular an optical or acoustic signal, can be triggered on the device by the movement of a body part. The signal can also change in relation to the movement of the object, or can remain fixed in position.

[0049] Although at least one exemplary embodiment is described above, it is noted that there are numerous variations of the described embodiments. It is also noted that the described exemplary embodiments merely form non-limiting examples and are not intended to limit the scope, applicability, or configuration of the apparatus and methods described herein. More precisely, the above description provides a skilled person with guidance for implementing at least one exemplary embodiment, and it is understood that different changes in the working mode and arrangement of the various elements described in the exemplary embodiments may be made without departing from the subject matter and legal equivalents thereof, respectively, identified in the appended claims.

[0050] Reference numerals list

[0051] 100 Flowchart

[0052] 110 Detecting multiple reference signals

[0053] 120 Detection of object signals

[0054] 130 Determine multiple reference positions

[0055] 140 Determine the position of an object

[0056] 150 Determine the transformation value

[0057] 160 Determine the positions of multiple additional objects

[0058] 200 Motor Vehicles

[0059] 210 Camera

[0060] 220 Analysis device

[0061] 230 Data Glasses

[0062] 240 Vehicle occupants

[0063] 250 Head of vehicle occupant

Claims

1. A method for tracking an object (230), the method The following steps are involved: Detecting a plurality of temporally successive reference signals by means of an electro-optical detection device (210), wherein the reference signals respectively represent a body part (250) of a user (240); detecting an object signal by the electro-optical detection device (210), the object signal being representative of the object (230), the object (230) being mechanically coupled to the body part (250) such that substantially comparable movement of the object (230) is achieved by movement of the body part (250), the object signal being detected substantially simultaneously with detection of a reference signal from among the plurality of reference signals; determining a plurality of reference positions and / or orientations of the body part (250) relative to a predetermined position of the electro-optical detection device (210) using a corresponding one of the plurality of reference signals; determining the object position and / or orientation of the object relative to the predetermined position of the electro-optical detection device (210) using the object signal; determining a transformation value of the geometric characteristic variable, which represents a difference between the object position and / or orientation of the object and a reference position and / or orientation, the reference position and / or orientation being respectively determined using a reference signal detected substantially simultaneously with the object signal; Using a respective one of a plurality of reference positions and / or orientations of the body part (250) and the transformation value, a plurality of further object positions and / or orientations are determined.

2. The method according to claim 1, further comprising: include: detecting a plurality of additional object signals at a plurality of predetermined time intervals; determining a plurality of further object positions and / or orientations of the object using a respective one of the plurality of further object signals; determining a plurality of further transformation values ​​using a respective one of the plurality of further object positions and / or orientations of the object and a reference position and / or orientation of the body part, wherein an associated reference signal is respectively detected substantially simultaneously with a respective one of the object signals; A respective temporally current transformation value of the plurality of transformation values ​​is used to determine the further object position.

3. The method according to any one of the preceding claims, in, The predetermined time interval is adapted as a function of a predetermined criterion.

4. The method according to any one of the preceding claims, in, The spacing is determined by determining the transformation value.

5. The method according to any one of the preceding claims, in, By determining the transformation value, the angle is determined.

6. The method according to any one of the preceding claims, in, Tracking of an object in the interior of a motor vehicle (200) is achieved, the user being a vehicle occupant (240).

7. A device for tracking an object (230), the device being designed to carry out the method according to any one of the preceding claims.

8. The device according to claim 7, comprising: An electro-optical detection device (210), the electro-optical detection device being configured to detect a plurality of temporally successive reference signals, the reference signals respectively representing a body part (250) of a user (240); and the electro-optical detection device being further configured to detect an object signal, the object signal representing the object (230), the object (230) being mechanically coupled to the body part (250) such that substantially comparable movements of the object (230) are achieved by movement of the body part (250); the object signal being detected substantially simultaneously with the detection of a reference signal among the plurality of reference signals; An analysis device (220), the analysis device being configured to determine a plurality of reference positions and / or orientations of the body part (250) relative to a predetermined position of the electro-optical detection device (210) by using a corresponding one of the plurality of reference signals; and the analysis device being further configured to determine an object position and / or orientation of the object relative to the predetermined position of the electro-optical detection device (210) by using the object signal; and the analysis device being further configured to determine a transformation value of a geometric characteristic parameter, the transformation value representing a difference between the object position and / or orientation of the object and a reference position and / or orientation of the body part, the reference position and / or orientation being respectively determined by using a reference signal detected substantially simultaneously with the object signal; and the analysis device being further configured to determine a plurality of other object positions and / or orientations of the object by using a corresponding one of the plurality of reference positions and / or orientations of the body part (250) and the transformation value.

9. The device according to claim 7 or 8, in, The object has a moving device (230).

10. The device according to any one of claims 7 to 9, in, The electro-optical detection device has an interior camera (210) of a motor vehicle (200). 11 . A computer program comprising instructions for causing a device according to claim 7 to 10 to carry out the steps of a method according to claim 1 .

Images