System, method, computer program, and computer-readable medium

Abstract

The invention relates to a system for ascertaining a feature of an object. The system comprises a primary module, an assisting module, and a controller, wherein the primary and the assisting module are designed to carry out a respective detection of the feature, generate measurement data with respect to the feature, communicate with the controller which is designed for this purpose, be actuated by the controller which is designed for this purpose, and transmit the measurement data to the controller which is designed for this purpose. The controller has means which are designed to actuate the primary module such that the measurement data generated by the assisting module is taken into consideration, and the controller has means which are designed to ascertain the feature of the object from the totality or from a part of the totality of the generated measurement data.

Description

System, method, computer program and computer-readable medium The present invention relates to a system for determining a feature of an object. The state of the art is represented by the following publications. [1] H. Klausing and W. Holpp, Real and Synthetic Aperture Radar, Boston: Oldenbourg Science Publishers, 2010. [2] M. Younis, C. Fischer and W. Wiesbeck, “Digital beamforming in SAR systems,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1735-1739, August 2003. [3] SS Ahmed, A. Schiessl and L. Schmidt, “A Novel Fully Electronic Active Real-Time Imager Based on a Planar Multistatic Sparse Array,” IEEE Transactions on Microwave Theory and Techniques, Vol. 59, No. 12, pp. 3567- 3576, 2011. [4] E. Fishier, A. Haimovich, R. Blum, D. Chizhik, L. Cimini and R. Valenzuela, “MIM0 radar: an idea whose time has come,” Proceedings of the 2004 IEEE Radar Conference, pp. 71-78, 2004. [5] A.M. Haimovich, R. Blum and L. Cimini, „MIM0 Radar with widely separated antennas.,“ Signal Processing Magazine, IEEE, Bd. 25, pp. 116-129, 2008. [6] M. Göttinger, M. Hoffmann, M. Christmann, M. Schütz, F. Kirsch, P. Gulden and M. Vossiek, „Coherent Automotive Radar Networks: The Next Generation of Radar-Based Imaging and Mapping,“ IEEE Journal of Microwaves, Nr. 1 , pp. 149-163, 2021. [7] Chen, V. C., „Detection and analysis of human motion by radar,“ 2008 IEEE Radar Conference, pp. 1 -4, 2008. [8] S. Björklund, H. Petersson, A. Nezirovic, M. B. Guldogan and F. Gustafsson, „Millimeter-wave radar micro-Doppler signatures of human motion,“ 2011 12th International Radar Symposium (IRS), pp. 167-174, 2011. [9] S. Z. Gurbuz and M. G. Amin, „Radar-Based Human-Motion Recognition With Deep Learning: Promising Applications for Indoor Monitoring,“ IEEE Signal Processing Magazine, Bd. 36, Nr. 4, pp. 16-28, 2019.

[0010] Y. Dobrev, "Radar method and system for determining the angular position, location, and / or, in particular, vectorial, velocity of a target." German Patent EP 3 470 874 A1, 11 / 10 / 2018.

[0011] JA Nanzer, “Millimeter-Wave Interferometry Angular Velocity Detection,” IEEE Transactions on Microwave Theory and Techniques, Vol. 58, No. 12, pp. 4128-4136, 2010.

[0012] C. Schüßler, M. Hoffmann, R. Ebelt, I. Weber and M. Vossiek, "Position and Velocity Fusion Using Multiple Monostatic Radar Sensors for Automotive Applications," 2021 IEEE Radar Conference (RadarConf21), pp. 1-6, 2021.

[0013] M. Zhao, Y. Tian, H. Zhao, MA Alsheikh, T. Li, R. Hristov, Z. Kabelac, D. Katabi and A. Torralba, "RF-based 3D skeletons," Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '18), pp. 267-281, 2018.

[0014] T. S. Ralston, G. L. Charvat and J. E. Peabody, „Real-time through-wall imaging using an ultrawideband multiple-input multiple-output (MIMO) phased array radar system,“ 2010 IEEE International Symposium on Phased Array Systems and Technology, pp. 551-558, 2010.

[0015] F. Adib, C. Hsu, H. Mao, D. Katabi and F. Durand, „Capturing the human figure through a wall,“ ACM Trans. Graph., Bd. 34, Nr. 6, 2015.

[0016] Z. Xiong, Y. Zhang, F. Wu and W. Zeng, „Computational Depth Sensing: Toward high-performace commodity depth cameras,“ IEEE Signal Processing Magazine, Bd. 3, Mai 2017.

[0017] M. Zollhöfer, P. Stotko, A. Görlitz, C. Theobalt, M. Nießner, R. Klein and A. Kolb, „State of the Art on 3D Reconstruction with RGB-D Cameras,“ Computer Graphics Forum, Nr. 37, 2018.

[0018] J. Wang, T. Karp, J. Munoz-Ferreras, R. Gomez-Garcia and C. Li, „A Spectrum-Efficient FSK Radar Technology for Range Tracking of Both Moving and Stationary Human Subjects,“ EEE Transactions on Microwave Theory and Techniques, Bd. 67, Nr. 12, pp. 5406-5416, 2019. Radar arrangements and methods for imaging objects in a variety of forms are known from the state of the art. Almost all methods for radar-based reconstruction of an image of an environment, scene, or object are based on the phase-correct superposition of multiple measurement signals generated by using multiple transmit and / or receive locations in an algorithm. Known radar-based reconstruction methods for multiple input multiple output (MIMO) radar systems, synthetic aperture radar, holography, tomography, etc. can be found, for example, in

[0001] and [2]. MIMO radars with many transmit and receive antennas, with corresponding numerical radar-based image reconstruction methods, allow high resolution and precise direction estimation, as is known from [3], [4], [5], [6]. In the aforementioned studies, it is generally assumed that the captured scenery is static or at least quasi-static. Quasi-static means that the captured scenery moves only slightly during the acquisition of the measurement data required for radar-based image reconstruction, thus avoiding major disturbances in the radar-based image reconstruction. If the objects to be imaged are moving, an image can be reconstructed using inverse synthetic aperture radar (ISAR). However, ISAR imaging requires that the relative motion between the radar and the object be known or estimated with high accuracy. To capture dynamic movements quickly, individual measurements must be taken very quickly. With a conventional imaging MIMO With radar technology, it is very difficult to achieve such a high measurement rate that dynamic movements can be continuously tracked from image to image. When using broadband radar measurement signals or from many different radar measurement frequencies and from many transmitting and receiving antennas, both the measurement process and the radar-based image reconstruction require a considerable amount of time. This is partly due to the fact that it is very difficult to find a large number of broadband transmission signals that are all uncorrelated and thus enable simultaneous transmission from all transmitting antennas. Consequently, in MIMO radar technology, especially with large MIMO arrays, so-called time division multiplexing is often used, in which all transmitters transmit one after the other. Due to its nature, this can lead to slowed measurements and motion artifacts in the radar-based image reconstruction.Real-time detection of body shells with radar fails when using time division multiplexing in combination with larger MIMO arrays, which are necessary for high-resolution imaging, and also usually due to the time that would be required for signal decorrelation and radar-based image reconstruction. In the field of radar technology, a variety of methods for detecting object movements are also known. These methods are not based on high-resolution imaging of the hull, but rather evaluate the Doppler signatures in the radar signals to detect movement. In particular, a variety of state-of-the-art methods are known for detecting human movements and also for reconstructing body poses. Human movement profiles consist of a multitude of individual movements. In addition to the movement of the torso, which is reflected in the radar signal as a Doppler signal component, the additional movement of the arms, legs, and fingers also generates less strong Doppler signals, which are referred to as micro-Doppler, as shown in [7]. Micro-Doppler signatures are used, for example, to classify and identify specific movements, often based on machine learning algorithms. This is disclosed in [8] and [9]. However, these works only aim at separating or classifying body parts or movements, but do not yet include any imaging. In Doppler velocity estimation, typically only the radial component of the velocity is evaluated. Determining the vector velocity is significantly more complex. Spatially distributed transmitter / receiver units are used, and complex algorithms are applied for scenes with differently moving substructures, as described in

[0010] ,

[0011] , and

[0012] . In

[0013] , 3D avatars of humans were reconstructed from FMCW radar measurement data using an artificial neural network consisting of a few measurement points. A phased array system for imaging human movement was presented in

[0014] . This system allows live tracking of a person, but not spatially resolved imaging. In

[0015] , a radar system was presented that can sequentially record and reconstruct image data at a frame rate of 12.5 Hz. Using a priori knowledge about the sequence of movement patterns and the associated reflection behavior of individual body regions, spatially and temporally resolved information about movement sequences is obtained. However, the aforementioned works contain neither a high-resolution imaging of the human body nor a direct measurement of the velocities distributed across the body surface. In optical imaging technology, many different methods for capturing the hull of objects are also known. Known preprocessing methods can be found in

[0016] and other referenced works. In conventional optical measurement systems, such as depth cameras, these methods are already integrated into the sensor. Optical measurement systems capture the surface of objects with a very high spatial resolution. However, this resolution, similar to the human eye, is limited by the field of view of the measurement system. Therefore, post-processing approaches based on the generated data are usually used. to complete and / or improve the captured envelope. Known approaches can be found in

[0017] , A further disadvantage of optical measuring systems is that they do not provide any direct additional speed information during preprocessing and are therefore only designed for static or at least quasi-static objects. To still be able to determine a movement, e.g., of a shell, additional post-processing algorithms are applied based on the data generated by the optical measurement system. If no further assumptions about the visible object can be made, these algorithms usually estimate the movement of the shell by tracking the successively generated spatial points over time. Known approaches for this can also be found in

[0017] . The accuracy of this tracking depends significantly on the ratio of the speed of the moving object to the measurement rate of the optical measurement system. Against this background, the present invention is based on the object of providing a system for the optimized detection of a feature of an object. This object is achieved by the subject matter having the features of independent claim 1. Advantageous developments of the invention are the subject matter of the dependent claims. Accordingly, the invention provides that the system comprises a primary modality, an assisting modality and a controller, wherein the primary and the assisting modality are each designed to detect the feature, generate measurement data with reference to the feature, communicate with the controller designed for this purpose, be controlled by the controller designed for this purpose and send the measurement data to the controller designed for this purpose, wherein the controller has means designed to control the primary modality in such a way that the measurement data generated by the assisting modality are taken into account and that the controller has means designed to are to determine the characteristic of the object from the totality or from a part of the totality of the measurement data generated. Preferably, it is provided that the feature of the object relates to the shell of the object and / or that the object is a living being and / or that the feature of the object is a position, speed and / or acceleration of a region of the object or the shell of the object and / or phase information of a wave reflected from a region of the object or the shell of the object. The term "control" preferably refers to any means, such as a circuit or a computer, designed to perform the functions attributed to the control or a control means within this invention. Control is thus preferably interpreted broadly and also includes, for example, closed-loop control. It is conceivable that the control comprises means for input, display and / or output, in particular of the determined feature of the object. It is preferably provided that the controller has means designed to store the measurement data. It can preferably be provided that the controller has means which are designed to compare a plurality of determined features and / or generated measurement data and thereby to determine a comparison value from which a further feature, in particular a position, a speed and / or an acceleration, is derived. In an advantageous embodiment, it is provided that the primary modality comprises a radar system with at least one transmitting antenna and at least one receiving antenna, wherein means are preferably provided which are designed so that waves with less than 10, preferably with exactly one, two or three different frequency support points are transmitted from a transmitting antenna and / or that the assisting modality comprises a, preferably three-dimensional, imaging sensor. Frequency support points can also be referred to as frequencies and preferably describe the temporal sequence of only one or more transmission frequencies. It is conceivable that the antennas of the radar system form a MIMO aperture, with means being available that are designed to generate a laterally focused image of the object through aperture synthesis. It is also conceivable that the primary modality comprises a CW, stepped frequency and / or frequency shift keying radar system. Preferably, the assisting modality comprises an optical measuring system, in particular a camera, depth camera, stereo camera or laser scanner. It is preferably provided that the control has means which are designed to optimize the determination of the feature of the object by means of features and / or measurement data. It can be provided that the control has means which are designed to use a statistical filter, preferably a Kalman filter, a motion model and / or a method known from information technology and / or that the means are designed to carry out a vectorial speed measurement. Preferably, the system is intended to be part of a sports, training, fitness measurement, or fitness information system, or to be used for a medical, psychological, diagnostic, or therapeutic purpose and / or to generate digital human avatars. The invention also relates to a method for determining a feature of an object using a system according to the invention, comprising the following steps: a) detecting the feature by the assisting modality and generating measurement data relating to the feature; b) detecting the feature by the primary modality, wherein the primary modality is controlled by the controller in such a way that the measurement data generated in step a) are taken into account, and generating measurement data relating to the feature; c) determining the feature from the measurement data. Preferably, it is provided that the feature of the object relates to the shell of the object and / or the object is a living being and / or that the feature of the object is a position, speed and / or acceleration of a region of the object or the shell of the object and / or phase information of a wave reflected from a region of the object or the shell of the object. It is conceivable that the generated measurement data is saved. It is conceivable that the method is carried out again, wherein the determined feature and / or the generated measurement data from the first implementation is compared with the determined feature and / or with the generated measurement data from the second implementation and wherein a comparison value is thereby determined from which a further and / or improved feature, in particular a position, a speed and / or an acceleration, is derived. It is also conceivable that the primary modality transmits waves with fewer than 10, preferably with exactly one, two or three different frequency support points from a transmitting antenna. Preferably, the method comprises a further step, in particular an aperture synthesis, wherein a laterally focused image of the object is generated. It may be provided that the method is carried out again and a feature and / or measurement data from a previous implementation are used to optimize the implementation. It is preferably provided that a statistical filter, preferably a Kalman filter, a motion model and / or a method known from information technology is used in the method and / or that a vectorial speed measurement is carried out. Preferably, the method is used in a sports, training, or fitness measurement or fitness information system or for a medical, psychological, diagnostic or therapeutic purpose and / or for generating digital human avatars. The invention also relates to a computer program which comprises instructions which cause the system according to the invention to carry out the method steps of a method according to the invention. The invention also relates to a computer-readable medium on which the computer program according to the invention is stored. A multimodal system for detecting the envelope of the body of living beings, in particular the body envelope of humans, and for determining temporal movements of the envelope is preferably proposed. Also explained are associated advantageous methods for operating the system and for detecting the envelope and its movements. An arrangement for detecting the body shell of a living being can also be provided, wherein the arrangement comprises a primary sensor modality which is a radar system with at least one transmitting antenna and at least one receiving antenna. antenna, the arrangement comprises a further assisting three-dimensional imaging sensor modality and in a first step an at least rough location and distance range is determined by the assisting sensor system in which at least one point of the shell to be detected is located and in a second step an image of this at least one point of the shell is reconstructed by the primary sensor modality using at least one radar signal frequency. It can be provided that in a third step the image of this at least one reconstructed point is compared with its image from a previous measurement with regard to the phase, and from this at least one phase difference value is determined and a movement feature is derived from the phase difference value. It can further be provided that the primary sensor modality is designed as a CW, stepped frequency and / or frequency shift keying radar and only a small number, preferably a number less than 10, in particular preferably only 1, 2 or 3 different radar signal frequency support points per transmitting antenna are used in a measurement. It can also be provided that the primary sensor modality is equipped with several transmitting antennas and several receiving antennas and the antennas form a MIMO aperture and this aperture is used in the sense of aperture synthesis to reconstruct a laterally focused image of points of the hull. Furthermore, it can be provided that the rough location and distance range in which at least one point of the shell of the object to be detected is located is determined by an optical measuring system, in particular by a depth camera, a stereo camera or a laser scanner. It can also be provided that, in addition to the first radar signal frequency, at least one further radar signal frequency is used to image this at least one point of the shell, and the phases of the images obtained with the first and with at least one further radar signal frequency are compared with regard to the phase values, and a distance value of at least one point of the body shell is derived from at least one phase difference value obtained in this way. An iterative process may be provided in which the location and speed information of at least one previous measurement is used to improve the representation or image of the shell and to improve the determination of its movement. The use of statistical filters, such as Kalman filters or the like, the use of motion models and / or an improvement of the methods known in information technology may be envisaged. It is also conceivable that a vectorial speed measurement is carried out. A sensor fusion of depth camera and radar can also be used. A system, an arrangement or a method is conceivable with which the body shell of a living being, preferably the body shell of a human being, or the temporal movement of the body shell is determined by measurement and at least one result of this measurement is used for a medical, psychological, diagnostic or therapeutic purpose. Furthermore, an arrangement is conceivable with which the body shell of a living being, preferably the body shell of a human being, or the temporal movement of the body shell is determined by measurement and at least one result of this measurement is used in a sports, training, or fitness measurement or fitness information system. Furthermore, an arrangement is conceivable with which the body shell of a living being, preferably the body shell of a human being, or the temporal movement of the body shell is determined by measurement and at least one result of this measurement is used to generate a digital human avatar. The present invention overcomes many of the limitations existing in the prior art. The invention also relates to an arrangement and advantageous methods for operating the arrangement for the high-precision, contactless detection of an object's hull as well as the velocity vector of one, several, or each point on this hull. For this purpose, a primary radar-based modality is preferably combined with an assisting 3D imaging modality, combining the strengths of the sensors. The hull and its movements are thereby preferably detected with comparatively little effort, with high spatial resolution, and with excellent velocity accuracy in all three spatial directions, and at a high measurement rate. For the invention, it is preferably irrelevant which radar-based reconstruction method is used. It is also preferably irrelevant which optical reconstruction method is used. The term "envelope" or "body envelope" is primarily understood to mean the interface or the sum of many coupled interfaces that create the boundary between the inside and outside of a living being's body. The outside is usually air or the atmosphere surrounding the living being, and the inside is preferably the body or the body's material structures. In many animals and humans, the covering or body covering can preferably be defined by the skin surface, although this is only one possible definition, since depending on the application it may be useful to exclude clothing, Hair, dirt or other adhesions to the body or other objects firmly attached to the body are to be counted as part of the covering or not. Preferably, the shell is also a shell of another object bounded by a shell. In this context, particular mention should be made of humanoid robots or bio-inspired robots, i.e., robots inspired by other living beings, or even handling or manufacturing robots. Likewise, technical mobile systems, in particular motor vehicles, transport systems, construction machinery, and mobile robots, preferably have a shell, with the body or chassis typically forming the shell. Detection of the shell is preferably understood to mean a metrological process in which the geometric shape of the shell is determined. In particular, the detection process is preferably an imaging process in which an image of the shell that is as geometrically correct as possible is generated. For this purpose, the spatial positions of several points on the shell are preferably determined and at least some points, but preferably all points of this set of points, are assigned further features, such as certain reflectivity properties such as the strength of the reflection, the frequency dependence of the reflection (called color in the optical field), or the polarimetric reflection behavior. Detection of the shell preferably also includes determining the positions of the shell points relative to a reference coordinate system. When determining the motion of the shell of a body or object, at least some of the points in the aforementioned set of points are also assigned at least one motion characteristic. A motion characteristic can be a scalar velocity, a vector velocity, a one-dimensional or multi-dimensional acceleration quantity, or even a complex multi-dimensional higher-order equation of motion. The detection of the hull and / or the determination of its movement are preferably divided into two categories: preprocessing and postprocessing. These differ in the dataset used. Preprocessing operates on data generated directly by one or more sensors or modalities. Preprocessing creates a preliminary, application-specific data set. When capturing the hull, preprocessing generates a preliminary geometric profile of the hull. When determining the hull's movement, preprocessing generates preliminary motion features. Post-processing, as the name suggests, occurs after pre-processing and expands the preliminary data set with additional information. This can, for example, be a correction or completion of existing data. Post-processing with regard to the detection of the shell can, for example, recalculate or adjust the previously determined geometric progression so that it approximates the real shell even better. In particular, post-processing of this type can also be used to complete the shell that could only be partially determined in pre-processing, for example because the back of an object could not be determined due to shadows from the sensors used. Post-processing to determine the movement of the shell can also be used to correct or refine the preliminary movement characteristics.In particular, they can extend the preliminary motion features assigned to a set of points by another set of points with motion features if the post-processing for determining the movement of the shell is combined, for example, with a post-processing for detecting the shell. Preferably, the detection of the body shell and its movement should be carried out by at least two different wave-based sensor modalities and their sensor data fusion. A wave-based sensor modality is preferably understood to mean a measuring arrangement that uses a waveform to capture measurement information, e.g., to capture an image of a body. Possible waveforms include electromagnetic waves, such as those in the microwave or optical range, or sound / ultrasound waves. The modality is preferably defined here in such a way that a modality can be distinguished both by its waveform (e.g., radar, optical, or ultrasonic) and by its measurement principle (e.g., narrowband CW or N-FSK radar or broadband FMCW or FSCW radar). As a preferred multimodal embodiment, a combination of at least one radar-based modality or a primary modality with at least one further assisting modality is preferably proposed. A three-dimensional imaging wave-based modality, in particular a three-dimensional optical imaging technique, preferably implemented with a depth camera, a stereo camera, or a laser scanner, is preferably proposed as the assisting modality. However, a broadband radar system can also be used as the assisting modality. However, this differs from the primary radar-based modality in that it delivers three-dimensional, high-resolution imaging in each measurement cycle and is significantly more complex and, in particular, can only measure significantly more slowly than the primary modality.As an assisting radar modality, imaging FMCW, FSCW, OFDM, or pulsed radar, as known from the state of the art, can be considered. The primary radar modality, however, is preferably designed to be optimized for detecting dynamic movements. The primary radar modality is preferably a narrowband continuous wave (CW) radar or a so-called stepped frequency radar (FSCW / SFCW), or an FSK or N-FSK radar. The radar is preferably designed as a multiple input multiple output (MIMO) radar with a plurality of transmit and receive antennas. Preferably, only a small number, preferably less than 10, of different frequency support points are transmitted via a transmit antenna, but in particular Preferably only one frequency support point, as with a CW radar, only two different frequency support points, as with a 2-FSK radar, or only three different frequency support points, as with a 3-FSK radar. The primary radar modality therefore preferably has good lateral resolution, i.e., seen perpendicular to the wave propagation direction, which is determined by the number of transmitting and receiving antennas or the size of the resulting synthetic aperture, and preferably only limited or even no axial resolution in the range direction, i.e., in the wave propagation direction. The ability to separate different targets in the range direction, i.e., the axial resolution, is preferably determined by the bandwidth of the radar signals used.Since a large signal bandwidth can complicate the measurement process, especially in MIMO radar systems, and in particular can slow down the measurement process, the primary radar modality is preferably designed to have a large bandwidth and thus also to have good resolution in the distance direction and thus also to have the ability to generate good three-dimensional images. Depending on the sensor technology or modality used, the general approach to reconstructing the image of objects also differs. Therefore, the entire set of reconstruction methods for an optical measurement system, which are preferably referred to as optical reconstruction methods or optical reconstruction, is preferably explicitly distinguished from the entire set of reconstruction methods for a radar-based measurement system, which are preferably referred to as radar-based reconstruction methods or radar-based reconstruction. It should be noted that the terms "a" and "an" do not necessarily refer to one of the elements, although this is a possible embodiment, but can also refer to a plurality of the elements. Likewise, the use of the plural also includes the presence of the element in question in the singular, and conversely, the singular also includes several of the elements in question. Furthermore, all features of the invention described herein may be combined with one another as desired or claimed in isolation from one another. Further advantages, features, and effects of the present invention will become apparent from the following description of preferred embodiments with reference to the figures, in which identical or similar components are designated by the same reference numerals. Herein: Fig. 1 : an embodiment of a system according to the invention. Fig. 2: another embodiment of possible structures of the modalities of a system according to the invention. Fig. 3: an embodiment of a possible embodiment of the modalities of a system according to the invention. Fig. 4: another embodiment of a system according to the invention. Fig. 5: two matrices with phase information or image phases per pixel. Fig. 6: a flow diagram of a preferred method. Figure 1 shows the combination of a primary modality 20 with an assisting modality 30 for determining the envelope of an object 10 and its movement. The object moves in Figure 1 with a velocity vector V with the reference symbol 11, which can be decomposed, for example, into the Cartesian components Vi and V2. The velocity vector V can also be a three-dimensional vector that can be broken down into three Cartesian components. The primary modality 20 and the assisting modality 30 are connected to a computer 40, which may be connected to a monitor 41, on which algorithms for determining the image and movement of the object 10 are executed. In particular, the individual orientations of the two modalities 20 and 30 are related to each other. Modalities 20 and 30 can be related to each other, for example, by the structures shown in Fig. 2. Modalities 20 and 30 capture the object 10 to be recorded. The orientations are illustrated, as in the following figures, by the unlabeled coordinate systems. In the setup shown on the left in Fig. 2, both modalities 20 and 30 are built into each other and thus have a common orientation, i.e., at least known relative to each other, a priori. Alternatively, both modalities 20 and 30 can be placed offset, and their relative orientation to each other can be calculated accordingly. For example, modalities 20 and 30 can be offset next to each other, as can be seen from the setup shown in the middle of Fig. 2. Likewise, modalities 20 and 30 can be oriented offset one above the other, as can be seen from the setup on the right in Fig. 2. Next to each other and one above the other preferably refers to the orientation to the object 10. A preferred combination is an imaging MIMO-FSK radar as the primary modality 20 with a depth camera as the assisting modality 30, which are installed one inside the other, as shown in Fig. 3. The MIMO-FSK radar or the primary modality 20 preferably has receiving antennas 21 and transmitting antennas 22. The two modalities 20 and 30 are pre- preferably placed in such a way that the data they generate overlap as extensively as possible in time and space, so that they can be fused and combined as precisely as possible. Fig. 4 shows an imaging MIMO-FSK radar as the primary modality 20 with a depth camera as the assisting modality 30, which are oriented offset from one another. The MIMO-FSK radar or the primary modality 20 preferably has receiving antennas 21 and transmitting antennas 22. For example, object 10 in Fig. 4 is a moving person or body part. Individual points of the person or body part move in Fig. 4, each with a velocity vector V with the reference symbol 11, which can be decomposed into the Cartesian components Vi and V2. The depth camera or the assisting modality 30 is connected to a computer 40 by a connection 35, via which distance characteristics of the object 10 detected by the depth camera or the assisting modality 30 are transmitted. The MIMO-FSK radar or primary modality 20 is connected to the computer 40 by a connection 25, via which the speed and distance characteristics of the object 10 detected by the MIMO-FSK radar or primary modality 20 are transmitted. The computer 40 can be connected to a monitor 41 on which the results of the algorithms executed on the computer 40 for determining the image and movement of the object 10 can be displayed. Without further combining the individual modalities 20 and 30, they initially provide the following data. The primary modality 20 initially provides the radial velocity by evaluating time-varying phase information. Continuous wave (CW) signals are particularly suitable signal forms for this purpose. These can consist of either one or multiple frequency support points. Using only one CW frequency, velocities as well as relative distance changes can be measured. Due to the periodicity of the phase, with each period running from 0 to 360°, absolute distances can only be determined within a comparatively narrow range of unambiguousness if the relevant period or the relevant distance range represented by this period is known. For signals with multiple frequency support points, an absolute distance measurement with a larger unambiguousness range is also possible. Particularly with the preferred configuration of the primary modality 20 in the form of an FSK-MIMO radar, which uses at least two frequency support points, an image of the recorded envelope of the object 10 is measured. Additionally, the spatially distributed velocity of this envelope can be determined using a suitable preprocessing method. In the simplest case, this can be a comparison of the consecutive radar images. Preference is given to either the evaluation of the complex phase of consecutive images or the evaluation of the continuous change in the reception phases. On the other hand, the assisting modality 30 initially provides spatially resolved distance data of the recorded shell of the object 10. The reconstruction method used to generate these distance data is preferably irrelevant. By combining the initially stand-alone modalities, for example, sensor data fusion, new, more comprehensive data with greater accuracy can be obtained. Post-processing methods used to fully capture the hull of Object 10 can also be applied more quickly and efficiently. As a first step in sensor data fusion, the assisting modality 30 roughly localizes the detected object 10 in space, thereby defining the area in which the hull of interest is located. Based on this, a radar-based reconstruction is performed in this area using a known method. By limiting the radar-based reconstruction to selected points in the relevant area, the radar-based reconstruction is significantly accelerated in its processing speed. In addition, the pre-established localization of object 10 allows the use of narrowband radar signal forms that have no or only very low range resolution. Their generation can be much faster than the broadband signal forms previously used in radar imaging and is therefore advantageous for time-critical applications. If the primary modality 20 uses at least two frequency support points, spatially resolved images of the shell can also be generated in addition to the existing data of the assisting modality 30. In addition to known methods, a new approach based on two frequency support points is preferred. This novel approach combines the frequency-shift keying approach, as known from

[0018] , with the currently common radar imaging methods. The disclosure content of

[0018] is hereby incorporated in its entirety into the present description. Based on the first rough localization by the assisting modality 20, two two-dimensional images of the shell are taken at the distance or z-coordinate d estimated by the assisting sensors or modality 20. susing two closely adjacent frequencies fi, f2 according to known methods. The starting point for the reconstruction is the two CW baseband signals nale s b fl and s bj2 , which is connected to the two frequency support points fi and f2 with a transmitting antenna at location x tx and a receiving antenna at location x rx recorded and measured by a point scatterer at location x s The following formulas show one way to determine the CW baseband signals. with Where c is the speed of light. The two hypotheses that apply to the two frequency support points and the estimated distance d s are represented by the following two formulas. 2d s s H ,fi = exp(J2nf1— ) 2d s s H ,f2 = exp(J2nf2— ) The estimated distance d sis the sum of the real distance d and either a positive or negative difference distance Ad. This results in the hypotheses shown in the following two formulas. 2d + Ad s H ,f2 = exp(J2nf2- - - ) When the baseband signals are correlated with the complex conjugate hypothesis according to the known methods for testing the signal hypothesis, the two correlation signals are obtained as shown in the two following formulas. In Fig. 5, two matrices with phase information or image phases per pixel are shown for a 2D reconstruction at the estimated distance d s for two frequency support where the left matrix shows the image phases for the frequency support point with the frequency and the right matrix shows the image phases for the frequency support point with the frequency f2. For each pixel of the estimated localization or the estimated envelope, the correlation is performed across all transmit-receive combinations and summed. This results in two complex radar images at two different frequencies and f2. By calculating the difference phase A of the two images and using the known frequency difference Af = f2- Provided there is a target at the corresponding pixel, Ad can be calculated and thus the exact distance of the hull to the radar can be calculated for each pixel, as shown in the following formula. When using at least two frequency support points, for example, according to the FSK-MIMO principle, on the part of the primary modality 20, with which a spatially resolved image of the hull is generated, this image is subsequently compared with the data from the assisting modality and, if necessary, complemented and fine-tuned. This step is omitted when using only one frequency. At this point, an image of various boundary surfaces of the hull can be generated, particularly by using different wavelengths, for example, by combining a radar-based primary modality 20 and an optical assisting modality 30. This is because, for example, radar can penetrate materials such as clothing or cardboard, whereas an optical measuring system cannot. Finally, a new data set is obtained by using a described measurement setup of both modalities 20 and 30 and the previously explained procedures for sensor data fusion, which provides as information per measurement cycle a most precise, spatially resolved image of the shell including spatially distributed velocity information. The previously explained preprocessing methods initially only provide data for individual measurement cycles. Postprocessing methods based on these estimate the continuous movement of the recorded shell over a longer period of time. As explained above, current methods in optical imaging technology cannot obtain any direct velocity information, especially spatially resolved ones. Therefore, the quality of postprocessing methods for estimating velocity is significantly influenced by the measurement rate of the optical measurement system or the assisting modality 30. In order to track the movement over time, correspondences between two consecutive images are preferably first found in order to then estimate the movement pattern of these individual correspondences.The greater the distance between the individual measurement cycles of the modality used, the larger the search space to find the correspondences and thus to determine the movement that occurred in between. With the availability of additional speed information, this search space can now be drastically reduced and the correspondences can thus be determined more quickly and accurately. Further approaches to reducing the search space include, for example, the use of application-specific models that estimate both the shell and the movement of the recorded object 10 in advance. In addition, the initially estimated speed information can be corrected retrospectively by using suitable statistical methods, for example a Kalman filter. In addition to additional speed information, the imaging of several interfaces by combining radar-based and optical measuring systems can be particularly advantageous if the recorded object 10 has several differently moving shells. This is the case, for example, when estimating the movement of a person's body, since an additional layer of clothing would obscure the movements of the human body in an optical measuring system. Preferably, the described methods enable a precise, high-performance determination of the movement of the shell of any object, which has the potential for real-time capability. The methods are particularly suitable for use in medical diagnosis and therapy, preventive care, and fitness and sports applications, as they can be applied to the movement of the human body, as mentioned above, in addition to the additional speed information, they also offer further advantages in sensor data fusion. It is also conceivable that the method could be used to generate a digital human avatar. An overview in the form of a flow chart of a preferred method for operating a preferred configuration is shown in Fig. 6. First, recording begins in step S1. Then, in step S2, a query is made as to whether the modalities have the same orientation. If not, calibration is performed in step S3. After calibration, or if the modalities have the same orientation, an optical reconstruction of the object or shell is performed by the assisting modality in step S4. This generates depth data D1. This depth data is used to localize the object or shell in step S5. In step S6, a radar-based reconstruction of the object or shell is performed by the primary modality. This results in data D2 with the speed and / or distance of the object or shell. In step S7, data D1 and D2 are fused using sensor data fusion. This leads to combined data D3 from the current recording. In step S8, the movement of the object or shell is determined.The data D3 and the data D4 from a previous recording serve as input data, on the basis of which the data D5 of the movement of the object or the. The shell is generated from the current image. Step S9 queries whether further images are available. If further images are available, step S8 is repeated. If no further images are available, the recording is ended in step S10.

Claims

Patent claims System for determining a feature of an object, characterized in that the system comprises a primary modality, an assisting modality and a controller, wherein the primary and the assisting modality are each designed to detect the feature, generate measurement data with reference to the feature, communicate with the controller designed for this purpose, be controlled by the controller designed for this purpose and send the measurement data to the controller designed for this purpose, wherein the controller has means designed to control the primary modality in such a way that the measurement data generated by the assisting modality are taken into account and that the controller has means designed to determine the feature of the object from the totality or from a part of the totality of the generated measurement data.System according to claim 1, characterized in that the feature of the object relates to the shell of the object and / or that the object is a living being and / or that the feature of the object relates to a position, speed and / or acceleration of a region of the object or the shell of the object. Object and / or phase information of a wave reflected from a region of the object or the shell of the object. System according to claim 1 or 2, characterized in that the controller has means designed to store the measurement data. System according to one of the preceding claims, characterized in that the controller has means designed to compare a plurality of determined features and / or generated measurement data and thereby determine a comparison value from which a further feature, in particular a position, a speed and / or an acceleration, is derived.System according to one of the preceding claims, characterized in that the primary modality comprises a radar system with at least one transmitting antenna and at least one receiving antenna, wherein means are preferably provided which are designed such that waves with fewer than 10, preferably with exactly one, two or three different frequency support points are emitted from a transmitting antenna and / or that the assisting modality comprises a, preferably three-dimensional, imaging sensor. System according to claim 5, characterized in that the antennas of the radar system form a MIMO aperture, wherein means are provided which are designed to generate a laterally focused image of the object by aperture synthesis. System according to one of the preceding claims, characterized in that the primary modality comprises a CW, stepped frequency and / or frequency shift keying radar system. System according to one of the preceding claims, characterized in that the assisting modality comprises an optical measuring system, in particular a camera, depth camera, a stereo camera or a laser scanner. System according to one of the preceding claims, characterized in that the controller has means which are designed to optimize the determination of the feature of the object using features and / or measurement data. System according to one of the preceding claims, characterized in that the controller has means which are designed to use a statistical filter, preferably a Kalman filter, a motion model and / or a method known from information technology and / or that the means are designed to carry out a vectorial velocity measurement.System according to one of the preceding claims, characterized in that the system is part of a sports, training or fitness measurement or fitness information system or that the system is used for a medical, psychological, diagnostic or therapeutic purpose and / or is used to generate digital human avatars. Method for determining a feature of an object using a system according to one of the preceding claims, comprising the following steps: a) detecting the feature by the assisting modality and generating measurement data with reference to the feature; b) detecting the feature by the primary modality, wherein the primary modality is controlled by the controller in such a way that the function selected in step... a) taking into account the generated measurement data and generating measurement data relating to the feature; c) determining the feature from the measurement data. Method according to claim 12, characterized in that the feature of the object relates to the shell of the object and / or the object is a living being and / or that the feature of the object is a position, velocity and / or acceleration of a region of the object or the shell of the object and / or phase information of a wave reflected from a region of the object or the shell of the object. Method according to claim 12 or 13, characterized in that the generated measurement data is stored.Method according to one of claims 12 to 14, characterized in that the method is carried out again, wherein the determined feature and / or the generated measurement data from the first implementation is compared with the determined feature and / or with the generated measurement data from the second implementation, and wherein a comparison value is thereby determined from which a further and / or improved feature, in particular a position, a speed and / or an acceleration, is derived. Method according to one of claims 12 to 15, characterized in that, by means of the primary modality, waves with fewer than 10, preferably with exactly one, two or three different frequency support points are emitted from a transmitting antenna. Method according to one of claims 12 to 16, characterized in that the method comprises a further step, in particular an aperture synthesis, wherein a laterally focused image of the object is generated. Method according to one of claims 12 to 17, characterized in that the method is carried out again and a feature and / or measurement data from a previous implementation are used to optimize the implementation. Method according to one of claims 12 to 18, characterized in that a statistical filter, preferably a Kalman filter, a motion model and / or a method known from information technology is used in the method and / or that a vectorial speed measurement is carried out. Method according to one of claims 12 to 19, characterized in that the method is used in a sports, training, or fitness measurement or fitness information system or for a medical, psychological, diagnostic or therapeutic purpose and / or for generating digital human avatars.A computer program comprising instructions that cause the system of any one of claims 1 to 11 to perform the method steps of any one of claims 12 to 20. A computer-readable medium on which the computer program of claim 21 is stored.

Images