System, method, computer program and computer readable medium

Abstract

The invention relates to a system for ascertaining features of an object, said system comprising a main modality, an auxiliary modality and a control device, said main modality and the auxiliary modality being designed to detect a feature in each case, to generate measurement data relating to the feature, to communicate with the control device, to be actuated by the control device, and transmitting the measurement data to a control device designed for this purpose, the control device having means designed to actuate the main modality such that the measurement data generated by the auxiliary modality are taken into account, and the control device having means designed to actuate the auxiliary modality such that the measurement data generated by the auxiliary modality are taken into account. The device is designed to determine a feature of the object from the entirety or part of the entirety of the generated measurement data.

Description

Technical Field

[0001] The invention relates to a system for determining characteristics of an object. Background Art

[0002] The prior art is formed by the following publications.

[0003] [1] H. Klausing and W. Holpp, Radar mit realer und synthetischer Apertur, Boston: Oldenburg, 2010.

[0004] [2] M. Younis, C. Fischer and W. Wiesbeck, “Digital beamforming in SAR systems”, IEEE Transactions on Geoscience and Remote Sensing, pp. 1735–1739, August 2003. [3] S.S. Ahmed, A. Schiessl, and L. Schmidt, “A Novel Fully Electronic Active Real-Time Imager Based on a Planar Multistatic Sparse Array,” IEEE Transactions on Microwave Theory and Techniques, vol. 59, No. 12, pp. 3567–3576, 2011. [4] E. Fishler, A. Haimovich, R. Blum, D. Chizhik, L. Cimini, and R. Valenzuela, “MIMO radar: an idea whose time has come,” in Proceedings of the 2004 IEEE Radar Conference, pp. 71–78, 2004.

[0005] [5] A. M. Haimovich, R. Blum, and L. Cimini, “MIMO Radar with widely separated antennas.” Signal Processing Magazine, IEEE, vol. 25, pp. 116–129, 2008.

[0006] [6] M. Gottinger, M. Hoffmann, M. Christmann, M. Schütz, F. Kirsch, P. Gulden, and M. Vossiek, “Coherent Automotive Radar Networks: The Next Generation of Radar-Based Imaging and Mapping,” IEEE Microwave Magazine, no. 1, pp. 149–163, 2021.

[0007] [7] Chen, V.C., “Detection and analysis of human motion by radar,” 2008 IEEE Radar Conference, pp. 1–4, 2008.

[0008] [8] S. H. Petersson, A. Nezirovic, M. B. Guldogan, and F. Gustafsson, “Millimeter-wave radar micro-Doppler signatures of human motion,” in 2011 12th International Radar Symposium (IRS), pp. 167–174, 2011.

[0009] [9] SZ Gurbuz and MG Amin, “Radar-Based Human-Motion Recognition With Deep Learning: Promising Applications for Indoor Monitoring,” IEEE Signal Processing Magazine, vol. 36, No. 4, pp. 16–28, 2019.

[0010]

[10] Y. Dobrev, "Radar-Verfahren und-System zur Bestimmune der Winkellage, des Ortes und / oder der, insbesondere vektoriellen, Geschwindigkeiteines Zieles", German Patent EP 3 470 874A1, October 11, 2018.

[0011]

[11] JANanzer, “Millimeter-Wave Interferometric Angular Velocity Detection”, IEEE Transactions on Microwave Theory and Techniques, Vol. 58, No. 12, pp. 4128–4136, 2010.

[0012]

[12] C. Schüβler, M. Hoffmann, R. Ebelt, I. Weber, and M. Vossiek, “Position and Velocity Fusion Using Multiple Monostatic Radar Sensors for Automotive Applications,” in 2021 IEEE Radar Conference (RadarConf21), pp. 1–6, 2021.

[0013]

[13] M. Zhao, Y. Tian, ​​H. Zhao, M. A. Alsheikh, T. Li, R. Hristov, Z. Kabelac, D. Katabi, and A. Torralba, “RF-based 3D skeletons,” in Proceedings of the 2018 ACM Special Interest Group on Data Communications (SIGCOMM’18), pp. 267–281, 2018.

[0014]

[14] TS Ralston, GL Charvat, and JEPeabody, “Real-time through-wall imaging using an ultrawideband multiple-input multiple-output (MIMO) phased array radar system,” in 2010 IEEE International Symposium on Phased Array Systems and Technology, pp. 551–558, 2010.

[0015]

[15] F. Adib, C. Hsu, H. Mao, D. Katabi, and F. Durand, “Capturing the human figure through a wall,” ACM Transactions on Graphics, vol. 34, no. 6, 2015.

[0016]

[16] Z. Xiong, Y. Zhang, F. Wu, and W. Zeng, “Computational Depth Sensing: Toward high-performance commodity depth cameras,” IEEE Signal Processing Magazine, vol. 3, May 2017.

[0017]

[17] M. P. Stotko,A. C. Theobalt, M. Nieβner, R. Klein, and A. Kolb, “State of the Art on 3D Reconstruction with RGB-D Cameras,” Computer Graphics Forum, No. 37, 2018.

[0018]

[18] J. Wang, T. Karp, J. - Ferreras, R. Gómez-García, and C. Li, “A Spectrum-Efficient FSK Radar Technology for Range Tracking of Both Moving and Stationary Human Subjects,” IEEE Transactions on Microwave Theory and Techniques, Vol. 67, No. 12, pp. 5406–5416, 2019.

[0019] Radar devices and methods for imaging objects in various forms are known from the prior art. Almost all methods for radar-based reconstruction of an environment, a scene or an imaging of an object are based on the in-phase superposition of multiple measurement signals in an algorithm, which are generated by using multiple transmission and / or reception positions. Known radar-based reconstruction methods for multiple-input multiple-output (MIMO) radar systems, synthetic aperture radars, holography, tomography, etc. are found, for example, in [1] and [2]. MIMO radars with many transmitting and receiving antennas allow high resolution and precise direction estimation by means of corresponding numerical radar-based image reconstruction methods, as is known from [3], [4], [5], [6].

[0020] In the above work, it is usually a prerequisite that the detected scene is static or at least quasi-static. Quasi-static means that the detected scene moves only slightly during the recording of the measurement data required for radar-based image reconstruction, so that no major interference occurs in the radar-based image reconstruction.

[0021] If the object to be imaged moves, the image can be reconstructed with the aid of inverse synthetic aperture radar (ISAR). However, a prerequisite for ISAR imaging is that the relative movement between radar and object is known or estimated with high accuracy.

[0022] In order to be able to detect dynamic movements quickly, the individual measurements must be performed very quickly. With conventional imaging MIMO radar technology, it is only very difficult to achieve a high measurement rate so that dynamic movements from image to image can be continuously tracked. When using broadband radar measurement signals or multiple different radar measurement frequencies and many transmitting and receiving antennas, not only the measurement process but also the radar-based image reconstruction requires significant time consumption. On the one hand, this is only very difficult to find multiple broadband transmission signals that are completely uncorrelated and can be transmitted simultaneously by all transmitting antennas. Therefore, in MIMO radar technology, especially in the case of large MIMO arrays, so-called time division multiplexing is usually used, in which all transmitters transmit successively. This will cause slow measurement and motion artifacts in radar-based image reconstruction due to the principle. When time division multiplexing is used in conjunction with a larger MIMO array, the real-time detection of the body envelope by radar fails, but this is necessary for high-resolution imaging, and usually also in the time required for signal decorrelation and radar-based image reconstruction.

[0023] From the field of radar technology, many methods are also known for detecting the movement of objects, which are not based on high-resolution imaging of the envelope, but rather on evaluating Doppler signatures in the radar signal in order to detect the movement. In particular, many methods are known from the prior art for detecting human movements and also for reconstructing body postures.

[0024] The movement trajectory of a person consists of a number of individual movements. In addition to the movement of the trunk, which is reflected in the radar signal as a Doppler signal component, additional movements of the arms and legs and fingers also generate less strong Doppler signals, which are called micro-Doppler signals, as is known from [7]. Micro-Doppler markers are used, for example, to classify and identify specific movements, usually based on machine learning algorithms. This is disclosed in [8] and [9]. However, the purpose of the work described is only to separate or classify body parts or movements, but has not yet included imaging.

[0025] In Doppler velocity estimation, usually only the radial component of the velocity is evaluated. Determining the vector velocity is significantly more demanding. Spatially distributed transmitting and receiving units are used here, and complex algorithms are applied in scenes with differently moving substructures, as described, for example, in

[10] ,

[11] ,

[12] .

[0026] In

[13] , a 3D avatar of a person is reconstructed from FMCW radar measurement data using an artificial neural network from several measurement points. In

[14] , a phased array system for imaging human movement is presented. The system allows real-time tracking of people, but does not allow spatially resolved imaging. In

[15] , a radar system is presented that can sequentially record and reconstruct image data at a frame rate of 12.5 Hz. By prior knowledge of the movement patterns of the individual body regions and the sequence of the associated reflection behavior, spatially and temporally resolved information about the movement process is obtained. However, the work does not include high-resolution imaging of the human body nor direct measurement of the velocities distributed over the body surface.

[0027] In optical imaging technology, many different methods are also known for detecting the envelope of an object. Known preprocessing methods can be found in

[16] and other reference books.

[0028] In known optical measuring systems, such as depth cameras, the method is already integrated into the sensor. Optical measuring systems detect the envelope of an object with a very high spatial resolution. However, similar to the human eye, the resolution is limited by the field of view of the measuring system. Therefore, most of the generated data are also followed by post-processing solutions in order to complete and / or improve the detected envelope. A known solution for this purpose is found in

[17] .

[0029] Another disadvantage of optical measuring systems is that they do not provide direct additional velocity information within the preprocessing and are therefore designed only for stationary or at least quasi-stationary objects.

[0030] In order to be able to determine the movement of the envelope, for example, post-processing algorithms are additionally applied based on the data generated by the optical measuring system. If no further assumptions can be made about the visible object, the algorithm usually estimates the movement of the envelope by tracking the spatial points generated successively over time. A known approach to this is also found in

[17] . The accuracy of the tracking depends decisively on the relationship between the speed of the moving object and the measurement rate of the optical measuring system. Summary of the invention

[0031] Against this background, the invention is based on the object of providing a system for optimally detecting features of an object.

[0032] This object is achieved by the subject matter having the features of independent claim 1. Advantageous developments of the invention are the subject matter of the dependent claims.

[0033] Therefore, according to the present invention, the system includes a main modality, an auxiliary modality and a control device, wherein the main modality and the auxiliary modality are configured to respectively detect features, generate measurement data about the features, communicate with a control device configured for this purpose, be controlled by the control device configured for this purpose, and send the measurement data to the control device configured for this purpose, wherein the control device has the following mechanism, which is configured to control the main modality so that the measurement data generated by the auxiliary modality is taken into account, and the control device has the following mechanism, which is configured to obtain the features of the object from the whole or part of the whole of the generated measurement data.

[0034] It is preferably proposed that the characteristic of the object relates to the envelope of the object and / or the object is a biological body and / or the characteristic of the object is the position, velocity and / or acceleration of an area of ​​the object or the envelope of the object and / or phase information of a wave reflected by the area of ​​the object or the envelope of the object.

[0035] A control device is preferably understood to be any means, such as a circuit or a computer, which is designed to perform the functions of the control device or means of the control device attributed to the present invention. Thereby, the control device should preferably be designed broadly and, for example, also include a regulating device.

[0036] It is conceivable that the control device comprises means for inputting, displaying and / or outputting, in particular, ascertained characteristics of the object.

[0037] It is preferably provided that the control device has means which are designed to store the measurement data.

[0038] It can preferably be provided that the control device has means which are designed to compare a plurality of ascertained characteristics and / or generated measurement data and to determine therefrom comparison values ​​from which further characteristics, in particular position, velocity and / or acceleration, are derived.

[0039] In an advantageous embodiment, the primary modality comprises a radar system having at least one transmitting antenna and at least one receiving antenna, wherein preferably there is a mechanism which is designed to emit waves having less than 10, preferably exactly one, two or three different frequency support points by the transmitting antenna, and / or the secondary modality comprises a sensor, preferably for three-dimensional imaging. A frequency support point can also be referred to as or can be a frequency and preferably describes the temporal sequence of only one or more transmission frequencies.

[0040] It is conceivable that the antennas of the radar system form a MIMO aperture, wherein means are present which form a laterally focused image of the object by aperture synthesis.

[0041] Furthermore, it is conceivable that the main modality comprises a CW radar system, a stepped frequency radar system and / or a frequency shift keying radar system.

[0042] It is preferably provided that the auxiliary modality comprises an optical measuring system, in particular a camera, a depth camera, a stereo camera or a laser scanner.

[0043] It is preferably provided that the control device has means which are designed to optimize the ascertainment of the characteristics of the object with the aid of the characteristics and / or the measurement data.

[0044] It may be provided that the control device has means which are designed to use a statistical filter, preferably a Kalman filter, a kinematic model and / or methods known from information technology and / or which are designed to carry out a vectorial speed measurement.

[0045] It is preferably provided that the system is a component of a sports, training or fitness measurement or fitness information system, or that the system is used for medical, psychological, diagnostic or therapeutic purposes and / or for generating a digital human avatar.

[0046] The present invention also relates to a method for ascertaining features of an object by means of the system according to the invention, the method comprising the following steps:

[0047] a) detecting features by an auxiliary modality and generating measurement data about the features;

[0048] b) detecting the feature by means of a master modality, wherein the master modality is controlled by the control device in such a way that the measurement data generated in step a) are taken into account and measurement data relating to the feature are generated;

[0049] c) Extract features from the measured data.

[0050] It is preferably proposed that the characteristic of the object relates to the envelope of the object and / or the object is a biological body and / or the characteristic of the object is the position, velocity and / or acceleration of an area of ​​the object or the envelope of the object and / or phase information of a wave reflected by the area of ​​the object or the envelope of the object.

[0051] It is conceivable to store the generated measurement data.

[0052] It is conceivable that the method is performed again, wherein the characteristics determined in the first execution and / or the measurement data generated are compared with the characteristics determined in the second execution and / or the measurement data generated, and wherein comparison values ​​are determined therefrom, from which additional and / or improved characteristics, in particular position, velocity and / or acceleration, are derived.

[0053] Furthermore, it is conceivable that a wave having fewer than 10, preferably exactly one, two or three different frequency support points is emitted by the transmitting antenna via the main mode.

[0054] It is preferably provided that the method comprises a further step, in particular aperture synthesis, in which a laterally focused image of the object is generated.

[0055] It may be provided that the method is executed again and that features and / or measurement data from the previous execution are used for optimizing the execution.

[0056] It is preferably provided that in the method a statistical filter, preferably a Kalman filter, a motion model and / or methods known from information technology are used and / or a vectorial velocity measurement is performed.

[0057] It is preferably provided that the method is used in sports, training or fitness measurement or fitness information systems or for medical, psychological, diagnostic or therapeutic purposes and / or for generating a digital human avatar.

[0058] The invention also relates to a computer program comprising instructions which cause a system according to the invention to perform the method steps of the method according to the invention.

[0059] The invention also relates to a computer-readable medium on which a computer program according to the invention is stored.

[0060] A multimodal device for detecting the body envelope of a living being, in particular a human body envelope, and for determining the temporal movement of the envelope is preferably provided. Furthermore, an associated advantageous method for operating the device and for detecting the envelope and its movement is described.

[0061] A device for detecting a body envelope of a biological body may also be proposed, wherein the device comprises a main sensor modality, the main sensor modality including a radar system, the radar system having at least one transmitting antenna and at least one receiving antenna, the device comprises another auxiliary three-dimensional imaging sensor modality, and in a first step, at least a rough location and distance range are determined by the auxiliary sensor system, at least one point of the envelope to be detected is located in the at least rough location and distance range, and in a second step, an image of the at least one point of the envelope is reconstructed by the main sensor modality using at least one radar signal frequency.

[0062] It can be provided that in a third step the image of the at least one reconstructed point is compared with its image in a previous measurement with regard to phase, at least one phase difference value is determined therefrom, and the movement characteristic is derived from the phase difference value.

[0063] Furthermore, it can be provided that the main sensor modality is implemented as a CW radar, a stepped frequency radar and / or a frequency shift keying radar and that only a small number, preferably less than 10, particularly preferably only 1, 2 or 3, of different radar signal frequency support points are used per transmitting antenna during the measurement.

[0064] It can also be provided that the main sensor modality is equipped with a plurality of transmitting antennas and a plurality of receiving antennas, and that the antennas form a MIMO aperture, and that the aperture is used in the sense of aperture synthesis to reconstruct a transversely focused image of a point of the envelope.

[0065] Furthermore, it can be provided that the approximate location and distance range at which at least one point of the envelope of the object to be detected is located is determined by an optical measuring system, in particular by a depth camera, a stereo camera or a laser scanner.

[0066] It can also be proposed that, in addition to the first radar signal frequency, at least one further radar signal frequency is used to image the at least one point of the envelope, and the phases of the images obtained with the first radar signal frequency and with the at least one further radar signal frequency are compared with respect to phase values, and a distance value for at least one point of the body envelope is derived from at least one phase difference value obtained in this way.

[0067] An iterative process may be provided in which position and speed information of at least one previous measurement is used to improve the representation or image of the envelope and to improve the determination of the movement of the envelope.

[0068] It may be provided to use statistical filters, such as Kalman filters or the like, to use motion models and / or to improve methods known in information technology.

[0069] It is also conceivable to carry out a vectorial velocity measurement.

[0070] Sensor fusion of depth cameras and radars is also possible.

[0071] A system, device or method is conceivable, with the help of which the body envelope of a living body, preferably the body envelope of a person, or the temporal movement of the body envelope is determined by measurement technology and at least one result of the measurement is used for medical, psychological, diagnostic or therapeutic purposes.

[0072] Furthermore, a device is conceivable by means of which the body envelope of a living body, preferably a human body envelope, or the temporal movement of the body envelope is determined by measurement technology and at least one result of the measurement is used in a sports, training or fitness measurement or fitness information system.

[0073] Furthermore, devices are conceivable by means of which the body envelope of a living being, preferably the body envelope of a person, or the temporal movement of the body envelope is determined by measurement technology and at least one result of the measurement is used to generate a digital human avatar.

[0074] The invention solves many of the limitations existing in the prior art. The invention also relates to a device for contactlessly and precisely detecting the velocity vector of an envelope of an object and of one, several or every point on the envelope and to an advantageous method for operating the device. For this purpose, a radar-based main modality is preferably combined with a 3D imaging auxiliary modality and the advantages of the sensor are combined. As a result, the envelope and its movement are preferably detected with relatively little effort and with high spatial resolution and with excellent velocity accuracy in all three spatial directions and at a high measurement rate.

[0075] It is preferably unimportant for the invention which radar-based reconstruction method is used. It is also preferably unimportant for the invention which optical reconstruction method is used.

[0076] The term envelope or body envelope is preferably understood as a boundary surface or the sum of a plurality of coupled boundary surfaces, which establishes a boundary between the inside and the outside in the body of an organism. The outside is usually the air or the atmosphere surrounding the organism, and the inside is preferably the body or the material structure of the body.

[0077] In many animals and in humans, the envelope or body envelope can preferably be defined by the skin surface, but this is only one possible definition since, depending on the application, it may make sense for clothing, hair, dirt or other attachments on the body or other objects that are fixedly connected to the body to belong to or not to the envelope.

[0078] Preferably, the envelope is also the envelope of other objects bounded by the envelope. In this context, mention should be made in particular of humanoid robots or bionic robots, ie robots inspired by other organisms, or also operating or production robots. Likewise, technical mobile systems, in particular motor vehicles, transport systems, construction machines and mobile robots, preferably have an envelope, wherein the body or chassis usually forms the envelope.

[0079] Detecting the envelope is preferably understood to be a measuring process in which the geometrical course of the envelope is determined. In particular, the detection process is preferably an imaging process in which a geometrically correct image of the envelope is generated. For this purpose, the spatial position of a plurality of points of the envelope is preferably determined, and at least some points of the point group, but preferably all points, are also associated with further features, such as a certain reflectivity property, such as the intensity of the reflection, the frequency dependence of the reflection, which is referred to as color in the optical range, or the polarization reflection behavior. The detection of the envelope preferably also includes determining the position of the points of the envelope relative to a reference coordinate system.

[0080] If the motion of the envelope of a body or object is determined, at least some of the points of the aforementioned point group are also associated with at least one motion feature. The motion feature can be a scalar velocity, a vector velocity, a one-dimensional or multi-dimensional acceleration variable, or a higher order complex multi-dimensional motion equation.

[0081] Preferably, the detection of the envelope and / or the determination of the movement of the envelope is divided into two categories: pre-processing and post-processing. The pre-processing and post-processing differ in the data pool used.

[0082] Preprocessing operates on data generated directly by one or more sensors or modalities. Preprocessing produces a temporary application-dependent data pool. When detecting an envelope, preprocessing produces a temporary geometric orientation of the envelope. When determining the motion of the envelope, preprocessing produces a temporary motion signature.

[0083] As the name already indicates, post-processing is arranged in time after pre-processing and expands the temporary data pool with further information. This can be, for example, a correction or completion of the previous data. Post-processing related to the detection of the envelope can, for example, compensate or adapt the previously determined geometrical course so that it approximates the real envelope better. In particular, post-processing can also be used to complete the envelope, which can only be partially determined in the pre-processing, because, for example, the back side of the object cannot be determined due to the shielding of the sensor device used. Post-processing for determining the movement of the envelope can also be used to correct or refine the temporary motion characteristics. In particular, if the post-processing for determining the movement of the envelope is combined, for example, with the post-processing for detecting the envelope, the temporary motion characteristics can expand the temporary motion characteristics associated with the point group with another point group with motion characteristics.

[0084] The body envelope and its movements are preferably detected by at least two different wave-based sensor modalities and their sensor data fusion. A wave-based sensor modality is preferably understood to be a measuring device which uses waveforms to detect measurement information, for example images of the body.

[0085] Waveforms include electromagnetic waves in the microwave or optical range or sound / ultrasound waves. The modes are preferably defined in such a way that they can be distinguished not only by the waveform, such as radar or optical or ultrasonic, but also by the measuring principle, such as narrowband CW radar or N-FSK radar or wideband FMCW radar or FSCW radar.

[0086] As a preferred multimodal implementation, it is preferably proposed to combine at least one radar-based modality or main modality with at least one other auxiliary modality. As an auxiliary modality, a wave-based three-dimensional imaging modality, in particular a three-dimensional optical imaging technique, is preferably proposed, preferably implemented with the aid of a depth camera, a stereo camera or a laser scanner. However, a broadband radar system can also be used as an auxiliary modality, but the difference between the broadband radar system and the radar-based main modality is that the broadband radar system provides three-dimensional high-resolution imaging in each measurement cycle and is significantly more expensive and can only be measured significantly slower than the main modality. As an auxiliary radar modality, FMCW, FSCW, OFDM or pulse radars for imaging as known from the prior art are considered. On the contrary, the main radar modality is preferably designed so that the main radar modality is optimized for detecting dynamic motion.

[0087] The primary radar modality is preferably a narrowband continuous wave radar or a continuous wave (CW) radar or a so-called stepped frequency radar (FSCW / SFCW), or an FSK (frequency shift keying) or N-FSK (narrowband frequency shift keying) radar. Preferably, the radar is implemented as a multiple-input multiple-output (MIMO) radar with a plurality of transmitting and receiving antennas. Preferably, only a small number of different frequency support points, preferably only a number less than 10, are emitted via the transmitting antenna, in particular, preferably only one frequency support point, as in a CW radar, only 2 different frequency support points, as in a 2-FSK radar, or only 3 different frequency support points, as in a 3-FSK radar. Therefore, the primary radar modality preferably has a good lateral resolution, i.e., observed perpendicularly to the wave propagation direction, which is determined by the number of transmitting and receiving antennas or the size of the resulting synthetic aperture, and preferably has only a limited axial resolution in the range direction, i.e., in the wave propagation direction, or even no axial resolution at all. The ability to separate different targets in the range direction, i.e., the axial resolution, is preferably determined by the bandwidth of the radar signal used. Since a large signal bandwidth, in particular in MIMO radar systems, complicates and in particular can slow down the measurement process, in the main radar modality a large bandwidth and thus also a good resolution in the range direction and thus also the ability to generate good three-dimensional images is preferably deliberately abandoned.

[0088] Depending on the sensor device or modality used, the general approach to reconstructing the image of the object also differs. Therefore, the overall reconstruction method for optical measuring systems, which is preferably referred to as optical reconstruction method or optical reconstruction, and the overall reconstruction method for radar-based measuring systems, which is preferably referred to as radar-based reconstruction method or radar-based reconstruction, are preferably clearly distinguished from each other conceptually.

[0089] It should be noted here that the terms "a" and "an" do not necessarily denote exactly one of the elements, although this is a possible embodiment, but may also denote a plurality of the elements. Likewise, the use of the plural also includes the presence of the elements involved in the singular, and vice versa, the singular also includes a plurality of the elements involved. Furthermore, all features of the invention described herein may be claimed in any combination with one another or in isolation from one another. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Further advantages, features and effects of the present invention are apparent from the following description of preferred exemplary embodiments with reference to the accompanying drawings, in which identical or similar components are denoted by the same reference numerals. Here, the following is shown:

[0091] Figure 1 One embodiment of the system according to the invention is shown.

[0092] Figure 2 A further embodiment of a possible configuration of the modalities of the system according to the invention is shown.

[0093] Figure 3 An embodiment of a possible representation of the modalities of the system according to the invention is shown.

[0094] Figure 4 A further embodiment of the system according to the invention is shown.

[0095] Figure 5 Two matrices are shown with the phase information or image phase of each pixel.

[0096] Figure 6 A flow chart showing a preferred method. DETAILED DESCRIPTION

[0097] exist Figure 1 , a combination of a main modality 20 together with an auxiliary modality 30 for determining the envelope of an object 10 and its movement is shown.

[0098] Object in Figure 1 , moves with a velocity vector V with reference numeral 11 , which can be decomposed into Cartesian components V1 and V2 , for example.

[0099] The velocity vector V can also be a three-dimensional vector in this case, which can be decomposed into three Cartesian components.

[0100] The primary modality 20 and the secondary modality 30 are connected to a computer 40 , which can be connected to a monitor 41 , on which algorithms for imaging and determining the movement of the object 10 are executed.

[0101] In this case, in particular the individual orientations of the two modes 20 and 30 are linked to one another.

[0102] For example, modes 20 and 30 can be Figure 2 The configurations shown in FIG. 2 are related to one another. The modalities 20 and 30 detect the object 10 to be recorded. The orientation is shown by an unlabeled coordinate system, also in the subsequent figures.

[0103] exist Figure 2 In the configuration shown on the left, the two modalities 20 and 30 are inserted into each other so that they have a common orientation, that is to say at least known relative to each other, a priori. Alternatively, the two modalities 20 and 30 can be placed offset and the relative orientation relative to each other can be calculated later. For example, the modalities 20 and 30 can be offset next to each other, such as from Figure 2 Likewise, the modes 20 and 30 can be oriented in an offset manner one above the other, as shown in FIG. Figure 2 In this case, next to each other and one above the other preferably refer to the orientation relative to the object 10 .

[0104] In this case, an imaging MIMO-FSK radar as the main modality 20 is preferably combined with a depth camera as the auxiliary modality 30, which are inserted into each other, as shown in FIG. Figure 3 The MIMO-FSK radar or main mode 20 preferably has a receiving antenna 21 and a transmitting antenna 22. The two modes 20 and 30 are preferably arranged in such a way that the data generated by the two modes overlap as extensively as possible in time and space, so that the data can be fused and combined with each other as accurately as possible.

[0105] exist Figure 4 , an imaging MIMO-FSK radar as main modality 20 and a depth camera as auxiliary modality 30 are now shown, which are oriented offset one above the other. The MIMO-FSK radar or main modality 20 preferably has a receiving antenna 21 and a transmitting antenna 22 .

[0106] Figure 4 The object 10 in FIG. 1 is, for example, a moving person or body part. Each point of the person or body part is Figure 4In each case, the vehicle moves with a velocity vector V having the reference numeral 11 , which can be broken down into Cartesian components V1 and V2 .

[0107] The depth camera or auxiliary modality 30 is connected to a computer 40 via a connection 35 , via which distance features of the object 10 detected by the depth camera or auxiliary modality 30 are transmitted.

[0108] The MIMO-FSK radar or master modality 20 is connected to a computer 40 via a connection 25 , via which the speed and distance characteristics of the object 10 detected by the MIMO-FSK radar or master modality 20 are transmitted.

[0109] The computer 40 may be connected to a monitor 41 , on which the results of the algorithms for imaging and determining the movement of the object 10 executed on the computer 40 may be displayed.

[0110] Without further combining the individual modalities 20 and 30 , these firstly provide the following data.

[0111] The main mode 20 first provides the radial velocity by evaluating the phase information that changes over time. Here, a continuous wave (CW) signal is first considered as a suitable signal form. The continuous wave signal can be composed of only one frequency support point or of multiple frequency support points. When using only one CW frequency, the velocity and the relative spacing change can be measured. The absolute distance can be determined in a relatively small range of uniqueness based on the periodicity of the phase only when the relevant period or the relevant distance range represented by the period is known, wherein each period is respectively from 0° to 360°.

[0112] In the case of a signal with multiple frequency support points, an absolute distance measurement with a larger range of unambiguity is additionally possible. In particular, in a preferred configuration of the main mode 20 in the form of an FSK-MIMO radar using at least two frequency support points, an image of the recorded envelope of the object 10 is measured. In addition, the velocity of the local distribution of the envelope can be determined by a suitable preprocessing method. This can be a comparison of successive radar images in the simplest case. Preferably, either the complex phase of the successive images is evaluated or the continuous change of the received phase is evaluated.

[0113] On the other hand, the auxiliary modality 30 firstly provides locally resolved distance data of the recorded envelope of the object 10. The reconstruction method for generating the distance data is preferably unimportant in this case.

[0114] By combining the previously separate modalities, for example in what is known as sensor data fusion, new and larger data sets can be created with greater accuracy. However, post-processing methods for completely detecting the envelope of object 10 can also thereby be used more quickly and more efficiently.

[0115] As a first step of sensor data fusion, the detected object 10 is roughly located in space by means of the auxiliary modality 30 and the region in which the envelope of interest is located is defined. Based on this, a radar-based reconstruction is performed in said region according to known methods. By limiting the radar-based reconstruction to selected points in the relevant region, the radar-based reconstruction is significantly accelerated in its processing speed. In addition, the already performed localization of the object 10 allows the use of narrowband radar signal forms, which have no range resolution or only a very low range resolution. The generation of said narrowband radar signal forms can be carried out much faster than the broadband signal forms used hitherto in radar imaging, which is advantageous in time-critical applications.

[0116] If the main modality 20 uses at least two frequency support points, then in addition to the existing data of the auxiliary modality 30 , a spatially resolved image of the envelope can also be generated.

[0117] In addition to the known methods, a new scheme based on two frequency support points is preferably used.

[0118] In the novel approach, a frequency shift keying approach as known from

[18] is combined with a radar imaging method customary to date. The disclosure content of

[18] is hereby incorporated in its entirety into the present description.

[0119] Based on the first rough positioning by the auxiliary modality 20, the distance or z-coordinate d estimated by the auxiliary sensor device or modality 20 is determined according to known methods using two closely adjacent frequencies f1, f2. s The two 2D images of the envelope are reconstructed at the starting point of the reconstruction. Two CW baseband signals s b,f1 and b,f2 The two CW baseband signals are at two frequency support points f1 and f2 by means of The transmitting antenna and the location The receiving antenna detects and passes the The following formula shows one possibility of how to obtain the CW baseband signal.

[0120]

[0121] in

[0122] Here c is the speed of light. The following two formulas describe the distance d for two frequency support points and the estimated distance d s Two hypotheses are drawn.

[0123]

[0124]

[0125] Estimated distance d s is the sum of the actual distance d and the positive or negative differential distance Δd. This leads to the assumptions shown in the following two formulas.

[0126]

[0127] In case the baseband signal is correlated with the complex conjugate hypothesis according to known methods for testing signal hypotheses, two correlated signals are obtained, as shown in the following two formulas.

[0128]

[0129] exist Figure 5 The estimated distance d for two frequency support points is shown in s Two matrices with phase information or image phase for each pixel of the 2D reconstruction at , wherein the left matrix has the image phase for the frequency support point with frequency f1, and the right matrix has the image phase for the frequency support point with frequency f2.

[0130] For each image point of the estimated position or estimated envelope, a correlation is performed on all transmit-receive combinations and added. Thus, two complex radar images are obtained at two different frequencies f1 and f2. By forming the difference phase of the two images And with the aid of the known frequency difference Δf=f2−f1, Δd can be calculated assuming that the target is at the corresponding image point, so that the exact distance of the envelope relative to the radar is calculated for each image point, as is shown in the following formula.

[0131]

[0132] When at least two frequency reference points are used, for example according to the FSK-MIMO principle, in the case of the main modality 20 for generating a spatially resolved image of the envelope, the image is subsequently compared with the data of the auxiliary modality and optionally supplemented and fine-tuned. This step is omitted when only one frequency is used. In this case, in particular, the imaging of different boundary surfaces of the envelope can be generated by using different wavelengths, for example by a combination of a radar-based main modality 20 and an optical auxiliary modality 30, since, for example, radar can penetrate materials such as clothing or paper, while optical measuring systems, on the other hand, cannot.

[0133] Finally, the described measurement configuration of the two modalities 20 and 30 and the above-described method for sensor data fusion results in a new data pool which, as information per measuring cycle, has the most accurate possible spatially resolved image of the envelope together with locally distributed velocity information.

[0134] The pre-processing method described above first only provides data for each measurement cycle. The post-processing method based on this estimates the continuous movement of the recorded envelope over a longer period of time. As mentioned above, direct speed information, preferably resolved at a certain point, cannot be obtained in the methods in optical imaging technology to date. Therefore, in the post-processing method for estimating the speed, the quality is decisively affected by the measurement rate of the optical measurement system or the auxiliary modality 30. In order to be able to track the movement in time, it is preferred to first find the consistency between two consecutive recordings so that the movement patterns of the respective consistency can be estimated subsequently. The larger the spacing between the individual measurement cycles of the modality used, the larger the search space is in order to find the consistency and thus to obtain the movement performed between them.

[0135] Due to the availability of additional speed information, the search space can now be significantly reduced so that the consistency can be determined more quickly and more accurately.

[0136] Further approaches for reducing the search space include, for example, the use of application-dependent models which estimate both the envelope of recorded object 10 and also the movement of recorded object 10 in advance.

[0137] Additionally, the initially estimated speed information can be corrected subsequently by using suitable statistical methods, such as a Kalman filter.

[0138] The imaging of a plurality of boundary surfaces by means of a combination of a radar-based measuring system and an optical measuring system, in addition to additional speed information, can be particularly advantageous when the recorded object 10 has a plurality of differently moving envelopes.

[0139] This is the case, for example, when estimating the movement of the body envelope of a person, since in this case additional layers of clothing would obscure the movement of the human body in an optical measuring system.

[0140] The described method advantageously enables precise, high-performance determination of the motion of the envelope of any object with the potential for real-time capability.

[0141] The method can preferably be applied to the movement of the human body in the field of medical diagnosis and treatment or in the field of nursing and in the field of fitness or sports, because, as mentioned above, in addition to the additional speed information, further advantages are obtained in the sensor data fusion. It is also conceivable that the method is used to generate a digital human avatar.

[0142] Figure 6 The overview is shown in the form of a flow chart of a preferred method for operating a preferred configuration.

[0143] First, the recording is started in step S1. Then, in step S2, it is inquired whether the modalities have the same orientation. If not, a calibration is performed in step S3. After the calibration or when the modalities have the same orientation, an optical reconstruction of the object or the envelope is performed in step S4 by means of the auxiliary modality. Depth data D1 are generated in this process. In step S5, the depth data are used to locate the object or the envelope. In step S6, a radar-based reconstruction of the object or the envelope is performed by means of the main modality. This results in data D2 with the speed and / or distance of the object or the envelope. In step S7, the data D1 and D2 are fused in the process of sensor data fusion. This results in combined data D3 in the current recording. In step S8, the movement of the object or the envelope is determined. In this case, the data D3 and the data D4 from the previous recording are used as input data, based on which data D5 of the movement of the object or the envelope are generated from the current recording. In step S9, it is inquired whether there are further recordings. If there are further recordings, step S8 is repeated. If there are no further recordings, the recording is ended in step S10.

Claims

1. A system for obtaining characteristics of an object, It is characterized in that The system includes a main modality, an auxiliary modality, and a control device, wherein the main modality and the auxiliary modality are configured to respectively detect the features, generate measurement data about the features, communicate with the control device configured for this purpose, be controlled by the control device configured for this purpose, and send the measurement data to the control device configured for this purpose, wherein the control device has the following mechanism, which is configured to control the main modality so that the measurement data generated by the auxiliary modality is taken into account, and the control device has the following mechanism, which is configured to determine the features of the object from the whole or part of the whole of the generated measurement data.

2. The system according to claim 1, It is characterized in that The characteristic of the object relates to the envelope of the object, and / or the object is a biological body, and / or the characteristic of the object is the position, velocity and / or acceleration of a region of the object or the envelope of the object and / or phase information of a wave reflected by the region of the object or the envelope of the object.

3. The system according to claim 1 or 2, It is characterized in that The control device has means designed to store measurement data.

4. A system according to any one of the preceding claims, It is characterized in that The control device has means which are designed to compare a plurality of ascertained characteristics and / or generated measurement data and to determine therefrom comparison values ​​from which further characteristics, in particular position, velocity and / or acceleration, are derived.

5. A system according to any one of the preceding claims, It is characterized in that The main modality comprises a radar system having at least one transmitting antenna and at least one receiving antenna, wherein preferably there is a means which is designed to emit waves having less than 10, preferably exactly one, two or three different frequency support points, from the transmitting antenna, and / or the auxiliary modality comprises a sensor which preferably performs three-dimensional imaging.

6. The system according to claim 5, It is characterized in that The antennas of the radar system form a MIMO aperture, wherein means are provided for generating a laterally focused image of an object by aperture synthesis.

7. A system according to any one of the preceding claims, It is characterized in that The main modes include a CW radar system, a stepped frequency radar system and / or a frequency shift keying radar system.

8. A system according to any one of the preceding claims, It is characterized in that The auxiliary modalities include optical measuring systems, in particular cameras, depth cameras, stereo cameras or laser scanners.

9. A system according to any one of the preceding claims, It is characterized in that The control device has means which are designed to optimize the ascertainment of the characteristics of the object with the aid of the characteristics and / or the measurement data.

10. A system according to any one of the preceding claims, It is characterized in that The control device has means which are designed to use statistical filters, preferably Kalman filters, kinematic models and / or methods known from information technology and / or which are designed to carry out vectorial speed measurement.

11. A system according to any one of the preceding claims, It is characterized in that The system is a component of a sports, training or fitness measurement or fitness information system, or the system is used for medical, psychological, diagnostic or therapeutic purposes and / or for generating a digital human avatar.

12. A method for ascertaining features of an object by means of a system according to any one of the preceding claims, the method comprising the following steps: a) detecting the feature by means of the auxiliary modality and generating measurement data about the feature; b) detecting the feature by means of the master modality, wherein the master modality is controlled by the control device in such a way that the measurement data generated in step a) are taken into account and measurement data relating to the feature are generated; c) Determining the feature from the measurement data.

13. The method according to claim 12, It is characterized in that The characteristic of the object relates to the envelope of the object and / or the object is a biological body and / or the characteristic of the object is the position, velocity and / or acceleration of a region of the object or the envelope of the object and / or phase information of a wave reflected by the region of the object or the envelope of the object.

14. The method according to claim 12 or 13, It is characterized in that The resulting measurement data is stored.

15. The method according to any one of claims 12 to 14, It is characterized in that The method is performed again, wherein the characteristics determined in the first execution and / or the measurement data generated are compared with the characteristics determined in the second execution and / or the measurement data generated, and wherein comparison values ​​are determined therefrom, from which additional and / or improved characteristics, in particular position, velocity and / or acceleration, are derived.

16. The method according to any one of claims 12 to 15, It is characterized in that By means of the main mode, a wave having less than 10, preferably exactly one, two or three different frequency support points is emitted by the transmitting antenna.

17. The method according to any one of claims 12 to 16, It is characterized in that The method comprises further steps, in particular aperture synthesis, in which a laterally focused image of the object is generated.

18. The method according to any one of claims 12 to 17, It is characterized in that The method is executed again and features and / or measurement data from the previous execution are used to optimize the execution.

19. The method according to any one of claims 12 to 18, It is characterized in that In the method, statistical filters, preferably Kalman filters, motion models and / or methods known from information technology are used and / or a vectorial velocity measurement is performed.

20. The method according to any one of claims 12 to 19, It is characterized in that The method is used in sports, training or fitness measurement or fitness information systems or for medical, psychological, diagnostic or therapeutic purposes and / or for generating digital human avatars.

21. A computer program comprising instructions causing a system according to any one of claims 1 to 11 to perform the method steps of any one of claims 12 to 20.

22. A computer readable medium having stored thereon the computer program according to claim 21.

Images