System and method for tracking deformations

Abstract

An imaging system for reconstructing reflectivity images of a scene including an object moving within the scene. A tracking system is used to track the deformations of the object to estimate the deformations of the object for each time step. A sensor acquires snapshots of the scene to produce a set of measurements of the object having a deformed shape over the plurality of time steps, each snapshot of the acquired object including a measurement of the object deformation for that time step. Corrections to the estimate of the object deformation for each time step are computed, matching the measurement of the object deformation corrected for each time step to the measurement in the snapshot of the object acquired for that time step. From the distance between the corrected deformation and the deformation estimate, the corrected deformation for each time step is selected that is better than the other corrected deformations to obtain a final estimate of the deformations of the deformable object moving in the scene.

Description

Technical Field

[0001] This disclosure generally relates to sensing systems, and more particularly to sensing deformable objects moving in a scene. Background Technology

[0002] In some remote sensing applications, acquiring high-resolution radar imagery is essential to meet specific application and operational requirements. For example, radar reflectivity imaging is used in various security, medical, and through-wall imaging (TWI) applications. While longitudinal resolution is primarily controlled by the bandwidth of the transmitted pulse, transverse (azimuth) resolution depends on the aperture of the radar sensor. Generally, regardless of whether the aperture is physical (large antenna) or synthetic (moving antenna), a larger aperture generally results in higher image resolution. Currently, the increase in antenna physical size has led to a significant increase in the cost of radar systems. Therefore, some radar imaging systems use synthetic aperture methods to reduce antenna size and decrease the cost of radar imaging. For example, synthetic aperture radar (SAR) and inverse synthetic aperture radar (ISAR) utilize the relative motion between the radar antenna and objects in the scene to provide finer spatial resolution with relatively small physical antennas (i.e., smaller than the antennas of beam-scanning radar).

[0003] However, the small size of the physical antenna in radar systems makes it difficult to track deformable moving objects. Specifically, tracking objects exhibiting arbitrary motion and deformation requires tracking sensitivity with a minimum resolution greater than the physical antenna resolution, making radar imaging systems impractical in terms of cost. Therefore, conventional radar and / or other electromagnetic or acoustic imaging systems require objects to be stationary in the scene or to move with highly controlled, rigid motion. Even with rigid motion, conventional radar imaging systems require challenging tracking steps to estimate the motion parameters of the moving object using only radar data before forming a radar image; see, for example, Martorella 2014 (Martorella, M (2014), "Introduction to inverse synthetic aperture radar"), Academic Press Library in Signal Processing (Vol. 2, pp. 987-1042), Elsevier.

[0004] Therefore, there is a need for imaging systems and methods suitable for determining the errors in the estimated deformation of signals caused by unknown deformations or other transformations or corrections that may affect the signal during the acquisition process. Summary of the Invention

[0005] This disclosure relates to an acquisition system that includes a sensing system for acquiring signals under unknown arrangements, such as sensing deformable objects moving in a scene.

[0006] Some implementations relate to imaging systems, and more specifically to imaging systems that image a deformable object in motion or undergoing deformation when the deformable object is acquired using one or more snapshots. In these implementations, the imaging system can reconstruct an image of the object under one or more deformations and can represent the object in its prototype deformation.

[0007] In some implementations, the imaging system may include one or more sensors such as cameras, depth cameras, radar, magnetic resonance imaging (MRI), ultrasound, computer-aided computed tomography (CAT), lidar, terahertz, and hyperspectral imaging. One or more of these sensors may be used to track deformation, and one or more may be used to image the object. In some implementations, the same one or more sensors may be used to achieve both tracking and imaging.

[0008] Some embodiments provide an imaging system, such as including an optical camera and a depth sensor, capable of tracking the motion of an object even if the object is deformable and the motion is not rigid. Some embodiments also provide a second imaging system (e.g., a radar or ultrasonic array) that images the object as it moves and deforms within a scene. The second imaging system reconstructs an image of the object moving within the scene with a resolution greater than that limited by the actual size of the physical sensor (e.g., an array of electromagnetic or ultrasonic sensors acquiring reflectivity images).

[0009] Some implementations provide a radar imaging system suitable for airport security applications that allows a person to move freely in front of the radar imaging system while the system reconstructs a radar reflectivity image of that person. Some sensor types used to collect image data include optical sensors (such as monochrome or color or infrared cameras), depth cameras, or combinations thereof. Optical sensors are less expensive than electromagnetic sensors, and their modal operation makes target tracking easier. Therefore, even if the target is deformable and the motion is not rigid, optical sensors can be used to track the target's movement.

[0010] Furthermore, some implementations are based on the understanding that in applications where radar imaging of deformable objects is necessary and useful, the object's movement is close enough and visible to the radar imaging system that the optical sensor can provide sufficient accuracy for tracking. Among these, some implementations are based on the understanding that by using optical motion tracking to assist radar reconstruction, the radar imaging system can image very complex moving targets.

[0011] An example where the target is clearly visible is a security application where a person walks in front of the scanning system, such as in an airport. In some airport security scanners, the subject is required to stand in a specific posture for scanning prohibited items. A scanning system according to one embodiment allows a subject (a deformable, mobile object, such as a person) to simply walk past the scanner while being scanned, without needing to stop.

[0012] Some embodiments of this disclosure include a radar imaging system configured to determine a radar reflectivity image of a scene, the scene including objects moving with the scene. The radar imaging system includes optical sensors for tracking the objects over a period of time to generate deformations of the objects for each time step. The radar imaging system may also include one or more electromagnetic sensors (such as millimeter-wave sensors, terahertz imaging sensors, or backscattered X-ray sensors, or combinations thereof) to acquire snapshots of the objects at multiple time steps. Each snapshot includes a measurement of a radar reflectivity image representing the object having a deformed shape defined by the corresponding deformation. It is recognized that one of the reasons hindering the electromagnetic sensors of a radar imaging system from tracking moving objects is the resolution of electromagnetic sensing, constrained by the physical size of the sensor's antenna. Specifically, for practical reasons, the antenna size of a radar imaging system only allows for estimating a coarse image of the object at each time step. This coarse image is suitable for tracking objects undergoing rigid and finite transformations, but cannot identify arbitrary non-rigid transformations typical of human motion.

[0013] Other embodiments of this disclosure are based on another understanding: radar imaging systems can share measurements of a scene acquired at multiple time steps. Such a measurement system can be used to improve the resolution of the radar reflectivity image beyond the resolution limited by the size of the radar imaging system's antenna. However, as an object moves through different time steps, it can be located at different positions and can have different shapes due to non-rigid motion. This misalignment and deformation of the object makes the measurement system ambiguous (i.e., ill-posed) and difficult or impossible to resolve. In particular, a non-rigidly moving object will have different shapes in different time instances. Therefore, the object shape may have different deformations relative to its nominal shape at different time steps, and the radar reflectivity image observed by the radar imaging system may have different transformations relative to the object's radar reflectivity image.

[0014] Other embodiments of this disclosure are based on another understanding: the imaging system may be mounted on a moving platform, acquiring snapshots of its surrounding environment as it moves, and the deformation of its input is caused by changes in the geometry of the environment as the imaging system moves with the platform. Therefore, each snapshot of the environment includes deformation, and the deformation itself provides information about the motion of the sensor and the moving platform within the environment. Furthermore, a coarse or more precise determination of the deformation can typically be achieved using one of the many methods in the art, collectively known as Simultaneous Localization and Mapping (SLAM) methods. Embodiments of this disclosure can be used to refine the output of methods that completely replace SLAM methods.

[0015] For example, some embodiments of this disclosure use existing SLAM algorithms in the art to compute deformation estimates of the scene observed by the sensor. This estimate is refined such that the data acquired by the sensor in each snapshot is matched when the refined deformation estimate is applied.

[0016] Among the many problems in this field, including SLAM, labelless sensing, and imaging of moving deformable objects, there is the problem of recovering signals from measures that have undergone unknown perturbations. Some embodiments of this disclosure are based on the understanding that, in most practical arrangements, unknown permutations are not arbitrary, but some unknown permutations are more likely to occur than others.

[0017] Based on this understanding, and to further utilize this, some embodiments of this disclosure include a regularization function that promotes more likely permutations in the solution. Through experimentation, the inventors have learned from this approach that, even if the general problem is not convex, by appropriately relaxing the resulting regularization, the well-developed mechanisms of optimal transport theory (OT) can be utilized, and a practical algorithm can be developed.

[0018] A key understanding that allows for the development of feasible algorithms using OT (Operational Technology) is that an unknown deformation or arrangement of one signal to another is equivalent to transmitting nominal quality between pixels, and the quality transmitted from one signal to another is inducing a deformation of the signal. Therefore, OT theory can guide this transmission to occur optimally, that is, to interpret and restore the optimal deformation or arrangement of the acquired snapshot.

[0019] A further understanding is that the concept of transmission cost inherent in OT theory can be used to provide regularization that favors more probable permutations or variations. Specifically, Optimal Transmission (OT) theory identifies a quality transmission plan that is optimal when considering the total cost of transmitting quality, where the cost of transmitting quality from one pixel to another can be determined by the application. If one variation of the signal is more likely to occur than another, then the corresponding total transmission cost per pixel in that variation is lower than the corresponding cost in the less likely variation. Therefore, with the help of OT restoration theory and algorithms, transmissions corresponding to more probable variations are preferred.

[0020] Another understanding is that in some practical applications, the most likely deformations and arrangements are those where pixels are not transported to locations very far from their original positions. Therefore, the cost of moving a pixel to a nearby location is lower than the cost of moving it to a more distant location. Thus, a regularization can be used where the transfer cost penalizes the quality of moving closer objects less than the quality of moving farther objects. Since this transfer cost is well-studied in the OT field, this understanding allows for the use of well-developed OT algorithms to estimate OT plans.

[0021] A similar understanding is that in some other practical applications, the most likely variations and arrangements are those where pixels are not transported to locations very far from where their neighboring pixels are transported. Therefore, if nearby pixels are also moved to the vicinity of the new location, the cost of moving a pixel to the new location is lower than the cost of moving it to a location farther from where its neighboring pixels are moved. Thus, the transfer cost, where the penalty for moving together is less than the penalty for moving separately, can be used as regularization. Since this transfer cost is also well studied in the field of optimal transport (OT), this understanding allows for the use of better-developed OT algorithms to estimate OT plans.

[0022] Another key understanding is that some distortions may involve partial signal occlusion, and different snapshots may exhibit different distortions, including varying degrees of signal occlusion. Furthermore, certain occlusion patterns are more likely to occur than others. For example, because nearby pixels of an object move together, they are more likely to be occluded along with another part of the object. For instance, a person walking in front of a camera might swing their arms as part of their walking motion. In this case, the entire arm, away from the camera, is likely to be occluded by the body. Additionally, when the arm moves behind the body, nearby points on the arm are likely to be occluded together. The closer these points are, the more likely they are to appear simultaneously behind the body.

[0023] In these cases, optimal transmission (OT) theory allows for the inclusion of additional costs in the total cost when adding or removing quality from a signal. This subfield is sometimes referred to as unbalanced OT or partial OT within the OT domain. However, existing methods in the art do not account for the fact that nearby quality (i.e., pixels) are more likely to appear or disappear together than quality that is not together. For this reason, some embodiments of this disclosure may introduce different costs when planning structures that include the quality difference between two deformations, to reduce the deformation cost of nearby pixels appearing or disappearing together, thus considering that such deformations are more likely to occur than deformations in which the appearing and disappearing pixels are not nearby.

[0024] Some embodiments of this disclosure include systems and methods for determining signals observed using multiple snapshots, each of which has undergone a different permutation. These systems and methods utilize knowledge that certain permutations are more likely than others to efficiently determine the signals. Thus, by incorporating this knowledge into the solution using optimal transmission theory, these systems and methods can determine unknown signals more efficiently than conventional imaging system methods.

[0025] For example, some testing methods involve using an alternative mode to track deformable objects. Other testing methods include multimodal imaging systems for deformation, which assume that one mode is used to determine the deformation and another mode is used for imaging. These methods are taught that deformation introduces errors when it is determined, and they provide some simple ways to correct these induced errors. Unfortunately, further testing revealed that these methods do not work well at all. For example, the mode used to track the object does not have the resolution required by the radar system, and tracking is prone to errors. Reconstruction is imprecise in this approach. Therefore, it is desirable, if possible, that the imaging process also refines the tracking and corrects for inaccuracies in estimating object deformation. Based on this finding, the systems and methods of this disclosure must perform much better in correcting deformation errors.

[0026] Some testing methods include applications that involve the necessary steps of reconstructing a signal observed through multiple snapshots, each of which has undergone unknown or partially known deformations or perturbations (i.e., signal permutations). In this case, the goal is not only to reconstruct the signal but also to restore the permutation. This has proven to be a difficult problem because the number of possible permutations grows exponentially with the size of the signal. In some testing applications, certain permutations are more likely than others. It is hoped that this information can be utilized to reduce the difficulty of the problem. However, under the current state of technology, it is not yet known how to effectively utilize this information.

[0027] Other testing methods have also been developed for analyzing the imaging of deformable moving objects using inverse synthetic aperture radar (ISAR). However, it was later found that these systems cannot account for object deformation, such as the movement of a person's hand while walking or the beating of a person's heart. It was also learned that these testing methods do not account for errors in the object's motion model. To handle these errors, these testing methods must employ computationally expensive techniques or be robust to errors, such as incoherent imaging in radar conditions, which would compromise image quality.

[0028] Other testing methods include reconstructing the signal observed through an unknown permutation. However, during testing, the inventors recognized that no known method discloses the reconstruction of a permutation signal measured by a measurement system. In these specific testing methods, it is clear that known methods only consider the signal of the permutation as directly observed. The inventors recognized that adding a measurement system is not significant because the measurement system incorporates the elements of the permutation signal. If the elements of the signal are incorporated into the measurement by the measurement system of some embodiments of this disclosure, then some of the methods used in these specific testing methods simply do not work. Furthermore, these specific methods used in the testing methods cannot utilize knowledge about the permutation matrix, i.e., that permutations that move image pixels closer together are more likely to occur than permutations that move image pixels further apart.

[0029] Some of the testing methods allow for corrections to the deformation using measurements. However, in these test cases, the calculations are simplistic and often fail. The result from these test cases is that some embodiments of this disclosure utilize formulas that provide accurate estimations of the deformation and its corrections using optimal transport theory and algorithms.

[0030] Some testing methods combine information from different modalities. A problem with these methods is that the sensors in the mode (or multiple modalities) used for tracking introduce errors, resulting in a resolution lower than required by the imaging sensor. These testing methods / applications assume these errors are absent, leading to poor performance. As a result of these testing methods, some embodiments of this disclosure are configured to provide tracking corrections with a resolution higher than required by the imaging sensor.

[0031] Furthermore, some important insights gained from the experiments are that both problems—i.e., imaging deformable objects under deformation and reconstructing observed signals with unknown permutations—can be expressed using the same fundamental formulas. This new knowledge was not immediately apparent after extensive experimentation because these are two very different problems with very different applications. The former (imaging deformable objects under deformation) has applications in medical imaging and security screening, while the latter (reconstructing observed signals with unknown permutations) has applications in label-free and partially labeled sampling, as well as simultaneous localization and mapping (SLAM).

[0032] Therefore, such formulas incorporated into some embodiments of this disclosure include:

[0033] (1) Measure the unknown signal x by taking one or more snapshots;

[0034] (2) Linear transformation of the signal measured in any snapshot (F) i );

[0035] (3) Unknown arrangements affecting the signal (P) i );

[0036] (4) Measurement system (A) i This may or may not be a recognition system that directly measures signals; and

[0037] (5) A set of measurements y of an arrangement signal with unknown transformation i .

[0038] Another important insight of this disclosure is that this formula can be further relaxed to allow for a more lenient solution. This allows for the computation of the gradient of the cost function, which enables optimization using gradient-based algorithms. Without relaxation, the cost is discrete and therefore has no gradient. In this case, the optimization is combinatorial, and its computational complexity is prohibitive for any problem of reasonable practical scale.

[0039] Another insight is that this particular relaxation choice allows for the use of efficient methods based on optimal transport, which can provide better solutions and are more likely to converge to a good optimum. The problem is non-convex; therefore, naive relaxations will eventually exhibit too many local minima and cannot provide a good solution. Another important insight is that the permutation matrix P... i Explicit estimation is not required; only an estimation of the signal x is needed. This further provides the opportunity to use optimal transmission methods, which provide a “transmission plan” of implicitly estimated permutations.

[0040] Another important insight is that when the problem is relaxed as described above, it becomes a bilinear problem. Therefore, this problem can be efficiently solved using alternating minimization, where the algorithm estimates the original signal x and the alternating signals x measured by the measurement system in each snapshot. i They alternate between each other.

[0041] According to one embodiment of this disclosure, an imaging system includes: a tracking system for tracking a deformable object within a scene at multiple time steps over a time period to generate an initial estimate of the deformation of the object moving at each time step. A measurement sensor captures measurements of the object deforming in the scene at the multiple time steps over the time period as measurement data by capturing snapshots of the object moving at the multiple time steps over the time period. A processor calculates deformation information of the deformable object based on the measurement data. Each acquired snapshot of the object includes measurements of the object deformed at that time step in the measurement data. For each of the multiple time steps, the processor sequentially calculates the deformation information of the object by calculating a correction to the deformation estimate of the object, such that the correction includes matching the measurement of the corrected deformation of the object for each time step with the measurement in the snapshot of the object acquired for that time step. Wherein, for each time step, a corrected deformation is selected that is superior to other corrected deformations based on the distance between the corrected deformation and the initial estimate of the deformation, to obtain a final estimate of the deformation of the deformable object moving in the scene and a final image of the object moving within the scene.

[0042] According to another embodiment of this disclosure, an image processing method includes: tracking a deformable object moving within a scene at multiple time steps over a time period via a tracking system to generate an initial estimate of the object's deformation for each time step. Measurement data is acquired by continuously capturing snapshots of the deformed object in the scene at the multiple time steps over the time period. Deformation information of the deformable object is calculated by generating a set of measurements of the object having deformed shapes at the multiple time steps from each acquired snapshot of the object, the snapshots including measurements of the object deformed during that time period from the measurement data. For each of the multiple time steps, deformation information of the object is calculated by calculating a correction to the deformation estimate of the object. Calculating the correction includes matching the measurement of the corrected deformation of the object for each time step with the measurements in the snapshots of the object acquired for that time step. Specifically, for each time step, based on the distance between the corrected deformation and the initial estimate of the deformation, a corrected deformation that is superior to other corrected deformations is selected for that time step to obtain a final estimate of the deformation of the deformable object moving in the scene and a final image of the object moving in the scene, and the final estimate and the final image are stored.

[0043] According to another embodiment of this disclosure, a production apparatus includes a tracking system for tracking deformable objects within a scene at multiple time steps over a period of time, to generate an initial estimate of the deformation of the object for each time step. Measurement sensors, including electromagnetic sensors, capture measurements of the object deforming in the scene at the multiple time steps over the period of time as measurement data by capturing snapshots of the object moving at those multiple time steps. A processor calculates deformation information of the deformable object based on the measurement data. Each acquired snapshot of the object includes measurements of the deformed object at that time step, to generate a set of measurements of the object having a deformed shape at the multiple time steps in the measurement data. For each time step in the multiple time steps, the processor sequentially calculates the deformation information of the object by calculating a correction to the deformation estimate of the object. The correction includes matching measurements of the corrected deformation of the object for each time step with measurements in the snapshots of the object acquired for that time step. Specifically, for each time step, based on the distance between the corrected deformation and the initial estimate of the deformation, a corrected deformation that is superior to other corrected deformations is selected for that time step to obtain a final estimate of the deformation of the deformable object moving in the scene and a final image of the object moving in the scene, and the final estimate and the final image are stored.

[0044] A radar system according to another embodiment of this disclosure. The system includes a tracking system for tracking a deformable object moving within a scene at multiple time steps over a time period to generate an initial estimate of the deformation of the moving object at each time step, such that each time step includes a different deformation. A sensor captures measurements of the object deforming in the scene at the multiple time steps over the time period as measurement data by capturing snapshots of the object moving at the multiple time steps. A processor calculates deformation information of the deformable object based on the measurement data. Each acquired snapshot of the object includes measurements of the deformed object at that time step to generate a set of measurements of the object having a deformed shape at the multiple time steps in the measurement data. For each time step in the multiple time steps, the processor sequentially calculates the deformation information of the object by calculating a correction to the deformation estimate of the object. Such correction includes matching the measurement of the corrected deformation of the object for each time step with the measurement in the snapshot of the object acquired for that time step. Specifically, for each time step, based on the distance between the corrected deformation and the initial estimate of the deformation, a corrected deformation that is superior to other corrected deformations is selected for that time step to obtain a final estimate of the deformation of the deformable object moving in the scene and a final image of the object moving within the scene. The output interface outputs the final estimate of the deformation of the deformable object, the final image of the object moving within the scene, or both, to one or more components of the radar system's output interface, to another system, or to a communication network associated with the radar system.

[0045] According to another embodiment of this disclosure, a radar imaging method is used to reconstruct a radar reflectivity image of a scene. A deformable object moving within the scene is tracked at multiple time steps over a time period to generate an initial estimate of the object's deformation for each of the multiple time steps. At least one electromagnetic sensor captures measurements of the object deforming in the scene at the multiple time steps over the time period as measurement data by capturing snapshots of the object moving at the multiple time steps. Each acquired snapshot of the object includes measurements of the deformed object at that time step to generate a set of measurements of the object having a deformed shape at the multiple time steps in the measurement data. The method includes using a processor to calculate deformation information of the object by calculating a correction to the deformation estimate of the object for each of the multiple time steps. Such that the calculation of the correction includes matching the measurement of the corrected deformation of the object for each time step with the measurements in the snapshot of the object acquired for that time step. Specifically, for each time step, based on the distance between the corrected deformation and the initial estimate of the deformation, a corrected deformation that is superior to other corrected deformations is selected for that time step to obtain a final estimate of the deformation of the deformable object moving in the scene and a final image of the object moving within the scene. The final estimate of the deformation of the deformable object or the final radar image of the object is output to one or more components of the radar system, or output to another system associated with the radar system.

[0046] According to another embodiment of this disclosure, a non-transitory computer-readable storage medium embodies a processor-executable program for performing a radar imaging method. The radar imaging method is used to reconstruct a radar reflectivity image of a scene. The scene includes deformable objects within the scene. A tracking system with optical sensors tracks the objects deforming at multiple time steps over a time period to generate an initial estimate of the deformation of the objects for each of the multiple time steps. Measurement data is acquired by continuously capturing snapshots of the objects deforming in the scene at the multiple time steps over the time period, such that different deformations are included at each time step. The method includes calculating deformation information of the deformable objects by generating a set of measurements of objects with deformed shapes at the multiple time steps from each acquired snapshot of the objects, the snapshots including measurements of the deformed objects in the measurement data for that time step. For each of the multiple time steps, deformation information of the objects is calculated by calculating a correction to the deformation estimate of the objects. Calculating the correction includes matching the measurement of the corrected deformation of the objects for each time step with the measurements in the snapshots of the objects acquired for that time step. Specifically, for each time step, based on the distance between the corrected deformation and the initial estimate of the deformation, a corrected deformation that is superior to other corrected deformations is selected for that time step to obtain a final estimate of the deformation of the deformable object moving in the scene and a final image of the object moving within the scene. The final estimate and the final image are then stored. The final estimate of the deformation of the deformable object, the final radar image of the object moving within the scene, or both, are output to one or more components of the radar system, or to a communication network associated with the radar system.

[0047] Embodiments of this disclosure will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, and their purpose is to illustrate the principles of embodiments of this disclosure. Attached Figure Description

[0048] [ Figure 1A ]

[0049] Figure 1A This is a schematic diagram illustrating a radar imaging system for determining a radar reflectivity image of an object moving within a scene, according to one embodiment of the present disclosure.

[0050] [ Figure 1B ]

[0051] Figure 1B This is a flowchart illustrating some method steps for implementing some embodiments of the method according to this disclosure;

[0052] [ Figure 2A]

[0053] Figure 2A This is a schematic diagram illustrating deformation of an imaged object according to some embodiments of the present disclosure;

[0054] [ Figure 2B ]

[0055] Figure 2B This is a schematic diagram illustrating some components that can be used in conjunction with a radar imaging system according to some embodiments of the present disclosure;

[0056] [ Figure 2C ]

[0057] Figure 2C This is a schematic diagram illustrating an MRI machine scanning a person using a radar imaging system according to some embodiments of the present disclosure;

[0058] [ Figure 3 ]

[0059] Figure 3 This is a schematic diagram illustrating a dual-grid representation of an object according to some embodiments of the present disclosure;

[0060] [ Figure 4 ]

[0061] Figure 4 A schematic diagram is shown illustrating the use of a dual-grid representation to capture object motion according to some embodiments of the present disclosure;

[0062] [ Figure 5 ]

[0063] Figure 5 A schematic diagram is shown illustrating the use of a dual-mesh representation to capture the transformations of an object caused by its motion, according to some embodiments of this disclosure;

[0064] [ Figure 6 ]

[0065] Figure 6 A schematic diagram of an electromagnetic sensor (such as a radar) for acquiring radar reflectivity images according to some embodiments of the present disclosure is shown;

[0066] [ Figure 7 ]

[0067] Figure 7 A schematic diagram illustrating the reconstruction of a radar reflectivity image according to some embodiments of the present disclosure is shown;

[0068] [ Figure 8 ]

[0069] Figure 8Examples of motion and deformation of an object in front of optical and radar sensors at each snapshot are shown according to some embodiments of the present disclosure;

[0070] [ Figure 9 ]

[0071] Figure 9 Some embodiments according to this disclosure are shown, using an optical sensor (using Figure 8 A schematic diagram illustrating the tracking process in an embodiment;

[0072] [ Figure 10A ]

[0073] Figure 10A A flowchart of an optimized procedure for recovering deformation and measuring signals, according to some embodiments of the present disclosure, is shown;

[0074] [ Figure 10B ]

[0075] Figure 10B Some embodiments according to this disclosure are shown, implementing Figure 10A The pseudocode for aspects of the flowchart in the document;

[0076] [ Figure 10C ]

[0077] Figure 10C Some embodiments according to this disclosure are shown, implementing Figure 10A Pseudocode for the flowchart aspect;

[0078] [ Figure 10D ]

[0079] Figure 10D Some embodiments according to this disclosure are shown, implementing Figure 10A Pseudocode for the flowchart aspect;

[0080] [ Figure 11A ]

[0081] Figure 11A The following are examples of experimental practices used in some embodiments according to this disclosure;

[0082] [ Figure 11B ]

[0083] Figure 11B The following are examples of experimental practices used in some embodiments according to this disclosure;

[0084] [ Figure 11C ]

[0085] Figure 11C The following are examples of experimental practices used in some embodiments according to this disclosure;

[0086] [ Figure 11D ]

[0087] Figure 11D The following are examples of experimental practices used in some embodiments according to this disclosure;

[0088] [ Figure 11E ]

[0089] Figure 11E The following are examples of experimental practices used in some embodiments according to this disclosure;

[0090] [ Figure 12A ]

[0091] Figure 12A Some embodiments according to this disclosure are shown, for Figure 11A , Figure 11B , Figure 11C , Figure 11D and Figure 11E The experimental examples in the document include performance analysis of some implementation methods under various experimental conditions and comparisons with conventional methods.

[0092] [ Figure 12B ]

[0093] Figure 12B Some embodiments according to this disclosure are shown, for Figure 11A , Figure 11B , Figure 11C , Figure 11D and Figure 11E The experimental examples in the document include performance analysis of some implementation methods under various experimental conditions and comparisons with conventional methods.

[0094] [ Figure 13 ]

[0095] Figure 13 Hardware diagrams of different components of a radar imaging system according to some embodiments of the present disclosure are shown; and

[0096] [ Figure 14 ]

[0097] Figure 14 This is a schematic diagram of a computing device illustrating some techniques that can be used to implement methods and systems according to some embodiments of the present disclosure. Detailed Implementation

[0098] While the accompanying drawings illustrate the embodiments currently disclosed, other embodiments are contemplated, as indicated in the discussion. This disclosure presents illustrative embodiments by way of statement and not limitation. Those skilled in the art can devise many other modifications and embodiments that fall within the scope and spirit of the principles of the embodiments currently disclosed.

[0099] Figure 1A This is a schematic diagram illustrating an imaging system 100A according to one embodiment of the present disclosure, used to determine a reflectance image of an object moving within a scene 105. The imaging system 100A may include: at least one tracking sensor 102 (e.g., an optical or ultrasonic sensor) configured to acquire an optical reflectance image of the scene 105; and at least one measurement sensor 104 configured to acquire measurements of the scene 105. Embodiments of the tracking sensor 102 include one or a combination of an optical camera, a depth camera, an infrared camera, and an ultrasonic sensor. Embodiments of the measurement sensor 104 include one or a combination of millimeter-wave (mmWave) radar, a terahertz (ThZ) imaging sensor, backscattered X-ray, magnetic resonance imaging, and tomographic X-ray sensors.

[0100] Tracking sensor 102 can be configured to track objects in scene 105 at multiple time steps over a period of time, so as to generate the object shape for each of the multiple time steps. In various embodiments, tracking sensor 102 may determine the object shape as an inaccurate deformation 115 of the object's nominal shape, wherein the deformation is inaccurate because it may contain tracking errors or may not exhibit the tracking resolution required for reconstructing the object using the measurement sensor in a measurement sensor mode. For example, the nominal shape of the object may be the object shape arranged in a prototype pose that is generally known in advance. In other embodiments, tracking sensor 102 may determine the object shape in one time step as an inaccurate deformation 115 of the object shape in different time steps, wherein the deformation is inaccurate because it may contain tracking errors or may not exhibit the tracking resolution required for reconstructing the object using the measurement sensor in a measurement sensor mode.

[0101] Still referencing Figure 1A The measurement sensor 104 can be configured to acquire snapshots of scene 105 at multiple time steps within a time period to produce a set of measurements 117 of an object having a deformed shape defined by a corresponding deformation of the object shape determined by the tracking sensor 102. It is noteworthy that, as the object moves within scene 105, at least two different measurement snapshots may include objects with different deformed shapes.

[0102] Imaging system 100A may include at least one processor 107. Processor 107 may be configured to determine, for each snapshot at each of a plurality of time steps, a correction 111 for a deformation 115 determined at the corresponding time step, the correction incorporating measurements of the scene at that time step to produce an accurate deformation using embodiments of the present disclosure. The processor may be further configured to determine an image of an object in a mode of measurement sensors, combining deformation corrections from one or more time steps and measurement snapshots from one or more time steps under a specific deformation.

[0103] Because the object is moving in scene 105, different measurement snapshots are obtained from different transformations for each of the multiple time steps. In some implementations, tracking and measurement snapshots can be synchronized, for example, simultaneously at corresponding time steps and / or at predetermined time migrations, thereby using deformations generated by the corresponding tracking sensors (synchronously or acquired at the same time step) to determine an image of the object.

[0104] In some embodiments, the tracking and measuring sensors may be the same sensor, wherein the processor 107 is further configured to determine inaccurate deformations before calculating corrections. In other embodiments, the tracking sensor may or may not be the same sensor as the measuring sensor, and the processor directly calculates the accurate deformation by combining tracking snapshots from one or more time steps and measurement snapshots from one or more time steps.

[0105] In some implementations, the processor may further incorporate other available information when determining inaccuracies and distortions at each time step. This information may include, but is not limited to:

[0106] i) The location of the sensor system at each snapshot;

[0107] ii) Orientation and field of view of each sensor at each time step;

[0108] iii) Measure the scene in advance, or learn the geometry of the scene from existing resources (such as maps, wireframe representations, and images);

[0109] iv) Dynamic information of object deformation, such as heart rate and beating model, lung breathing rate and deformation model, etc.

[0110] v) If the sensor is mobile, odometry for the platform on which the sensor is mounted, including rate, acceleration, pitch, yaw, and kinematic motion models.

[0111] vi) Predetermined reflectivity patterns or markings, such as QR codes; corner reflectors, or motion capture markers;

[0112] vii) Pre-existing sensor landmarks in the scene and their precise geometry, as well as any other information that may help the processor determine object deformation.

[0113] Still referencing Figure 1A Some implementations can be based on the recognition that imaging systems can jointly use measurements of a scene acquired at multiple time steps. Such measurement systems can be used to improve the resolution of the measured images beyond the resolution limited by the size of the imaging system (referred to in the art as aperture size). As an object moves or deforms at different time steps over time, it can be located in different positions and can have different shapes caused by non-rigid motion. This misalignment and deformation of the object makes the measurement system obscure (i.e., ill-posed) and difficult or impossible to resolve. However, this implementation can construct a larger synthetic aperture by determining and using the transformation between each measurement snapshot, which allows for higher effective resolution, thereby taking advantage of the diversity brought about by motion. This is referred to in the art as inverse synthetic aperture imaging (ISAR). However, methods in this art cannot incorporate deformable objects into the tracking.

[0114] Some embodiments of this disclosure provide ISAR for deformable objects. Therefore, this embodiment can combine measurements of a scene acquired at multiple time steps to generate images of one or more specific poses or deformations of the object. For example, an image of a person can be reproduced as the person walks across the system or in a pose where all parts of the person's body are visible and unobstructed. As another embodiment, an image of a beating heart or lungs can be reproduced at predetermined stages of the beating or breathing pattern.

[0115] Some implementations are based on the recognition that at least one reason for using a single sensor to measure and track a moving object is due to the resolution of the measuring sensor, which is constrained by the physical size of the sensor's antenna. Specifically, for practical reasons, the antenna size of the imaging system may allow only a coarse image of the object to be estimated at each time step. This coarse image may be suitable for tracking objects undergoing rigid and finite transformations; however, this type of component configuration may fail to recognize the typical arbitrary non-rigid transformations in human motion.

[0116] Still referencing Figure 1ASome implementations are based on the recognition that other sensors (such as optical monochrome or color or infrared cameras, or depth cameras, or ultrasonic sensors, or combinations thereof) are cheaper than measurement sensors, have comparable resolution, and are better suited for tracking. Therefore, tracking sensors can be used to track the movement of a target, even if the target is deformable and the movement is not rigid. On the other hand, tracking sensors using a different modality than measurement sensors may not provide the information or resolution required for the sensing system to function. For example, optical sensors cannot see covered objects, so while they can be used in security applications to track people moving through the system, they cannot detect dangerous weapons or contraband. Similarly, ultrasonic sensors are also very inexpensive and capable of detecting and tracking the breathing patterns of a beating heart or lungs. However, they cannot image a beating heart or lungs with sufficient accuracy and resolution as MRI or CAT systems.

[0117] Some implementations are based on the recognition that, for some applications, it is sufficient to determine a radar reflectivity image of an object in a certain prototype pose, rather than the object's current pose in the current time instance. For example, in some security applications, a person's prototype pose is standing with their arms outstretched upwards or to the sides. Objects arranged in a prototype pose have a nominal shape, which can change, i.e., deform, as the object moves.

[0118] Still referencing Figure 1A Some implementations are based on another understanding: optical sensors can track objects in a scene using relative deformations of previous poses, rather than deformations from a common prototypical pose. For example, instead of determining the pose and / or absolute current shape of an object in the current snapshot of the scene, the optical sensor can determine the relative shape of the object as a deformation of the object's shape in another snapshot. This implementation is based on the understanding that deformations of the nominal shape of an object determined by a tracking sensor can be used to reconstruct an image of the object in a certain pose.

[0119] Figure 1B This is a flowchart illustrating some method steps for implementing a method according to some embodiments of the present disclosure. The method begins at 118 and obtains a plurality of snapshots 120 of a signal of interest, which include a subset or combination of tracking data and measurements of the signal of interest, which can be used to reconstruct an image of the signal of interest. The signal of interest may include a specific object, scene, or combination thereof in the entire or part of the sensor's field of view, and the signal of interest undergoes deformation in each snapshot.

[0120] Still referencing Figure 1BIn some implementations, if possible, using available tracking data and measurements, and employing methods known in the art, an estimate of the approximate deformation of the signal of interest in each snapshot is calculated 122. Among other possible available information, a cost function 124 relating to the actual deformation of the signal of interest, the approximate estimate of the deformation, the signal of interest, the measurement of the signal of interest, and the tracking data is iteratively reduced 127 until convergence 126, as described below.

[0121] Still referencing Figure 1B If necessary, in some implementations, the calculated deformation is used to reconstruct the signal of interest 128. Depending on the application and the requirements of further processing steps, the signal of interest, the calculated deformation, or both are output by the method 132.

[0122] It is anticipated that some steps of a radar method for estimating the deformation of a deformable object moving in a scene may include the following steps: for example, tracking a deformable object in the scene via a tracking system with tracking sensors at multiple time steps over a period of time, to estimate the deformation of the object for each time step. One step uses an electromagnetic sensor to capture measurements of the deformable object in the scene as measurement data at the multiple time steps by capturing snapshots of the moving object at the multiple time steps over the period of time. Another step uses a processor to calculate deformation information of the deformable object based on the measurement data. This may include an electromagnetic sensor capturing snapshots of the deformable object at the multiple time steps. Each acquired snapshot of the object in the measurement data includes a measurement of the deformation of the object at that time step, to produce a set of measurements of the object with a deformable shape at the multiple time steps. The processor sequentially calculates the deformation information of the object by calculating a correction to the deformation estimate of the object for each of the multiple time steps.

[0123] In one step, correction may include matching measurements of the corrected deformation of the object for each time step with measurements in a snapshot of the object acquired for that time step. Specifically, for each time step, based on the distance between the corrected deformation and the initial estimate of the deformation, a corrected deformation that is superior to other corrected deformations for that time step is selected to obtain a final estimate of the deformation of the deformable object moving in the scene and a final image of the object moving within the scene.

[0124] Depending on the specific objectives of the user or operator, a step may include one or more components that output a final estimate of the deformation of the deformable object to at least one output of a radar system or another system associated with the radar system.

[0125] Figure 2AThis is a schematic diagram illustrating relative tracking using at least one measuring device (e.g., an optical sensor) according to some embodiments of the present disclosure. For example, some embodiments can track a moving object (e.g., a person) by determining the deformation 220 of the nominal shape 210 of the object at each instant or time step (this deformation produces a deformed shape 230 of the object).

[0126] Some implementations are based on the understanding that the deformation 220 indicating the change of the object in the tracking mode also indicates the change of the object in the measurement mode, even if the two modes are different. Therefore, an approximate deformation can be calculated from the output of the tracking sensor.

[0127] Figure 2B This is a schematic diagram illustrating some components of a radar imaging system that can be utilized according to some embodiments of the present disclosure. For example, object 240 may be a person walking or moving relative to a sensor configuration, wherein person 240 can be imaged by a measuring sensor. Some types of measuring sensors may include radar, millimeter wave, backscattered X-ray, or terahertz sensors 260A to 260C, 261A to 261C, 262A to 262C, which may take snapshots for each time step to generate an image of the person in a certain posture, which may be a three-dimensional (3D) image, wherein the posture may be the posture of the person at a certain time step or a posture that the person has never used, such as a prototype posture with arms outstretched and extended, so that all sides of the body are visible.

[0128] Some types of tracking sensors may include optical and depth sensors 265A to 265C, which can additionally detect a three-dimensional 3D model of the object 240 (i.e., a person) to track deformations of the person as they move past the imaging system. For example, tracking the deformation of the object may include determining the position and orientation of each part of the person's body (such as arms and legs) relative to the imaging system. It may also include using a wireframe model of the body and, for each acquired snapshot, determining the position of each point of the wireframe model within the sensing system and / or determining whether that point is occluded to the camera at the time step of the snapshot. Tracking the deformation of the body may also include mapping pixels or voxels from one snapshot to another, such that pixels mapped from one snapshot to another correspond to the same part of the body as it moves between the two snapshots.

[0129] Still referencing Figure 2B In operation, the sensor configuration's measurement sensor generates data, which is then sent to a hardware processor (such as...). Figure 1AThe signal is processed by a hardware processor 107. The person 240 moves relative to the sensor configuration, and the transmitters of the measuring sensors 260A to 260C, 261A to 261C, 262A to 262C can transmit signals continuously or in pulses. The wave of the transmitted signal is reflected onto the person or object 240 and detected by the receiving sensor. Some measuring sensors may be passive, i.e., only having a receiving sensor. The signal is digitized and routed to the hardware processor, i.e., for example... Figure 1A The processor 107. Regarding sensor configuration, other implementations can be conceived, including more sensors, different combinations of various types of sensors, and even combinations of various types of radar systems. Depending on the user's specific objectives and sensor configuration requirements, other components may include... Figure 13 and Figure 14 Higher-performance components are provided. Different types of sensors in the sensor configuration can be synchronized with sufficiently high accuracy so that they can work in a coordinated manner. Each receiving unit can be equipped with a receiving channel and an analog-to-digital conversion system.

[0130] Rotating or other moving structures 256, 258 can be configured with different types of sensors to address specific user objectives and application requirements. As one embodiment, these rotating structures can rotate clockwise (D1, D2, D3, D4) or counterclockwise (not shown) directions, depending on specific user requirements. Additionally, rotating structures 256, 258 can be positioned on tracks to increase or decrease the height of the rotating structure (not shown), and can even travel on tracks (not shown) along horizontal axes H and / or Y. The reason why sensor configurations can include several aspects of moving features can be related to specific user application requirements. For example, users can configure sensors for security-related applications, including airports, buildings, etc., to identify potential weapons, etc. In this case, using measuring sensors positioned on rotating structures 256, 258 to image an object 360° reduces cost due to the need for fewer sensors. It is anticipated that other types of sensors (i.e., audio, temperature, humidity, etc., and lighting) can be mounted on rotating structures 256, 258 and other structures A, B, C. Some advantages of using rotating structures 256 and 258 can include: the measurement sensor can cover a larger target area; and a larger effective aperture, which provides higher resolution images.

[0131] Figure 2CThis is a schematic diagram illustrating a patient / person 272 on a scanning stage 274 using an MRI or CAT machine 270 as an imaging system, according to some embodiments of the present disclosure. The MRI or CAT machine is used to image the internal structures of the body to detect medical conditions and assist physicians in diagnosis. The MRI or CAT machine takes multiple snapshots of the imaged body at different angles, thereby creating a synthetic aperture for imaging areas of interest within the body.

[0132] To ensure that synthetic aperture imaging reconstructs accurate images without blurring or motion artifacts, the patient should remain as still as possible during the imaging process. This is problematic, especially when imaging moving and deformable organs such as the heart or lungs. In such applications, embodiments of this disclosure may use one or more tracking sensors, including but not limited to ultrasound sensors, heart rate monitors, or respiratory rate sensors.

[0133] Still referencing Figure 2C Because the imaged organ moves and deforms during the imaging process, the tracking sensor estimates the organ's deformation inaccurately for each snapshot taken by the measurement system. Therefore, using some embodiments of this disclosure, it is possible to determine an image of the organ under any deformation, even if each snapshot is taken under different deformations. Using some embodiments of this disclosure, Figure 1A The processor 107 in the process considers the measurements of each snapshot, determines the correction for inaccurate deformation, and images of organs in the imaging modality, as described in this disclosure.

[0134] Figure 3 This is a schematic diagram illustrating a dual-mesh representation of an object according to some embodiments of the present disclosure. For example, a deformable object 310 (a person in this embodiment) is in a prototype pose. To construct a radar reflectivity image in prototype form, a mesh 320 can be defined on the prototype pose of the object. In other words, the first mesh 320 of the dual-mesh representation is a prototype mesh that discretizes the object itself. For example, the mesh in the image has mesh positions 330, indexed as 1, 2, ..., N. The radar reflectivity of each point in mesh 340 is represented as x1, x2, ..., x... N Several methods exist for indexing prototype meshes, but generally, they can always be mapped to continuous meshes, as shown in the figure. Any mesh indexing method can be used in the embodiments of this disclosure.

[0135] The second grid in the dual-grid representation is a radar grid that discretizes the scene itself. For example, in one implementation, the second grid is a rectangular (Cartesian) grid of 350. However, other grids (such as radial grids) can also be used in different implementations. Similar to the prototype grid, the radar grids used in different implementations have several indexing methods. For example, in... Figure 3 In the embodiment shown, the radar grid is indexed using Cartesian coordinates 360. Measurement sensors (e.g., radar 370) and / or individual radar transceivers 380 are located at known locations within the radar grid.

[0136] Still referencing Figure 3 In some implementations, the two grids in the bi-mesh representation are identical, for example, both are Cartesian coordinate grids. This representation is particularly useful when deformation is determined as a deformation of the object's pose between two snapshots, because the representation of the first grid corresponds to the grid of the first snapshot, and the representation of the second grid corresponds to the grid of the second snapshot.

[0137] Figure 4 A schematic diagram is shown illustrating the use of a dual-grid to characterize the deformation of a captured object according to some embodiments of this disclosure. Figure 4 The pose of an object at a first grid 400 and the pose of the object in a second grid in front of a measurement sensor in a single snapshot 450 are shown. The pose of the object in front of the measurement sensor can be described by the deformation 440 of the object in the first grid to represent the deformation of the object in the second grid. The object in the second grid is observed by a measurement sensor (e.g., radar 470) and its respective transceivers 480 according to measurement operators associated with the measurement sensor hardware. As mentioned above, the deformation of the radar grid can be inferred from a tracking sensor, which may be the same as the measurement sensor, or it may be a tracking model or other information.

[0138] Figure 5 A schematic diagram is shown illustrating a method of representing the transformations of a captured object caused by its motion using a dual-grid representation according to some embodiments of the present disclosure. This embodiment defines each transformation as a subsampled arrangement that arranges the positions of certain points in the object image within a first grid, and removes distinct points in the object image within the first grid based on the deformation of the object's nominal shape in the first grid.

[0139] refer to Figure 4 and Figure 5 Specifically, the deformation is Figure 5 The sub-sampling coordinate arrangement is 545, which is a transformation that maps the exponent in the coordinate system of the first grid to the exponent in the coordinate system of the second grid. Therefore, the image of the object measured by the measurement sensor is a simple arrangement with erasure, which will... Figure 5 The Figure 4 The image in the first grid 410 is mapped 560 to the image of the object in the second grid, consistent with the pose of the object.

[0140] Still referencing Figure 5More generally, in some implementations, the image of the deformable object in the second mesh is a linear transformation of the object image in the prototype pose, which can be described as...

[0141]

[0142] Where x is the image of the object's pose in the first grid, and z is the image of the deformed object in the radar grid.

[0143] Figure 6 A schematic diagram of a measurement sensor (using radar as an example) for acquiring radar reflectivity images according to some embodiments of this disclosure is shown. Sensor 670 includes transceiver 680 (or separate transmitter and receiver) for sensing a scene. Specifically, one or more transceivers transmit pulses 620 or otherwise excite the scene. Depending on the reflectivity of the object, these pulses are absorbed or reflected back by the object 650. The reflected pulses 630 are acquired by one or more transceivers 680. Pulse transmission and acquisition are controlled by radar control, pulse and acquisition system 690, which can control the pulse shape and timing, as well as which transceivers transmit and which transceivers receive. In some modalities (such as MRI), the excited object may generate its own signal instead of reflecting, such as resonating according to the object's response characteristics. Generally, these response characteristics (which may include reflectivity) comprise an image of the object.

[0144] This system is configured to acquire signals received by a receiver in response to received pulses from a scene, for example, using a data acquisition system. The data acquisition system may, in particular, include one or more amplifiers, one or more modulators, and one or more analog-to-digital converters. The system outputs data y 695, which represents a record of pulse reflections. These records are samples of the reflections or their functions, such as demodulation, filtering, de-modulation, or other preprocessing functions known in the art. This data includes measurements of the scene in each snapshot.

[0145] Still referencing Figure 6 The acquired data y is a linear measurement of z, i.e., the radar reflectivity image of the deformable object 650 in the radar scene, obtained through a radar acquisition function, also known in the art as the forward operator (denoted as A here). Therefore, the data acquired for a single snapshot is equal to

[0146]

[0147] If a radar system has a sufficient number of sensors and a large aperture, then data y might be sufficient to reconstruct z, i.e., a radar reflectivity image of the object in its deformed posture. However, reconstructing a high-resolution image would require a large and expensive radar array. Furthermore, in specific deformities, parts of the object may not be visible to the array, regardless of its size, making their radar reflectivity impossible to reconstruct.

[0148] Still referencing Figure 6 Therefore, some implementations of imaging systems acquire measurements of several snapshots of an image under different deformations.

[0149]

[0150] Where i = 1, ..., T are the indices of the snapshots, and T is the total number of snapshots. In each implementation, the only change between snapshots is the deformation of the object, and therefore the deformation of the radar reflectivity image. In some implementations, the measurement operator may differ, for example, when different transceivers are used to acquire each snapshot, or when the sensor moves or rotates on a mobile platform. In other implementations, the forward operator is always the same in each snapshot, in which case A is equal to the forward operator for all i. i =A.

[0151] If all deformations are fully known, then the image of the object can be reconstructed by combining measurements of the object's image with the deformed shape transformed by the corresponding transformation. For example, using multiple snapshots, the reconstruction problem becomes one of the problems of restoring x according to equation (4).

[0152]

[0153] Assumption It is known that this can be done using, for example, least squares inversion. The solution in this art (4) can impose additional regularization constraints, such as sparsity or smoothness, on the reconstructed image x by expressing x in some low-dimensional basis or in a large dictionary (i.e., x = Bh, where B is a low-dimensional basis or dictionary, and h is a set of coefficients lower in dimension than x or sparse coefficients). Alternatively, or additionally, other solutions in this art can impose a low total variation structure on x, i.e., sparsity of its gradients. All these regularization constraints can be imposed by using different techniques for regularization.

[0154] Still referencing Figure 6 Some implementations determine image deformation based on the deformation of images acquired in different modalities (e.g., optical images). In other words, by using different modalities, it is possible to infer the physical deformation of an object. Since the reflectivity of each point in the image of an object in the measurement modality does not change with the position of that point (i.e., not with the deformation of the object), the deformation inferred by sensors in different modalities can be used to infer the deformation of the object image in the measurement modality.

[0155] Optical sensors (such as monochrome, color, or infrared cameras) record snapshots of an object's reflectivity as it moves through a scene. Using two or more such cameras (placed at a distance from each other), it is possible to determine the distance of each point of the object to each camera, referred to in the art as the depth of that point. Similarly, depth cameras use the time-of-flight of optical pulses or structured light patterns to determine depth. By acquiring the optical reflectivity and / or depth of an object as it moves, methods exist in the art for tracking points of an object, i.e., determining the deformation of the object based on the deformation of the optical or depth image in each snapshot. Even if the optical reflectivity of the object changes with deformation due to illumination, occlusion, shadows, and other effects, determining this deformation is possible in the art.

[0156] Still referencing Figure 6 Therefore, optical sensors (such as cameras or depth sensors) can be used to infer deformation in each snapshot. As described in (3), the optical sensor acquires a snapshot of the object simultaneously with the radar sensor acquiring a snapshot of the radar reflectivity image. This radar reflectivity image can then be used to track the deformation of the object in order to reconstruct its radar reflectivity image. In some embodiments, the optical sensor may acquire snapshots at different time instances than the radar sensor. The deformation of the object at the time instance in which the radar snapshot was acquired can then be inferred using techniques known in the art (such as interpolation or motion modeling).

[0157] Similarly, in other embodiments, other tracking sensors can be used to infer deformation. In some embodiments, for example, it is known in the art how to use, for example, ultrasound sensors to infer deformation of internal organs (e.g., a beating heart or breathing lungs). In other embodiments, methods commonly referred to in the art as Simultaneous Localization and Mapping (SLAM) can be used to infer deformation caused by the motion of the sensor platform.

[0158] Still referencing Figure 6 Most of the time, the deformation estimated from the tracking sensor is inaccurate, i.e., it includes tracking errors and the resolution is lower than that required for accurate image reconstruction using the measurement sensor (4). Some embodiments of this disclosure correct for tracking errors to produce accurate deformation and, where necessary, an accurate reconstructed image of the object.

[0159] Figure 7A schematic diagram illustrating the reconstruction of a radar reflectivity image according to some embodiments of the present disclosure is shown. In this embodiment, the radar imaging system includes one or more electromagnetic sensors (such as radar array 710) and one or more optical sensors 720. An object 730 (e.g., a person) moves and deforms across the radar and optical sensor surfaces while the sensors acquire snapshots. The data acquired by the optical sensors is processed by an optical tracking system 740, which generates tracking of the object and inaccurate deformation 750 from snapshot to snapshot. The optical tracking system 740 can also map the optical deformation to the object's original pose, i.e., determine the mapping for each snapshot. This mapping is used in conjunction with data acquired from each radar snapshot to correct for inaccurate estimates of deformation and to reconstruct a radar reflectivity image of object 770, 780. The reconstructed radar reflectivity image can be represented by the system in its prototype pose, or it can be transformed and represented in any pose, with any modifications made to suit the system or its users, such as highlighting a portion of the image for further examination of objects 790, 792.

[0160] In this way, the radar imaging system includes an optical tracking system comprising optical sensors to generate an optical transformation for each deformation between points in the optical reflectivity image (including the deformed shape of the object) and points in the original optical reflectivity image (including the nominal shape of the object). The processor of the radar imaging system determines this transformation as an optical transformation function.

[0161] Figure 8 Embodiments of the motion and deformation of an object in front of optical and radar sensors at each snapshot, according to some implementations of this disclosure, are shown. In this embodiment, a person 890 walks in front of the sensors. The sensors take snapshots at different time instances, where the object is in different poses. For example, in Figure 8 In each snapshot, the person is in a different position in front of the sensor, walking from left to right and in a different posture, depending on the timing of each snapshot relative to the person's stride.

[0162] Figure 9 Some embodiments of the present disclosure are shown, using optical sensors. Figure 8 The illustration shows an example of tracking. It is noteworthy that there is a snapshot-based one-to-one correspondence between the deformation of the object shape in the optical reflectivity image and the corresponding transformation in the radar reflectivity image.

[0163] Every point on the person 900 is tracked by the camera at every instant and then mapped to a corresponding point in the prototype pose 990. Each point may be visible or invisible in some snapshots. For example, points on the right shoulder 910, right knee 920, or right ankle 930 may always be visible, while a point on the left hand 950 may be occluded when the hand is behind the body and is invisible to the sensor 960. A correspondence 980 is established between the points tracked in different snapshots and their corresponding points in the prototype image. These correspondences are used to generate... If a point is not visible to the sensor in a specific snapshot (e.g., 960), then This point is not mapped to the radar grid; that is, the corresponding column of the operator contains all zeros. In this sense, Subsampling of the prototype radar image involves mapping only the points visible in the specific snapshot, as determined by the optical sensor.

[0164] Still referencing Figure 9 Some implementation methods are based on the understanding that: The estimated F i It is imprecise and contains errors. These errors can be modeled as Where P i This is a correction to the inaccurate estimate of deformation. Correction P i With F i Similar structure, but less likely to deviate from consistency. In other words, P i It is also an arrangement of subsamples, or more generally, an operator that allows for, for example, fuzziness. However, due to P i Errors in motion tracking were modeled, and the motion tracking was approximately correct, P i The mapping performed only allows points to be mapped from their estimated F. i Move the position slightly. In summary, F is calculated from the system's motion tracking section. i Place the target mesh in approximately the correct position, and P i Make a small correction at this position. To estimate the precise correction P... i Some embodiments of this disclosure employ the following characteristic: P i The greater the deviation from consistency, the less likely it is to be accurate.

[0165] To this end, in some implementations, the processor adjusts each transformation using local error correction and simultaneously determines the radar image of the object in the prototype pose and each local error correction. For example, the processor uses one or a combination of alternating minimization, projection, and constraint regularization to simultaneously determine the radar image of the object in the prototype pose and each local error correction.

[0166] Still referencing Figure 9 Those implementation methods are based on the understanding that: using P i The correction deformation error is generally unknown. Otherwise, if the error is known, then the correction deformation error is negligible. Therefore, the measurement system also estimates P based on the snapshot. i This is to correct for errors. In other words, in some implementations, the processor of the imaging system is configured to solve...

[0167]

[0168] Except for x, all P i All of these are unknown.

[0169] At least one key understanding in this disclosure is that each unknown error correction P i F i The elements of x (i.e., those formed by the inaccurate deformation F) i The deformed element (x) is moved to a different position in the second mesh. Since the inaccurate deformation has already moved the element of x to an approximately correct position, the deformation correction P... i They should not be removed from the already F i The location has been moved too far. Therefore, in estimating P... i At that time, it should not be preferred to cause F i The solution for a large shift in element x.

[0170] Still referencing Figure 9 On the other hand, deformation correction P i F should be moved i Elements of x that enable it to interpret (i.e., match) measurement data y. i To interpret the measurement data, the corrected deformation signal P i F i x, when x is generated by the forward operator A i During measurement, the measurement data (y) should be as close as possible to the actual measurement data. i Therefore, in estimating P i At that time, the corrected deformation signal A should not be preferred. i F i x generates measurements and measurement data y i A mismatched solution.

[0171] The aforementioned preferences represent different objectives that the desired solution should satisfy. Since these objectives are often competing with each other, some embodiments of this disclosure balance these objectives by determining a solution that incorporates them into a single cost function. To do this, some embodiments of this disclosure determine a penalty or cost function that increases the further the solution deviates from the objective.

[0172] Still referencing Figure 9For example, to determine how well the solution interprets the measurement, some implementations can use a norm or distance function to calculate the corrected deformation signal A. i P i F i Measurement of x and measurement data y i The distance. In some implementations, this norm may be the l2 norm, typically denoted as ||y||. i -A i P i F i x‖2, but other norms (such as l1 or l) can be used. ∞ Norm) or distance or divergence function, such as Kulbak-Leibler divergence. If for some candidate solution, the corrected deformed signal A i P i F i The measurement of x does not match the measurement data of y. i If the norm, distance, or divergence is large, then the candidate solution will be penalized more than other solutions. Conversely, if for a candidate solution, the corrected deformed signal A... i P i F i Measurement of x and measurement data y i If the solution matches, then the norm, distance, or divergence will be small and will not penalize the solution.

[0173] Similarly, in order to determine whether a solution will lead to signal F i The correction of elements of x results in large distortion; some implementations use a regularization function R(P). i The regularization function penalizes such solutions. A regularization function is a term in this art describing a function that depends only on the solution and not on the measurement data, and has a larger value for undesirable solutions and a smaller value for desired solutions, similar to how a distance or divergence function takes a larger or smaller value depending on how well the solution matches the data (as described above).

[0174] Still referencing Figure 9 Some embodiments of this disclosure use regularization functions if the variant P i The deformation signal F i The regularization function takes a large value if the elements of x have moved very far from their positions in the imaging domain, where the imaging domain can have one, two, or more dimensions. For example, the regularization function might include the sum of the distances each element has moved within the image, where the distance could be Euclidean (l2) distance, Manhattan (l1) distance, or squared Euclidean distance. or maximum deviation (l) ∞ Distance or other distances suitable for the application.

[0175] To balance the competing objectives of matching measurements and determining deformations that prevent elements from straying too far from their positions, embodiments of this disclosure attempt to minimize a cost function that is a weighted sum of the two objectives.

[0176]

[0177] Here, the cost is added over all deformations across all snapshots, indexed by i. The weight β determines the balance between matching data and regularization, and minimizes the restoration deformation correction P. i Both and the imaged signal x.

[0178] Still referencing Figure 9 This minimization is non-convex, making it difficult to solve. Furthermore, a correct solution should further constrain P... i It is a permutation, or a subsampled permutation, so that the solution mathematically describes the permutation. However, determining the permutation is a problem with combinatorial complexity, which is difficult and very expensive to solve.

[0179] To address this problem, various embodiments of this disclosure utilize the understanding that as deformation corrections are estimated, each deformation correction may generate a deformation signal x. i The intermediate estimate helps interpret the measurement data, but does not perfectly match the corrected deformation signal P. i F i x. Therefore, a separate cost component can be included in minimizing (6) to balance the degree of matching between the intermediate signal and the corrected deformed arrangement.

[0180]

[0181] Among them, the last item Determine the intermediate signal x i With the corrected deformed arrangement P i F i The degree of matching of x. It should be noted that although (7) uses the square of the l2 norm (i.e., This can be used to quantify both the degree of matching between the intermediate signal and the corrected deformed arrangement, and the degree of matching between the measurement of the intermediate signal and the measurement data. However, other norms or distances can be used, such as those listed above.

[0182] Still referencing Figure 9 This implementation is not obvious because it relaxes the problem and introduces more unknown variables (intermediate signal x). i This makes the problem appear more difficult to solve. However, the benefit of this relaxation is that it allows for the use of optimal transport (OT) theory and algorithms to solve parts of the problem.

[0183] In particular, due to P i It is a permutation, and minimizing the last term in (7) can be expressed as ∑ n,n′ (x i [n]-(F i x)[n′]) 2 P i [n,n′], where the symbol u[n] selects the nth element of vector u, and the symbol A[n,n′] selects P. i The nth row and n′th column. In this expression, n and n′ are the exponents on the first and second grids, respectively; that is, n′ represents the position where the nth element of the first grid will move to on the second grid. Furthermore, regularization R(P) i ) can be expressed as Where l[n] and l′[n′] are the coordinates of points n and n′ in the first and second grids, respectively.

[0184] Still referencing Figure 9 A further understanding is that P can be decomposed from these two expressions. i [n,n′], which are combined into a single cost metric.

[0185]

[0186] Its relationship with P i The product of [n, n′] can be expressed using the OT algorithm known in the art with respect to P as a permutation. i Optimize [n,n′]. Using this factorization, the overall minimization (7) can be expressed as

[0187]

[0188] Wherein, as is well known in the art, the symbols <·,·> denote the standard inner product, that is, the sum of the element-wise product of each component of the first independent variable and the corresponding component of the second independent variable, i.e.,<A,B> =∑ n,n′ A[n,n′]B[n,n′].

[0189] Still referencing Figure 9 The OT literature provides algorithms and methods to determine the permutation P that minimizes the inner product. i This is called the transmission plan. The internal minimization calculation in (9) is known in the art as the balanced OT problem.

[0190]

[0191] Wherein, P of the minimization OT problem iThis is an OT (Operational Technology) plan. Solving the OT problem requires computing the optimal plan. The optimal plan provides a variation where all elements in one snapshot are mapped to elements in other snapshots. Therefore, the OT problem does not allow occlusion or other forms of element loss, although this is frequently encountered in applications.

[0192] Other embodiments of this disclosure may use the unbalanced OT or partial OT problem in (9) to replace the balanced OT in (10), more generally.

[0193]

[0194] Where OT(x,x) i The term ) represents an OT problem, which can include balanced, imbalanced, partial, or other OT problems known in the art. Partial or imbalanced OT literature provides a determination of subsampling P. i The algorithm and method are such that a portion of one signal is occluded (i.e., not part of another signal), and vice versa.

[0195] Still referencing Figure 9 The OT problem is also known in the art as the 2-D assignment problem because it only calculates the transmission plan between a pair of signals and can be solved efficiently using linear programming. Solving (11) provides a method for solving the known ND assignment problem in the art, which simultaneously calculates all direct assignments between two or more signals and is generally known to be very difficult. At least one key understanding provided by some embodiments of this disclosure for solving the ND assignment problem is that rather than letting all signals x... i Deformation to match all other signals is less effective than using partially known deformations F. i x, only allow each signal x i Transformation to match a common signal x is more effective. This common signal serves as a template in a sense, from which all other signals are transformed.

[0196] By deforming each signal so that it matches only the common signal, the solution now only requires computing the deformation between a pair of signals (the common signal and each signal in the snapshot). Therefore, since only two signals are involved, the problem is reduced to computing multiple pairwise assignments (i.e., 2-D assignments) instead of a single multi-signal assignment (i.e., ND assignment). This is advantageous because the 2-D assignment problem is well-studied in the field and is easier to solve. A further insight is that even if the deformation is completely unknown, and F... i Consistency (i.e., achieving non-deformation) also plays a role in this reduction.

[0197] Still referencing Figure 9A drawback of computing multiple 2-D assignments may be that it increases the number of unknowns in the solution. It is now important to compute the common signal x in this process, making this reduction from ND assignments to 2-D assignments crucial. Some embodiments of this disclosure rely on the fact that the gradient of the 2-D assignment problem can be computed so that gradient descent methods can be used to compute x, thus making the reduction from ND assignments to 2-D assignments feasible.

[0198] Problem (11) involves several variables x, x' ... i P i Minimizing these variables, which are multiplicatively coupled. Although in this field, for P... i The internal minimization of is understood as an OT problem, but for x, x i The external minimization of x is a non-convex problem, which is difficult to solve. To address this problem, some embodiments of this disclosure consider x as a fixed value. i Minimize and consider x i The process alternates between minimizing x when x is fixed. Other implementations reduce x as x is fixed. i The cost of the function and considering x i The process alternates between reducing the cost as a function of x when the value is fixed.

[0199] Figure 10A A schematic diagram of a method for minimizing the cost function in (11) according to some embodiments of this disclosure is provided. x and x i The initial estimate is used as the starting point 1010. This estimate can be calculated from the measurement using methods well-known in the art (including, but not limited to, least squares inversion, matched filtering, back projection, and sparse inversion). In some implementations, the initial estimate can be set to 0 or a randomly generated signal.

[0200] Figure 10B According to some embodiments of this disclosure, it can be used with respect to x i An algorithm to reduce the cost function 1025.

[0201] Figure 10C According to some embodiments of this disclosure, it can be used in algorithms for reducing the cost function 1020 with respect to x.

[0202] Referring to Figures 10A, 10B, and 10C, by reducing the cost function 1020 with respect to x and with respect to x i The initial estimates are updated by alternating between reducing the cost function l025 until it converges to l070. Figure 10B The text shows information about x. i An exemplary implementation of the cost reduction function 1025, Figure 10C An exemplary implementation of the cost reduction function 1020 for x is shown in the figure. Figure 10D The document outlines one implementation of the alternating update process. In these embodiments, the algorithm performs a fixed number of iterations tMax. However, in other implementations, a convergence criterion can be used instead, as described below.

[0203] Referring again to Figure 10A, some embodiments of this disclosure consider a system that converges after a fixed number of iterations. Other embodiments consider the change in the cost function after each iteration, and consider the system to converge if the change is below a certain threshold for a fixed number of iterations. Other embodiments consider the gradient of the cost function after each iteration, and consider the system to converge if the magnitude of the gradient is below a certain threshold for a fixed number of iterations. Other embodiments consider combinations of the above conditions or other conditions (which may include total processing time, the magnitude of changes in the estimated signal, and whether the calculated transmission plan changes from one iteration to the next).

[0204] Still referencing Figure 10A , Figure 10B and Figure 10C In order to reduce or minimize the cost 1020 as a function of x or as x i The cost of the function is 1025, and some embodiments of this disclosure relate to x or x respectively. i Calculate the gradients 1030 and 1035 of (11), and use these gradients to modify x or x respectively. i (1050 and 1055), to reduce or minimize the cost in (11). In some implementations, the gradient is computed by evaluating the derivative expression that has been analytically derived. Other implementations may use automatic differentiation methods now widely used in the art, which are capable of automatically computed derivatives of functions even when explicit analytical expressions are unavailable. In some implementations, the computation of the gradient requires computation of an internal minimization, i.e., the OT problem and the OT plan used in computed derivatives 1040 and 1045. In an exemplary implementation, the plan is in Figure 10B The calculation is performed in step 5 of Figure 10C.

[0205] To calculate the OT plan, some implementation methods require calculating the original and target mass distributions of the problem, such as... Figure 10B Steps 2 and 4 in the middle and Figure 10C Steps 1 and 4 are shown in the diagram. Some implementations may, for example, use signal estimation as a quality distribution, or signal points with values ​​above a certain threshold, or a uniform distribution at locations where signal values ​​are above a certain threshold, or a uniform distribution at all possible signal locations, or a normalized distribution, or some other positive function of the signal value at each location, or a combination thereof.

[0206] Still referencing Figure 10A , Figure 10B and Figure 10C In some implementations, reducing or minimizing with respect to x or x i The general process of cost calculation requires the use of x and x. i The current estimate, to calculate with respect to x or x i The derivative of the current estimate is obtained, and then gradient steps 1050 and 1055 are used respectively to update x or x based on the derivative. i The estimate. The update step is as follows:

[0207]

[0208]

[0209] in It is the cost function in (11). and They represent x and x respectively. i gradient, γ t x is the gradient step size of step t, which can be the same or different for (12) and (13). t and The variables that are updated in step t are x and x. i x is a variable that is considered fixed in the corresponding step. t+1 and It is an updated variable.

[0210] Still referring to Figure 10A, Figure 10B and Figure 10C Other implementations use different methods to minimize (11), for example by estimating x and x i The outer product is then applied to the resulting object, and a low-rank structure is applied to elevate it to a higher-rank space. However, these methods significantly increase the dimensionality of the problem and the resulting computational complexity, making them impractical in many applications.

[0211] After convergence, the implementation can generate the calculated optimal transmission plan and x or x in the output. i The final estimate is a combination of 1080.

[0212] Figure 11A , Figure 11B , Figure 11C , Figure 11D and Figure 11E Experiments performed on exemplary embodiments of this disclosure are shown. In particular, Figure 11A The signal at the prototype location is shown. The signal is deformed in two different snapshots (e.g., ...). Figure 11D and Figure 11E As shown in the figure, this is observed by the measurement acquisition system. Figure 11B and Figure 11C The approximate deformations for each snapshot estimate are shown below. The purpose of this experiment is to correct... Figure 11B and Figure 11C The approximate deformation estimated in order to... Figure 11A Signal restoration.

[0213] Figure 12A Aspects according to this disclosure are shown. Figure 11A , Figure 11B , Figure 11C , Figure 11D and Figure 11E The experimental results are shown in the figure. This figure depicts the reconstruction accuracy at different measurement rates (i.e., the number of measurements in a snapshot) compared to the signal size. Reconstruction accuracy is reported relative to the normalized mean square error in the reconstructed signal, where smaller errors are better. The figure also shows a comparison with a naive method, where it is assumed that the approximate deformation is correct and no correction is computed, while still attempting to reconstruct the signal (labeled "ignore P"). i "No noise" is shown (and is calibrated with a solid black line and an ×). A comparison with methods for correcting approximate deformations known in the art is also shown (labeled "gradient, no noise" and calibrated with a solid black line and a +). Both comparisons assume the absence of measurement noise, an unrealistic assumption that favors both methods.

[0214] The performance of the embodiments of this disclosure at various noise levels is indicated using dashed and light-colored lines, labeled "Input SNR = XX dB", where XX represents the input noise level. Since these are noisy experiments, shaded areas around the lines are used to delineate the variability of the method; these shaded areas represent one standard deviation above and below the mean.

[0215] As is evident in the figure, even under ideal conditions of noise-free measurement and high measurement rate, existing techniques have failed to accurately reconstruct signals. In contrast, embodiments of this disclosure are able to reconstruct signals with high fidelity by assuming a sufficient measurement rate given a noise level.

[0216] Figure 12B Further experimental results are shown, illustrating the performance of this embodiment at a fixed input SNR of 20 dB as the number of views increases. The graph plots the performance of each view for two different measurement rates. As shown, performance improves with increasing number of views, and therefore, the total measurement rate also increases; a higher measurement rate per view results in better performance. In each experiment, the total measurement rate is equal to the number of views multiplied by the measurement rate per view.

[0217] Figure 13Hardware diagrams of different components of a radar imaging system 1300 according to some embodiments of the present disclosure are shown. The radar imaging system 1300 includes: a processor 1320 configured to execute stored instructions; and a memory 1340 storing instructions executable by the processor. The processor 1320 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 1340 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 1320 is connected to one or more input and output devices via a bus 1306.

[0218] These instructions implement a method for reconstructing a radar reflectivity image of an object in its prototype pose. For this purpose, the radar imaging system 1300 may also include a storage device 1330 adapted to store different modules for storing executable instructions for the processor 1320. The storage device stores: a deformation module 1331 configured to estimate the deformation of the object in each snapshot using measurements 1334 of optical sensor data; and a transformation module 1332 configured to obtain a radar reflectivity image F. i Transformation of radar reflectivity image F i It is an optical distortion F i The estimation; and the reconstruction module 1333, which is configured to use the estimation F i To replace the real F i And optionally, regularization can be applied to solve x in equation (5) above, as described above. Storage device 1330 can be implemented using a hard disk drive, optical disk drive, thumb drive, drive array or any combination thereof.

[0219] Still referencing Figure 13 The radar imaging system 1300 includes an input interface for receiving measurements 1395 from optical and electromagnetic sensors. For example, in some implementations, the input interface includes a human-machine interface 1310 within the radar imaging system 1300 that connects the processor 1320 to a keyboard 1311 and a pointing device 1312, wherein the pointing device 1312 may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, etc.

[0220] Alternatively, the input interface may include a network interface controller 1350 adapted to connect the radar imaging system 1300 to a network 1390 via a bus 1306. Through the network 1390, measurements 1395 can be downloaded and stored in a storage system 1330 as training and / or operational data 1334 for storage and / or further processing.

[0221] Still referencing Figure 13The radar imaging system 1300 includes an output interface for rendering a prototype radar reflectivity image of an object in a prototype pose. For example, the radar imaging system 1300 may be linked via a bus 1306 to a display interface 1360 suitable for connecting the radar imaging system 1300 to a display device 1365, wherein the display device 1365 may include a computer monitor, camera, television, projector, or mobile device, etc.

[0222] For example, radar imaging system 1300 can be connected to system interface 1370, which is adapted to connect the radar imaging system to a different system 1375 controlled based on the reconstructed radar reflectivity image. Alternatively, radar imaging system 1300 can be connected via bus 1306 to application interface 1380, which is adapted to connect radar imaging system 1300 to application device 1385 that can operate based on the results of image reconstruction.

[0223] Figure 14 This is a schematic diagram illustrating a computing device 1400 by way of a non-limiting embodiment according to embodiments of the present disclosure. The computing device or apparatus 1400 represents various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers.

[0224] The computing device 1400 may include a power supply 1408, a processor 1409, a memory 1410, and a storage device 1411, all connected to the bus 1450. Additionally, a high-speed interface 1412, a low-speed interface 1413, a high-speed expansion port 1414, and a low-speed connection port 1415 may be connected to the bus 1450. Furthermore, a low-speed expansion port 1416 is connected to the bus 1450. Through non-limiting embodiments, various component configurations that may be mounted on a general-purpose motherboard depending on the specific application are contemplated. Furthermore, an input interface 1417 may be connected to an external receiver 1406 and an output interface 1418 via the bus 1450. A receiver 1419 may be connected to external transmitters 1407 and 1420 via the bus 1450. External memory 1404, external sensors 1403, a machine 1402, and an environment 1401 may also be connected to the bus 1450. Furthermore, one or more external input / output devices 1405 may be connected to the bus 1450. The network interface controller (NIC) 1421 may be adapted to be connected to the network 1422 via the bus 1450, wherein data or other data may be presented on a third-party display device, a third-party imaging device and / or a third-party printing device other than the computer device 1400.

[0225] Still referencing Figure 14It is conceivable that memory 1410 may store instructions executable by computer device 1400, as well as historical data and any data available to the methods and systems of this disclosure. Memory 1410 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Memory 1410 may be one or more volatile memory cells and / or one or more non-volatile memory cells. Memory 1410 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0226] Still referencing Figure 14 Storage device 1411 may be adapted to store supplementary data and / or software modules used by computer device 1400. For example, storage device 1411 may store historical data and other relevant data mentioned above with respect to this disclosure. Additionally or alternatively, storage device 1411 may store historical data similar to the data mentioned above with respect to this disclosure. Storage device 1411 may include hard disk drives, optical disk drives, thumb drives, drive arrays, or any combination thereof. Furthermore, storage device 1411 may contain computer-readable media (including devices in storage area networks or other configurations), such as floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, flash memory, or other similar solid-state storage devices or device arrays. Instructions may be stored in an information carrier. When executed by one or more processing devices (e.g., processor 1409), the instructions perform one or more methods, such as the methods described above.

[0227] The system can be optionally linked to a display interface or user interface (HMI) 1423 via bus 1450, which is adapted to connect the system to a display device 1425 and a keyboard 1424, wherein the display device 1425 may include a computer monitor, camera, television, projector or mobile device, etc.

[0228] Still referencing Figure 14 The computer device 1400 may include a user input interface 1417 adapted for a printer interface (not shown), or may be connected via a bus 1450 and adapted to be connected to a printing device (not shown), wherein the printing device may include a liquid inkjet printer, a solid ink printer, a large commercial printer, a thermal printer, a UV printer, or a dye-sublimation printer, etc.

[0229] High-speed interface 1412 manages bandwidth-intensive operations of computing device 1400, while low-speed interface 1413 manages lower bandwidth-intensive operations. This functional allocation is just one embodiment. In some embodiments, high-speed interface 1412 may be connected to memory 1410, user interface (HMI) 1423, keyboard 1424, and display 1425 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 1414, which can accept various expansion cards (not shown) via bus 1450. In an implementation, low-speed interface 1413 is connected to storage device 1411 and low-speed expansion port 1415 via bus 1450. Low-speed expansion port 1415 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be connected (e.g., via a network adapter) to one or more input / output devices 1405, other devices (including keyboard 1424, pointing device (not shown), scanner (not shown), or network devices (e.g., switches or routers)).

[0230] Still referencing Figure 14 The computing device 1400 can be implemented in several different forms, as shown in the figure. For example, the computing device can be implemented as a standard server 1426, or multiple times in a group of such servers. Furthermore, the computing device can be implemented in a personal computer such as a laptop computer 1427. The computing device can also be implemented as part of a rack server system 1428. Alternatively, components from the computing device 1400 can be integrated with other components (such as…) Figure 13 (The implementation methods) are combined. Each of such devices may include one or more of computing devices 1300 and 1400, and the entire system may consist of multiple computing devices that communicate with each other.

[0231] feature

[0232] One aspect may include a measurement sensor that captures measurements of an object deforming in a scene at multiple time steps within a time period by continuously capturing snapshots of the object at multiple time steps, and sequentially sends the measurement data to a processor, wherein the object exhibits different deformations at each of the multiple time steps. Another aspect involves a tracking system that tracks the deformable object during the same time period as or different time periods in which the measurement sensor captures snapshots of the object's deformation.

[0233] In another aspect, the deformation is caused wholly or partially by the movement of the object in the scene, or by the movement of the measuring sensor while capturing the scene. In another aspect, the system is a coherent imaging system, such as a radar imaging system, a magnetic resonance imaging system, or an ultrasound imaging system. In yet another aspect, an optimization that minimizes a cost function is used to calculate a correction for the deformation estimate of the object for each time step, the cost function including: the estimated distance by which the deformation moves the elements of the object; and a measurement level at which the deformed object matches the measurement of the tracking system. In yet another aspect, based on matching the measurement of the corrected deformation of the object for each time step with the measurement in a snapshot of the object acquired at that time step using the cost function, the cost function penalizes the amount of distance between the measurement of the corrected deformation of the object and the measurement of the object in the snapshot at that time step. Another aspect is that the correction deformation for which the time step estimate is superior to other correction deformations is based on the distance between the correction deformation and the initial estimate of the deformation and on the use of a cost function, which penalizes the deformation correction more for deformations in which the elements of the object have moved a greater distance than their deformation positions.

[0234] One aspect is the optimal transport problem, which includes penalizing the cost of deformations based on the amount of distance by which the elements of the object image are moved from their positions, and penalizing the cost of deformations based on a matching score level of the degree of matching between the measurement of the corrected deformation of the object and the measurement of the tracking system. This aspect is that the deformed object in the scene is one of a mammal (including humans), an amphibian, a bird, a fish, an invertebrate, or a reptile, wherein the deformed object in the scene is an organ within the body of the human, an organ within the amphibian, an organ within the bird, an organ within the fish, an organ within the invertebrate, or an organ within the reptile.

[0235] Another aspect involves labeling the final estimate of the deformation of the deformable object, the final image of the object, or both, as an object report and outputting it to and receiving it via a communication network associated with an entity such as an operator of the system. The operator generates at least one action command, which is sent to and received by a controller associated with the system. The controller implements the generated at least one action command to change the characteristics of the object based on the object report. In one aspect, the characteristics of the object include one or a combination of the object's defects, the object's medical condition, the presence of weapons on the object, or the presence of unwanted artifacts on the object. In another aspect, the at least one action command includes one or a combination of the following: an object defect inspection level from a set of different levels of object defect inspection; an object medical testing level from a set of different levels of object medical testing; and an object safety and security inspection level from a set of different levels of object safety and security checks.

[0236] Another aspect is that the tracking sensor has one or a combination of an optical camera, a depth camera, and an infrared camera, wherein the electromagnetic sensor includes one of a millimeter-wave radar, a ThZ imaging sensor, and a backscatter X-ray sensor. A further aspect is that the electromagnetic sensor is a plurality of electromagnetic sensors with a fixed aperture size, wherein the processor estimates a radar image of the object from a radar reflectivity image of the scene for each of the plurality of electromagnetic sensors by combining measurements from each of the plurality of electromagnetic sensors for each of the plurality of time steps. The plurality of electromagnetic sensors move according to a known motion, and the processor adjusts the transformation of the radar reflectivity image of the object acquired by the plurality of electromagnetic sensors at the corresponding time step based on the known motion of the plurality of electromagnetic sensors for the corresponding time step. One aspect is that the resolution of the radar reflectivity image of the scene is greater than the resolution of an initial estimate of the deformation of the object at each time step.

[0237] definition

[0238] Types of radar and radar sensors: Radar can appear in various configurations as transmitters, receivers, antennas, wavelengths, scanning strategies, etc. For example, some radars may include bistatic radar, continuous wave radar, Doppler radar, frequency-modulated continuous wave (FM-CW) radar, monopulse radar, passive radar, planar array radar, pulse radar with arbitrary waveforms, pulse Doppler radar, multistatic radar, synthetic aperture radar, synthetic sparse aperture radar, over-the-horizon radar with a linear frequency modulated transmitter, interferometric radar, polarization radar, array-based radar, or MIMO (multiple-input multiple-output) radar, etc. It is contemplated to combine one or more types of radar and radar sensors with one or more embodiments of the radar imaging system of this disclosure.

[0239] Implementation

[0240] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an advantageous description of implementing one or more exemplary embodiments. It is contemplated that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the subject matter of the disclosure as set forth in the appended claims.

[0241] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adjustments and modifications can be made within the spirit and scope of this disclosure. Therefore, the appended claims cover all such variations and modifications within the true spirit and scope of this disclosure.

Claims

1. An imaging system, the imaging system comprising: A tracking sensor configured to track deformable objects within a scene at multiple time steps over a period of time to generate an initial estimate of the deformation of the objects, the tracking sensor being configured to acquire an optical reflectance image of the scene; A measurement sensor configured to capture measurement data by capturing snapshots of the object deformed in the scene at said multiple time steps within said time period; as well as The processor is configured as follows: Based on each snapshot of the acquired object, deformation information of the deformable object is calculated for the measurement data to produce a set of measurement results of the object having a deformed shape at the plurality of time steps, wherein each snapshot has the measurement results of the object being deformed during that time period, and for each of the plurality of time steps, the processor sequentially calculates the deformation information of the object by calculating a correction to the deformation estimate of the object. The calculation of the correction includes: For each time step, the measurement results of the corrected deformation of the object for the corresponding time step are matched with the measurement results in the snapshot of the object acquired for that corresponding time step, and For each time step, based on the distance between the corrected deformation and the initial estimate of the deformation, a corrected deformation that is superior to other corrected deformations is selected for that time step, and Obtain a final estimate of the deformation of the deformable object moving in the scene, and a final image of the object moving within the scene. The processor is further configured to solve an optimal transport problem, which includes: penalizing the cost of deformations based on the amount of distance by which the elements of the object image are moved from their positions; and penalizing the cost of deformations based on a matching score level of the degree of matching between the measurement results of the corrected deformation of the object and the measurement results of the tracking sensor.

2. The imaging system according to claim 1, wherein, The measurement sensor captures the measurement results of the object deforming in the scene at the multiple time steps within the time period by continuously capturing snapshots of the object at the multiple time steps within the time period, and sequentially sends the measurement data to the processor, wherein the object exhibits different deformations at each of the multiple time steps.

3. The imaging system according to claim 1, wherein, The tracking sensor tracks the deformable object during different time periods than the measurement sensor captures snapshots of the object's deformation.

4. The imaging system according to claim 1, wherein, The deformation is caused wholly or partially by the movement of the object in the scene, or the deformation is caused wholly or partially by the movement of the measuring sensor when it captures the scene.

5. The imaging system according to claim 1, wherein, The imaging system is one of a radar imaging system, a magnetic resonance imaging system, or an ultrasound imaging system.

6. The imaging system according to claim 1, wherein, The tracking sensor is one or a combination of an optical camera, a depth camera, and an infrared camera, and the measurement sensor is at least one electromagnetic sensor, which includes one or a combination of a millimeter-wave radar, a terahertz imaging sensor, and a backscatter X-ray sensor.

7. The imaging system according to claim 1, wherein, The optimization of minimizing a cost function is used to calculate the correction for the deformation estimate of the object for each time step. The cost function includes: the estimated distance by which the deformation moves the elements of the object; and a measurement level of how well the deformed object matches the measurements of the tracking sensor.

8. The imaging system according to claim 7, wherein, Based on matching the measurement results of the corrected deformation of the object for each time step with the measurement results in a snapshot of the object acquired at that time step using a cost function, the cost function penalizes the amount of distance between the measurement results of the corrected deformation of the object and the measurement results of the object in the snapshot acquired at that time step, or Specifically, the correction deformation for which the time step estimate is superior to other correction deformations is based on the distance between the correction deformation and the initial estimate of the deformation and is performed using a cost function, which penalizes the correction deformation for deformations where the elements of the object have moved a greater distance than their deformation positions.

9. The imaging system according to claim 1, wherein, The deformed object in the scene is one of the following: a mammal (including a human), an amphibian, a bird, a fish, an invertebrate, or a reptile. The deformed object in the scene is an organ inside the human body, an organ inside the amphibian, an organ inside the bird, an organ inside the fish, an organ inside the invertebrate, or an organ inside the reptile.

10. The imaging system according to claim 1, wherein, The final estimate of the deformation of the deformable object, the final image of the object, or both are marked as an object report and output to and received by a communication network associated with an entity of the system's operator. The operator generates at least one action command, which is sent to and received by a controller associated with the system. The controller implements the generated at least one action command to change the properties of the object based on the object report.

11. The imaging system according to claim 10, wherein, The characteristics of the object include one or a combination of the object's defects, the object's medical condition, the presence of weapons on the object, or the presence of unwanted artifacts on the object.

12. An image processing method, the image processing method comprising: Deformable objects within a scene are tracked at multiple time steps over a period of time using a tracking sensor to estimate the deformation of the objects, wherein the tracking sensor acquires an optical reflectance image of the scene; Measurement data is obtained by continuously capturing snapshots of the deformed object in the scene at the multiple time steps within the time period; Based on the measurement data, deformation information of the deformed object is calculated such that each snapshot of the acquired image includes the measurement results of the deformed object at that time step, to produce a set of measurement results of the object having a deformed shape at the plurality of time steps. For each of the plurality of time steps, the deformation information of the object is calculated by calculating a correction to the deformation estimate of the object. The calculation of the correction includes: For each time step, the measurement results of the corrected deformation of the object for the corresponding time step are matched with the measurement results in the snapshot of the object acquired for that corresponding time step, and For each time step, based on the distance between the corrected deformation and the initial estimate of the deformation, a corrected deformation that is superior to other corrected deformations is selected for that time step. Obtain a final estimate of the deformation of the deformable object moving in the scene and a final image of the object moving within the scene; store the final estimate and the final image. The image processing method further includes solving an optimal transport problem, which includes: penalizing the cost of deformation based on the amount of distance by which the elements of the object image are moved from their positions; and penalizing the cost of deformation based on a matching score level of the degree of matching between the measurement results of the corrected deformation of the object and the measurement results of the tracking sensor.

13. A radar system for estimating the deformation of a deformable object, the radar system comprising the imaging system according to claim 1.

14. The radar system according to claim 13, wherein, The measurement sensor includes multiple electromagnetic sensors with a fixed aperture size, wherein the processor estimates a radar image of the object from a radar reflectivity image of the scene for each of the multiple electromagnetic sensors by combining the measurement results from each of the multiple electromagnetic sensors. The plurality of electromagnetic sensors move according to a known motion, and The processor adjusts the transformation of the radar reflectivity image of the object acquired by the plurality of electromagnetic sensors at the corresponding time step based on the known motion of the plurality of electromagnetic sensors at the corresponding time step.

15. The radar system according to claim 13, wherein, The resolution of the radar reflectivity image of the scene is greater than the resolution of the initial estimate of the deformation of the object at each time step.

Images