Systems and methods for motion estimation - Patents.com

By integrating video-based motion estimation with mmWave imaging, the system addresses the challenge of moving objects in commercial applications, achieving improved image quality and accuracy.

JP7580630B2Active Publication Date: 2024-11-11THRUWAVE INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023562264
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-04-09
Filing Date
2022-03-31
Publication Date
2024-11-11
Estimated Expiration
2042-03-31

AI Technical Summary

Technical Problem

Existing lensless millimeter wave (mmWave) imaging systems struggle with accurate object position information when objects move independently of the imaging system, limiting their effectiveness in commercial applications.

Method used

A system that combines a video camera with a mmWave antenna array to calculate motion vectors and position estimates, using compressed video data to correct motion and generate high-quality 3D mmWave images of moving objects.

Benefits of technology

Improves image quality by accurately estimating object motion, reducing position errors to less than 1/8 of a wavelength, enabling high-throughput imaging in commercial settings.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580630000012
    Figure 0007580630000012
  • Figure 0007580630000013
    Figure 0007580630000013
  • Figure 0007580630000014
    Figure 0007580630000014
Patent Text Reader

Abstract

This application is directed to a video-based motion extraction system using mmWave imaging. In one example, object velocity is extracted in real time from at least one motion vector obtained from a compressed video. The velocity information is combined with range information obtained from a range sensor. The combination of range and velocity is used to calculate the position of the object at each time instance. A velocity profile may be generated. The velocity profile may then be used to reconstruct an image of the object passing in front of a mmWave antenna array.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to systems and methods for estimating motion from video for high-throughput lensless 3D mmWave imaging. [Background technology]

[0002] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 173,318, filed April 9, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0003] (overview) Lensless millimeter wave (mmWave) imaging of moving objects using sparse arrays relies on knowledge of the relative position between the moving object and the imaging system to enable coherent image reconstruction. However, accurate object position information is rarely available in commercial applications where the moving object, such as a conveyor belt or robot, is controlled independently of the imaging system or the imaged object moves autonomously. This can be a hurdle for many commercial mmWave imaging applications.

[0004] Therefore, there is an increasing need for systems and methods that can address the challenges of millimeter wave imaging of objects when accurate object position information is unavailable due to object motion that is controlled independently of the imaging system.

[0005] In some embodiments, a system for generating a mmWave image may include a video camera operative to capture video frames of a moving object, a mmWave antenna array proximate to the video camera and operative to capture mmWave data frames, and a motion vector processor. In some embodiments, the motion vector processor may be configured to calculate one or more motion vectors using the video frames, the one or more motion vectors representing relative motion between the video camera and the object. The system may include at least one of a position processor and a velocity processor, the position processor configured to calculate one or more position estimates, and the velocity processor configured to calculate one or more velocity estimates of the object relative to the mmWave antenna array using the one or more motion vectors. The system may also include an image reconstruction processor configured to calculate a mmWave image of the object using the mmWave data frames and at least one of the corresponding position and velocity estimates.

[0006] In some aspects of the disclosed technology, the video camera generates a compressed video dataset. In some aspects, one or more motion vectors are extracted from the compressed video dataset. In various aspects, the compressed video dataset includes an MPEG video dataset. In some aspects, the mmWave image includes a three-dimensional image of the object. In some aspects of the technology, the position and velocity estimates are interpolated between video frames. In various aspects, the object is transported via one of a conveyor belt or a robot. In some aspects, the object is stationary and the video camera and mmWave antenna array are transported by a moving platform. In some aspects, the moving platform includes a moving robot. In some aspects, the relative motion between the video camera and the object includes a non-constant velocity.

[0007] In some embodiments, a method for generating a mmWave image of a moving object can include capturing two or more video frames of the moving object using a video camera proximate to a mmWave antenna array and capturing two or more mmWave frames of data using the mmWave antenna array. The method can include calculating one or more motion vectors representing relative motion between the video camera and the object, and using the motion vectors to calculate at least one of one or more position estimates and one or more velocity estimates of the object relative to the mmWave antenna array. The method can also include calculating a mmWave image using the mmWave frames of data and calculating corresponding at least one of the position estimates and velocity estimates using an image reconstruction algorithm.

[0008] In some embodiments, a method for estimating motion of an object in a video can include receiving at least one recording of an object passing in front of a mmWave antenna array and calculating at least one velocity associated with the object, the velocity being calculated based on at least one motion vector. The method can include receiving at least one distance indicator associated with the object from a distance sensor and combining the at least one velocity and the at least one distance indicator. The method can also include generating a velocity profile of the object based on a combination of the at least one velocity and the at least one distance indicator, and constructing at least one image based on the velocity profile of the object.

[0009] It is with respect to these and other general considerations that the aspects disclosed herein have been made, and although relatively particular problems may be discussed, it should be understood that the embodiments should not be limited to solving the particular problems identified herein. [Brief description of the drawings]

[0010] [Figure 1] 1A-1C are diagrams illustrating an example mmWave antenna array and camera configuration showing the field of view of each component and the path of an imaged object as it passes in front of the array. [Diagram 2] 1 is an exemplary flow chart illustrating processing steps for incorporating range and motion data into mmWave image reconstruction. [Figure 3A] FIG. 13 illustrates an example horizontal velocity measured across the field of view of frames taken at two different times during a single measurement. [Figure 3B] FIG. 13 illustrates an example horizontal velocity measured across the field of view of frames taken at two different times during a single measurement. [Figure 4] 1 is an exemplary graph illustrating a correction factor as a function of estimated speed. [Figure 5A] FIG. 13 illustrates an example graph of an estimated velocity profile compared to ground truth data and the position error resulting from the discrepancy between actual and estimated motion. [Figure 5B] FIG. 13 illustrates an example graph of an estimated velocity profile compared to ground truth data and the position error resulting from the discrepancy between actual and estimated motion. [Figure 6A] As shown in FIG. 5, an example scene with and without motion compensation for three different velocity profiles is shown. [Figure 6B] As shown in FIG. 5, an example scene with and without motion compensation for three different velocity profiles is shown. [Figure 6C] As shown in FIG. 5, an example scene with and without motion compensation for three different velocity profiles is shown. [Figure 6D] As shown in FIG. 5, an example scene with and without motion compensation for three different velocity profiles is shown. [Figure 6E] As shown in FIG. 5, an example scene with and without motion compensation for three different velocity profiles is shown. [Figure 6F] As shown in FIG. 5, an example scene with and without motion compensation for three different velocity profiles is shown. [Figure 7A] FIG. 13 illustrates an exemplary graph showing the accuracy of motion estimation for a validation dataset of 100 sample images. [Figure 7B] FIG. 13 illustrates an exemplary graph showing the accuracy of motion estimation for a validation dataset of 100 sample images. [Figure 8] FIG. 1 is a block diagram illustrating an overview of a device in which some embodiments of the disclosed technology can operate. [Figure 9] FIG. 1 is a block diagram illustrating an overview of an environment in which some embodiments of the disclosed technology can operate. [Figure 10] FIG. 1 is a block diagram illustrating components that can be used in systems employing the disclosed technology in some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which form a part of this specification and show certain exemplary aspects. However, different aspects of the present disclosure may be implemented in many different forms and should not be construed as being limited to the aspects set forth herein. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the aspects to those skilled in the art. The aspects may be implemented as a method, a system, or a device. Thus, the aspects may take the form of a hardware implementation, a fully software implementation, or an implementation combining software and hardware aspects. Thus, the following detailed description should not be construed in a limiting sense.

[0012] A video-based motion extraction approach for active mmWave imaging is presented. In one example, object velocities are extracted in real-time from motion vectors obtained from compressed videos. This information may be combined with readouts from range sensors to infer the object's position at each point in time. Leveraging video-derived motion vectors may allow offloading the computational complexity of 2D spatial correlation to highly optimized algorithms operating on the camera frames.

[0013] Image quality in commercial high-throughput 3D mmWave imaging systems can be improved by this approach when the target velocity is unknown and time-varying, and image quality can also be improved by comparing it to a known average motion profile of the imaged object.

[0014] In one example, the root mean square (RMS) position error may be 2.5 mm over a travel length of 0.52 m, which may provide advantages including achieving errors of less than 1 / 8 of a wavelength at K-band (24 GHz) along the track, and therefore sufficient to achieve improved image quality at K-band and longer wavelengths.

[0015] The region of the electromagnetic spectrum known as millimeter waves (mmWave) includes wavelengths between 1 and 20 mm, corresponding to a frequency range of 15 to 300 GHz. Imaging performed at mmWaves may offer advantages compared to either optical or x-ray wavelengths. Many materials that are opaque at optical wavelengths, such as drywall, cardboard, and many plastics, are nearly transparent to mmWaves, providing the opportunity to image areas that would otherwise be hidden, for example behind walls or within sealed packages or containers. Furthermore, in contrast to x-rays, mmWaves are non-ionizing electromagnetic waves, making them safe for humans without the need for cumbersome and expensive shielding.

[0016] These characteristics make mmWave imaging an ideal choice for many commercial and industrial applications. Previous implementations of lensless computational mmWave imaging systems have been limited to relatively low frame rates (e.g., 0.25 Hz), but recent advances in mmWave imager design enable 3D mmWave imaging for high-throughput applications. For example, mmWave imagers can be used in warehouse settings to look inside packaged goods to count objects, detect anomalies such as lost or damaged items or liquid leaks, and estimate the amount of free space within a package to ensure commercially efficient shipping.

[0017] In warehouse or industrial applications, for example, when mmWave is used to image continuously moving objects, precise position information of each object relative to the imaging system can help generate high quality images. In such applications, the scanned objects are typically transported by conveyor belts or robots whose movements are not controlled by the imaging system and whose speeds may not be constant. It may therefore be advantageous to measure the object's motion over the time interval of each mmWave scan, so that appropriate motion corrections can be applied to the mmWave image reconstruction to ensure that the mmWave image is properly focused throughout the duration of the scan.

[0018] A video-based motion extraction approach is described herein. A camera located near a mmWave antenna array may record a scene (or multiple scenes) as an object passes in front of the array. Object velocity is then extracted in real time from motion vectors derived from the compressed video. This information is combined with readouts from a range sensor to infer the exact position of the object at each time point over the imaging interval. An output of a velocity profile may be generated that is used in image reconstruction to generate high quality images.

[0019] Exemplary Imaging System FIG. 1 is an exemplary mmWave antenna array and camera configuration, showing the field of view of each component and the path of an object to be imaged as it passes in front of the array. An example 3D mmWave imaging hardware system may be used in conjunction with the motion estimation system described below with respect to FIG. 3A and FIG. 3B. The imaging system is composed of 32K-band multiple-input multiple-output (MIMO) mmWave subarrays arranged in two 1-D vertical arrays of 16 subarrays each. The columns of subarrays are arranged along the sides of a conveyor, which carries the object to be imaged in front of the entire array at an average speed of 0.3 m / s. Two columns are arranged adjacent to each other parallel to one side of the conveyor with a spacing of 160 mm, spanning a vertical length of 1800 mm. Note that the two columns are slightly offset in the z-direction by 50 mm. However, both columns form a single logical array. An optical trigger sensor is used to trigger measurements as an object passes. An optical camera arranged adjacent to the array records the video used for motion estimation. A diagram of the imaging system is shown in Figure 1.

[0020] This setup is designed to closely model a commercial package scanning application, where the object to be imaged is transported via a conveyor belt or robot along a path parallel to the mmWave sensor array. In this particular setup, a conveyor belt is used to supply the motion, but modern warehouses may use autonomous robots to transport goods between locations. The autonomous robot may calculate the optimal path through the warehouse to complete its assigned tasks. It may also adjust its motion to optimize traffic flow, adapt to differences in product weight, and avoid collisions. As a result, some robots may move at the same speed, while others may have different speed profiles, and the speed of individual robots may vary as they pass the imaging array. This non-constant speed may be modeled by varying the speed of the conveyor belt, thus allowing for simulating a robotic system in a conveyor-based test setup.

[0021] (Motion Estimation System) Motion vectors are used in many types of video systems, including those that comprise elements of MPEG format video compression. Motion vectors are two-dimensional vectors that represent the change in coordinates of blocks of pixels from one frame to the next. They are related to the optical flow of the image frames, which is the apparent motion that results from the projection of a three-dimensional scene onto a two-dimensional image. Motion vectors can be a convenient proxy for representing optical flow, since they are a sparse representation of the scene (and therefore fast to analyze compared to optical flow) and can be easily extracted directly from the compressed MPEG data. For these reasons, the extraction of motion vectors from compressed MPEG data has found many applications in object detection and tracking.

[0022] The video camera is placed adjacent to the antenna array so that the camera's field of view overlaps with the field of view of the antenna array, as shown in FIG. 1. The video may be recorded in MPEG4 format at 59 frames per second (fps) with a resolution of 1440×1080 pixels. Lens correction may be applied, and then the video is re-encoded to 20 fps. The selection of the video frame rate involves a trade-off between quantization noise and distortion effects in the motion vectors. Quantization noise is present because the motion vectors represent a shift in position over multiple pixels. A lower frame rate may produce a larger shift between frames, which reduces the effects of this quantization. However, a larger shift may also increase the effects of changes due to perspective, uneven lighting conditions, and lens distortion. Thus, a lower frame rate may reduce correlation between frames, resulting in more noise.

[0023] FIG. 2 is an exemplary flow chart showing the processing steps for incorporating distance and motion data into mmWave image reconstruction. The video image processing system may use ffmpeg and mpegflow as readily available tools to perform lens correction and extract motion vectors, or it is possible to perform lens correction directly on the motion vectors. Frames are recorded over the interval of the mmWave scan, and a motion vector corresponding to the scene velocity is extracted for each frame. Compared to using break beam sensors to detect scene motion, this approach may allow continuous observation over the entire duration of the imaging scan, which may allow for correction of time-varying motion, whereas break beam sensors or similar position detectors can only generate an average velocity over the sensor interval. FIG. 2 shows the process by which the video data is converted into a motion profile and incorporated into the image reconstruction.

[0024] 3A and 3B show example horizontal velocities measured across the field of view of frames acquired at two different times during a single measurement. After motion vectors are extracted from the camera frames using mpegflow, the overall velocity of the object is estimated by taking the quantile average over regions with consistent motion and applying a correction factor derived from the training dataset.

[0025] Using quantiles rather than the overall average reduces the effect of inevitable noise due to low correlation between frames, for example, when an object moves in and out of the camera's field of view. When correlation is low, motion vectors can jump in any direction, since there may be blocks of pixels with good correlation randomly in different directions. Considering only the quantile average helps to eliminate motion vectors with low correlation. In addition, by using high quantiles instead of the median, the algorithm uses the front of the object rather than parts of the field of view that are located further back, since points that are further away appear to move at a slower speed due to perspective.

[0026] Figure 3 shows two sample frames from the camera capture with an overlay of the estimated horizontal velocity across the entire field of view. In this test, the object's velocity increased to a peak of 0.4 m / s and then decreased over the time interval of the measurement. The difference in the estimated velocity between the two frames is clear. Furthermore, it is possible to observe the presence of distortions, and the estimated velocity is not constant across the region of interest. Using only the high quantiles of the motion vectors to estimate the overall velocity minimizes the error that would be caused by these distortions.

[0027] Figure 4 is an exemplary graph showing the correction factor as a function of estimated speed. As shown in Figure 1, the camera is offset from the radar array along an axis parallel to the conveyor, and the field of view of the camera is not identical to the field of view of the radar. As a result, the motion profile obtained by analysis of the camera frames cannot be directly applied to the radar data, but rather a correction to the motion profile must be applied to compensate for the different field of view.

[0028] A training set with 50 randomly generated velocity profiles and known ground truth may be used to estimate the correction factor. Figure 4 shows the resulting correction factor as a function of velocity estimated from the camera data. This correction factor is also converted from pixels per frame measured by the motion vector to velocity in mm / s observed by the mmWave imaging system.

[0029] (Image reconstruction) For the purpose of reconstructing an image from mmWave data, the observed scenes are each characterized by a complex reflection coefficient ρ n The mmWave antenna array is modeled as a collection of N point scatterers with m∈{1,...,M} and l∈{1,...,L}. For each location m∈{1,...,M} and each frequency l∈{1,...,L}, the received signal y m,l is the sum of the reflections from the point scatterers that comprise the scene.

[0030]

number

[0031] In the formula, c m,l,n represents the path loss and antenna response, and ω l is the angular frequency, and τ m,n is the round trip time delay between the antenna position m and the nth point scatterer, and ν m,l is the measurement noise,

[0032]

number

[0033] It is.

[0034] The measurement vector for a single antenna pair m at frequency l is y m Vector y = [y1 T y M T ] as an element of y m , ρ = [ρ n ρ N ] T As ρ n where (.) T represents the transpose operator, and the vector is

[0035]

number

[0036] can be written as, where H is the measurement matrix giving the degree of dependence between the point scatterers and each measurement, and v is the measurement noise.

[0037] The goal of the image reconstruction process is to estimate the complex reflectance coefficients of the points that compose the scene. This is achieved by solving the inverse problem defined in (2). This may not be solved exactly, but it is possible to approximate the inverse using a matched filter in H, resulting in an estimate

[0038]

number

[0039] You can get

[0040]

number

[0041] In the formula, (.) * denotes the complex conjugate transpose operator.

[0042] The value of each round trip time delay τ depends on the relative position of the antenna array and precise knowledge of the points that form the scene. Given a known position y of an object at time t0=0, when a measurement is triggered, the position of the object relative to the array in subsequent frames can be found by integrating the velocity profile. n For a discrete set of velocity measurements taken at, the position of the object at the corresponding time is

[0043]

number

[0044] is given by:

[0045] The MmWave imaging array forms a velocity profile, over time t n Different from time t m Capture a frame of data at t m ) is set to obtain the best approximation to the velocity in each mmWave frame, v(t n ) The object's position in mmWave frame m is then

[0046]

number

[0047] is given by:

[0048] The number of mmWave data frames is typically much larger than the number of points in the estimated velocity profile over the same time span, and therefore (t m-1 ,t m ) there may be no advantage to interpolating the velocity profile to the midpoint.

[0049] (Results and Discussion) 5A and 5B show exemplary graphs of estimated velocity profiles (solid lines) compared to ground truth data (dashed lines), as well as position errors resulting from discrepancies between actual and estimated motion. Line 502 represents a profile with an approximately constant velocity, line 504 illustrates a scenario where the velocity increases and then decreases, and line 506 represents a slowly increasing velocity profile. We illustrate by considering three scenarios for the velocity profile of the imaged object: an object with an approximately constant velocity close to 0.3 m / s, an object with a non-uniform velocity that first increases and then decreases, and an object whose velocity increases over the time interval of the measurement. Some of these scenarios may occur in real-world applications because of changes in conveyor speed, or because the imaged object is being carried by an autonomous robot whose path and velocity are adjusted to its surroundings. The robot may temporarily speed up to avoid a collision or accelerate slowly due to a high load. Similarly, the conveyor belt speed may be responsive to the weight of the objects on the belt, the available power from a drive motor, or a control signal from an upstream or downstream control system (such as a programmable logic control, or warehouse execution system) that controls the particular conveyor. Figure 5a shows these three speed profiles estimated from the data, along with a ground truth profile for reference. Figure 5b shows the position error that occurs due to the discrepancy between the estimated speed and the ground truth.

[0050] 6A-6F show an example scene with and without motion compensation for three different velocity profiles as shown in FIG. 5. FIG. 6A-6F display image reconstruction with and without motion compensation. Image reconstruction without motion compensation uses an estimated constant velocity of about 0.3 m / s. Image reconstruction with motion compensation applies the velocity profile shown in FIG. 5a. When the estimated motion is taken into account, image quality improves in some cases. The greater the difference between the true motion of the object and the average velocity, the more dramatic the improvement in image quality is.

[0051] One measure of image focus is the total power per pixel, (6)

[0052]

number

[0053] where f(y,z) represents the image values. The increase in total power for each of the three scenarios when motion estimation is applied is shown in Table 1. The increase in total power corresponding to the improvement in image focus is 1.14 times for constant velocity, 2.15 times for increasing and decreasing velocity, and 1.08 times for slowly increasing velocity. As expected, the second scenario with the largest velocity variation may benefit most from motion compensation.

[0054] [Table 1]

[0055] Another measure of image focus is the range of the image histogram, (7)

[0056]

number

[0057] where h(k) is the number of image pixels with value k. For a histogram of 100 bins between -5 and 0, motion compensation improves this measure of image focus for several images, as shown in Table 2.

[0058] [Table 2]

[0059] Figures 7A and 7B show exemplary graphs illustrating the accuracy of motion estimation for a validation dataset of 100 sample images. A validation dataset was collected with 100 sample runs using random velocity profiles and the results are summarized in Figure 7. The three exemplary scenarios mentioned above form part of this validation dataset. In Figure 7a, it can be seen that the standard deviation of the horizontal position error of these 100 profiles is 2.5 mm over a travel length of 0.52 m. This is better than 1 / 8 of a wavelength at K-band (24 GHz) along the track and is therefore sufficient to achieve excellent image quality below this frequency band. The associated phase error is shown in Figure 7b.

[0060] (Applicable system) The techniques disclosed herein may be embodied as special purpose hardware (e.g., circuitry), as programmable circuitry suitably programmed with software and / or firmware, or as a combination of special purpose and programmable circuitry. Thus, embodiments may include a machine-readable medium having stored thereon instructions that may be used to cause a computer, microprocessor, processor, and / or microcontroller (or other electronic device) to perform a process. Machine-readable media may include, but are not limited to, optical disks, compact disk read-only memory (CD-ROM), magneto-optical disks, ROM, random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing electronic instructions.

[0061] Some implementations are described in more detail below with reference to the drawings. FIG. 8 is a block diagram showing an overview of a device in which some implementations of the disclosed technology can operate. The device can include hardware components, such as a device 800 that captures mmWave images. The device 800 can include one or more input devices 820 that provide input and signal actions to a CPU (processor) 810. Actions are typically mediated by a hardware controller that interprets signals received from the input devices and communicates information to the CPU 810 using a communication protocol. The input devices 820 include, for example, a mouse, a keyboard, a touch screen, an mmWave imaging sensor (e.g., a K-band mmWave imaging sensor operating at or near 24 GHz), an infrared sensor, a touch pad, a wearable input device, a camera or image-based input device, a microphone, or other user input device.

[0062] The CPU 810 may be a single processing unit or multiple processing units in a device, or may be distributed across multiple devices. The CPU 810 may be coupled to other hardware devices using a bus, such as a PCI bus or a SCSI bus. The CPU 810 may communicate with a hardware controller for a device, such as a display 830. The display 830 may be used to display text and graphics. In some examples, the display 830 provides visual feedback of graphics and text to a user. In some implementations, the display 830 includes an input device as part of the display, such as when the input device is a touch screen or when equipped with an eye direction monitoring system. In some implementations, the display is separate from the input device. Examples of display devices are LCD display screens, LED display screens, projection, holographic, or augmented reality displays (such as head-up display devices or head-mounted devices), etc. Other I / O devices 840, such as network cards, video cards, audio cards, USB, FireWire or other external devices, sensors, cameras, printers, speakers, CD-ROM drives, DVD drives, disk drives, or Blu-ray devices, may also be coupled to the processor. In some examples, the I / O devices 840 may also include control of the speed of a robotic material handling system or a conveyor belt.

[0063] In some implementations, the device 800 also includes a communication device capable of wireless or wired-based communication with a network node. The communication device can communicate with another device or server over a network, for example, using the TCP / IP protocol. The device 800 can utilize the communication device to distribute operations across multiple network devices.

[0064] The CPU 810 may have access to memory 850. Memory may include one or more of a variety of hardware devices for volatile and non-volatile storage, and may include both read-only and writeable memory. For example, memory may include random access memory (RAM), CPU registers, read-only memory (ROM), and writeable non-volatile memory, such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, device buffers, and the like. Memory is not a propagating signal separate from the underlying hardware, and thus memory is non-transient. Memory 850 may include program memory 860 that stores programs and software, such as an operating system 862, motion estimation from a video platform 864, and other application programs 866. Memory 850 may also include data memory 870, which may include database information, and the like, that may be provided to the program memory 860 or any element of the device 800.

[0065] Some implementations may be operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the technology include, but are not limited to, personal computers, server computers, handheld or laptop devices, cellular telephones, mobile phones, wearable electronics, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0066] 9 is a block diagram illustrating an overview of an environment 900 in which some embodiments of the disclosed technology can operate. The environment 900 can include one or more client computing devices 905A-D, examples of which can include device 800. The client computing devices 905 can operate in a networked environment using logical connections via a network 930 to one or more remote computers, such as a server computing device 910.

[0067] In some implementations, server computing device 910 may be an edge server that receives client requests and coordinates fulfillment of those requests via other servers, such as servers 920A-C. Server computing devices 910 and 920 may include computing systems, such as device 1000. Although each server computing device 910 and 920 is logically viewed as a single server, each server computing device may be a distributed computing environment encompassing multiple computing devices located at the same or different geographic physical locations. In some implementations, each server computing device 920 corresponds to a group of servers.

[0068] Each of the client computing device 905 and the server computing devices 910 and 920 can function as a server or a client to other server / client devices. The server 910 can be connected to a database 915. The servers 920A-C can be connected to corresponding databases 925A-C, respectively. As mentioned above, each server 920 can correspond to a group of servers, each of which can share a database or have its own database. The databases 915 and 925 can store (e.g., store) information. Although the databases 915 and 925 are logically viewed as a single unit, each of the databases 915 and 925 can be a distributed computing environment encompassing multiple computing devices, can be located within their corresponding servers, or can be located in the same or different geographical physical locations.

[0069] The network 930 may be a local area network (LAN) or a wide area network (WAN), but may also be other wired or wireless networks. The network 930 may be the Internet or some other public or private network. The client computing devices 905 may connect to the network 930 via a network interface, such as a wired or wireless communication. Although the connections between the servers 910 and 920 are shown as separate connections, these connections may be any local network, wide area network, wired network, or wireless network, including the types of networks 930 or separate public or private networks.

[0070] 10 is a block diagram illustrating components that may be used in a system employing the disclosed technology in some implementations. The components 1000 include hardware 1002, general software 1020, and dedicated components 1040. As mentioned above, a system implementing the disclosed technology may use a variety of hardware including a processing unit 1004 (e.g., CPU, GPU, APU, etc.), working memory 1006, storage memory 1008, and input / output devices 1010. The components 1000 may be implemented in a client computing device, such as client computing device 1105, or in a server computing device, such as server computing device 910.

[0071] The general software 1020 can include various applications including an operating system 1022, local programs 1024, and a basic input / output system (BIOS) 1026. The dedicated components 1040 can be subcomponents of the general software applications 1020, such as the local programs 1024. The dedicated components 1040 can include an imaging module 1044, a velocity extraction module 1046, a velocity profile module 1048, an image reconstruction module 1050, and components that can be used to transmit data and control the dedicated components, such as the interface 1042. In some implementations, the components 1000 can be in a computing system distributed across multiple computing devices or can be an interface to a server-based application that executes one or more of the dedicated components 1040.

[0072] Those skilled in the art will appreciate that the components illustrated in each of the above-described Figures 6-8 and the above-described flow diagrams may be modified in various ways. For example, the order of logic may be rearranged, substeps may be performed in parallel, illustrated logic may be omitted, other logic may be included, etc. In some implementations, one or more of the components described above may perform one or more operations described herein.

[0073] (Conclusion) The results presented here show that image quality can depend on knowledge of the precise motion of the object being imaged across the field of view of the mmWave imaging system. Quantitative measures of image focus, such as total power and histogram range, can show improvements when the motion profile of the imaged object is taken into account, even when the object's velocity is close to constant. The degree of improvement can be even greater when large variations in velocity are present.

[0074] In real-world commercial and industrial applications, the objects being imaged may move at inconsistent velocities for a variety of reasons, and the motion may be accounted for by the radar imaging system to generate the image. The use of motion vectors extracted from an optical camera placed adjacent to the imaging array may yield beneficial results, allowing for varying velocity profiles to be compensated for in real-time. The combined motion-estimation and radar imaging system may be utilized for high-throughput scanning applications for scenes with non-constant velocities.

[0075] From the foregoing, it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without deviating from the scope of the invention. Accordingly, the invention is not limited except as by the appended claims.

Claims

1. 1. A system for generating a millimeter wave image, comprising: a video camera operative to capture video frames of a moving object; a millimeter wave antenna array proximate to said video camera and operative to capture millimeter wave data frames; a motion vector processor configured to calculate one or more motion vectors using the video frames, the one or more motion vectors representing a relative motion between the video camera and the moving object; at least one of a position processor and a velocity processor, the position processor configured to use the one or more motion vectors to calculate one or more position estimates of the moving object relative to the mmWave antenna array, and the velocity processor configured to use the one or more motion vectors to calculate one or more velocity estimates of the moving object relative to the mmWave antenna array; an image reconstruction processor configured to use the mmWave data frames and at least one of the corresponding position and velocity estimates to compute a mmWave image of the moving object; A system comprising:

2. The system of claim 1 , wherein the video camera produces a compressed video data set.

3. The system of claim 2 , wherein the one or more motion vectors are extracted from the compressed video data set.

4. The system of claim 2 , wherein the compressed video data set comprises an MPEG video data set.

5. The system of claim 1 , wherein the mmWave image comprises a three-dimensional image of the moving object.

6. The system of claim 1 , wherein the position and velocity estimates are interpolated between the video frames.

7. The system of claim 1 , wherein the moving object is transported via one of a conveyor belt or a robot.

8. The system of claim 1 , wherein the moving object is stationary and the video camera and the millimeter wave antenna array are carried by a moving platform.

9. The system of claim 8 , wherein the mobile platform comprises a mobile robot.

10. The system of claim 1 , wherein the relative motion between the video camera and the moving object includes a non-constant velocity.

11. 1. A method for generating a millimeter wave image of a moving object, the method comprising: capturing two or more video frames of the moving object using a video camera proximate to a mmWave antenna array; capturing two or more mmWave data frames using the mmWave antenna array; calculating one or more motion vectors representing relative motion between the video camera and the moving object; using the motion vectors to calculate at least one of one or more position estimates and one or more velocity estimates of the moving object relative to the mmWave antenna array; calculating the mmWave image using the mmWave frames of data and at least one of the corresponding position and velocity estimates; The method includes:

12. The method of claim 11 , wherein the video frames comprise a compressed video data set.

13. The method of claim 12 , wherein the one or more motion vectors are extracted from the compressed video data set.

14. The method of claim 12 , wherein the compressed video data set comprises an MPEG video data set.

15. The method of claim 11 , wherein the mmWave image comprises a three-dimensional image of the moving object.

16. The method of claim 11 , wherein the position and / or velocity estimates are interpolated between the video frames.

17. The method of claim 11 , wherein the moving object has a non-constant velocity.

Citation Information

Patent Citations

  • Millimeter wave image system

    JP1998153655A

  • Vehicle control method and vehicle warning method

    US20020032515A1

  • Motion stabilization

    US20060066728A1

  • Millimeter wave imager with visible or infrared overlay for brownout assist

    US20090002220A1