System and processing device

Abstract

To provide a system and a processing device, which can accurately detect an article.SOLUTION: A system includes: a radar that irradiates a target with electromagnetic waves and receives reflected electromagnetic waves from the target; an imaging section that images the target and outputs a target image; a measurement section that measures a position of the radar; a processing section that obtains a first area, which is not irradiated with the electromagnetic waves, in the target on the position of the radar, and generates a first image representing the first area; and a display section that displays the target image and the first image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION Embodiments of the present invention relate to a system and a processing device. [Background technology]

[0002] In the security field, systems for inspecting dangerous objects are used. The system irradiates a person with radio waves and receives radio waves reflected from dangerous objects hidden in the person's bag or pockets, creating an image of the dangerous object. The process of irradiating radio waves and receiving reflected radio waves is called scanning. The system is equipped with a portable or handheld scanner. An officer points the scanner at the person and transmits and receives radio waves to perform a security check.

[0003] To make the scanner easier to handle during inspections, it is necessary to make it smaller and lighter. However, when the scanner is made smaller, the antenna aperture length becomes smaller, which reduces the resolution of the images of dangerous objects.

[0004] Although it is possible to increase the antenna aperture length by scanning an area according to the scanner's field of view, then shifting the scanner to scan at least one adjacent area and generating an image based on the reflected radio waves from multiple areas, it is not possible for an agent to shift the scanner so that multiple non-overlapping, gap-free areas are scanned. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Chinese Patent Application Publication No. 202010153540 [Non-patent literature]

[0006] [Non-Patent Document 1] D. Nister, O. Naroditsky and J. Bergen, "Visual odometry," Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Washington, DC, USA, 2004, pp. II, doi: 10.1109 / CVPR.2004.1315094. [Non-patent document 2] ME Yanik and M. Torlak, "Near-Field MIMO-SAR Millimeter-Wave Imaging With Sparsely Sampled Aperture Data," in IEEE Access, vol. 7, pp. 31801-31819, 2019. [Non-patent document 3] Z. Yang, YR Zheng, “Near-Filed 3-D Synthetic Aperture Radar Imaging via Compressed Sensing,” ICASSP, pp. 2513-2516, 2012. [Non-patent document 4] N. Parikh and S. Boyd, “Proximal algorithms,” Foundations and Trends in Optimization, vol. 1, no. 3, pp. 123‐231, 2013 [Non-patent document 5] M. Wang et al., "RMIST-Net: Joint Range Migration and Sparse Reconstruction Network for 3-D mmW Imaging," in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-17, 2022. Summary of the Invention [Problem to be solved by the invention]

[0007] SUMMARY OF THE INVENTION An object of the present invention is to provide a system and processing device that can detect an article with high accuracy. [Means for solving the problem]

[0008] A system according to an embodiment includes a radar, an imaging unit, a measurement unit, and a processing unit. The radar irradiates a target with radio waves and receives radio waves reflected from the target. The imaging unit captures an image of the target and outputs target image information. The measurement unit measures the position of the radar. The processing unit outputs first image information representing a first region in the target that is not irradiated with radio waves based on the position of the radar. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an example of a system according to a first embodiment. [Figure 2] FIG. 2 is a diagram for explaining an example of the appearance of the scanner according to the first embodiment. [Figure 3] FIG. 2 is a diagram for explaining an example of a transmitting array antenna and a receiving array antenna according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining an example of a transmission circuit and a reception circuit according to the first embodiment. [Figure 5] FIG. 2 is a diagram for explaining an example of a chirp signal according to the first embodiment. [Figure 6] FIG. 3 is a diagram for explaining an example of the operation of the radar unit according to the first embodiment. [Figure 7] 4 is a flowchart for explaining an example of the operation of the system according to the first embodiment. [Figure 8] 4 is a flowchart for explaining an example of the operation of the system according to the first embodiment. [Figure 9] 4 is a flowchart for explaining an example of the operation of the system according to the first embodiment. [Figure 10]FIG. 2 is a diagram for explaining an example of primary scanning by a scanner according to the first embodiment. [Figure 11] FIG. 2 is a diagram for explaining an example of a first image according to the first embodiment. [Figure 12] FIG. 3 is a diagram for explaining an example of designating an inspection area according to the first embodiment. [Figure 13] FIG. 2 is a diagram for explaining an example of a virtual array antenna according to the first embodiment. [Figure 14] 10A and 10B are diagrams for explaining an example of superimposed display of the subject image and the first image by the display unit according to the first embodiment when the determination result is safety. [Figure 15] 10A and 10B are diagrams for explaining an example of a display by the display unit according to the first embodiment that prompts inspection in the depth direction when the determination result indicates safety. [Figure 16] FIG. 10 is a diagram for explaining an example of a display of a determination result by the display unit according to the first embodiment when the determination result indicates danger. [Figure 17] FIG. 10 is a diagram for explaining an example of a display on the display unit according to the first embodiment when the determination result is unknown. [Figure 18] FIG. 10 is a diagram for explaining an example of primary scanning by a scanner according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following describes embodiments with reference to the drawings. The following description exemplifies devices and methods embodying the technical concepts of the embodiments. The technical concepts of the embodiments are not limited to the structures, shapes, arrangements, materials, etc. of the components described below. Modifications that can be easily conceived by those skilled in the art are naturally within the scope of the disclosure. For clarity of explanation, the drawings may schematically depict elements with different sizes, thicknesses, planar dimensions, shapes, etc. compared to the actual elements. Elements with different dimensional relationships or ratios may be included in multiple drawings. Corresponding elements may be designated by the same reference numerals in multiple drawings, and redundant description may be omitted. Some elements may be designated by multiple names, but these names are merely examples and do not exclude the use of other names for these elements. Elements without a plural name may also be designated by other names. "Connection" may include not only direct connection but also connection via other elements. Unless the number of elements is specified as being plural, the element may be a singular element or multiple elements.

[0011] First embodiment FIG. 1 is a diagram illustrating an example of a system according to a first embodiment. The system is used in airports, baseball stadiums, concert halls, etc. for anti-terrorism purposes. The system determines whether a target person is carrying a specific item. The specific item includes dangerous items such as handguns, knives, and explosives, as well as illegal items such as narcotics that are not permitted to be possessed.

[0012] The system comprises a radar unit 10, an imaging unit 12, a measurement unit 14, a processing unit 16, and a display unit 18. At least the radar unit 10 is implemented as a portable or handheld scanner. An attendant asks the subject to stop, points the scanner at the subject, transmits radio waves to the subject, and receives the radio waves reflected from the subject.

[0013] 2 is a diagram illustrating an example of the appearance of the scanner 30 according to the first embodiment. The scanner 30 includes a main body 32 and a grip 36. The main body 32 houses at least the radar unit 10. The main body 32 may house the measurement unit 14 and the processing unit 16 in addition to the radar unit 10. A radio wave transmitting and receiving surface 34 made of a material that transmits radio waves is disposed on one side (xy plane) of the main body 32. A display unit 18 is disposed on the top surface of the main body 32 or on the side opposite to the radio wave transmitting and receiving surface 34.

[0014] An example of the display unit 18 is a flat display unit such as a liquid crystal panel. The display unit 18 may be a touch panel with an input function. If the display unit 18 does not have an input function, an input device such as a keyboard or a mouse is connected to the display unit 18. The display unit 18 is not limited to a display unit disposed on the main body 32, but may also be a display unit that is separate from the main body 32 and connected to the processing unit 16 by cable or wirelessly. Furthermore, the display unit 18 is not limited to a flat display unit, but may also be a face-mounted display unit that is connected to the processing unit 16 by cable or wirelessly. Examples of face-mounted display units include eyeglass-type display units and goggle-type display units.

[0015] The display unit 18 displays an image of the subject and an image useful for positioning the scanner 30. By viewing these displayed images and holding the scanner 30 over the subject, the staff member can scan multiple areas without overlapping or gaps.

[0016] The grip 36 is attached to the bottom of the main body 32. The staff member holds the grip 36 and points the radio wave transmitting and receiving surface 34 of the scanner 30 toward the subject. The grip 36 may be equipped with a scan start button. When the staff member presses the scan start button, the radar unit 10 begins transmitting radio waves and receiving reflected radio waves, and the scanner 30 begins scanning. This allows the staff member to start scanning at any time. The radar unit 10 transmits and receives radio waves in an area according to the field of view and obtains the received signals for each area. The processing unit 16 generates image information of the items within the area based on the received signals. After finishing scanning one area, the staff member moves the scanner 30 and presses the scan start button to scan an adjacent area. If a scan start button is not equipped, the radar unit 10 transmits radio waves at any time interval and performs scanning. Since a portable or handheld scanner 30 is used, inspections can be easily performed.

[0017] 1, the radar unit 10 includes a transmitting array antenna 42, a transmitting circuit 44, a receiving array antenna 46, and a receiving circuit 48. The transmitting circuit 44 and the receiving circuit 48 may each be configured as an integrated circuit.

[0018] FIG. 3 is a diagram for explaining an example of the transmitting array antenna 42 and the receiving array antenna 46 according to the first embodiment.

[0019] The transmitting array antenna 42 includes a first transmitting array antenna 42A located at the left end of a square region and a second transmitting array antenna 42B located at the right end of the square region. The first transmitting array antenna 42A includes multiple transmitting antennas Tx arranged along the y-axis. The second transmitting array antenna 42B includes multiple transmitting antennas Tx arranged along the y-axis.

[0020] The receiving array antenna 46 includes a first receiving array antenna 46A located at the top of the square region and a second receiving array antenna 46B located at the bottom of the square region. The first receiving array antenna 46A includes multiple receiving antennas Rx arranged in the x-axis direction. The second receiving array antenna 46B includes multiple receiving antennas Rx arranged in the x-axis direction.

[0021] The array antenna shown in FIG. 3, which is configured with multiple transmitting antennas Tx and multiple receiving antennas Rx, is called a Multiple-input Multiple-output (MIMO) array antenna.

[0022] The transmitting antennas Tx are arranged at equal intervals, for example, at wavelength λ intervals. The transmitting array antenna 42 is an equally spaced array antenna. The receiving antennas Rx are arranged at equal intervals, for example, at wavelength λ intervals. The receiving array antenna 46 is an equally spaced array antenna. The intervals between the transmitting antennas Tx and the receiving antennas Rx may be different.

[0023] The distance in the y-axis direction between the uppermost transmitting antenna Tx of each of the transmitting array antennas 42A and 42B and the first receiving array antenna 46A is half (λ / 2) of the spacing between the transmitting antennas Tx. The distance in the y-axis direction between the lowermost transmitting antenna Tx of each of the transmitting array antennas 42A and 42B and the second receiving array antenna 46B is half (λ / 2) of the spacing between the transmitting antennas Tx.

[0024] The distance in the y-axis direction between the leftmost receiving antenna Rx of each receiving array antenna 46A, 46B and the first transmitting array antenna 42A is half (λ / 2) of the spacing between the receiving antennas Rx. The distance in the y-axis direction between the rightmost receiving antenna Rx of each receiving array antenna 46A, 46B and the second transmitting array antenna 42B is half (λ / 2) of the spacing between the receiving antennas Rx.

[0025] The equally spaced array antenna can receive waves reflected in multiple directions from objects within the irradiation area, with few missed signals.

[0026] Each of the transmitting array antenna 42 and the receiving array antenna 46 may be an unequally spaced array antenna consisting of multiple antennas spaced at unequally spaced intervals. Furthermore, each of the transmitting array antenna 42 and the receiving array antenna 46 may be a Minimum Redundancy Array (MRA) antenna. In an MRA antenna, antennas are spaced at unequally spaced intervals, and the unequally spaced intervals include several intervals. The unequally spaced intervals include, for example, intervals of m1 times the wavelength and intervals of m2 times the wavelength, where m1 and m2 are coprime. Using an MRA antenna can reduce the number of antennas constituting the array antenna, efficiently widening the aperture length and enabling higher resolution in image generation using synthetic aperture processing.

[0027] 1, the transmission circuit 44 is connected to the transmission array antenna 42 (first transmission array antenna 42A and second transmission array antenna 42B). The reception circuit 48 is connected to the reception array antenna 46 (first reception array antenna 46A and second reception array antenna 46B).

[0028] Although not shown in FIG. 1 , the transmitter circuit 44 may include a first transmitter circuit and a second transmitter circuit, with the first transmitter circuit connected to the first transmit array antenna 42A and the second transmitter circuit connected to the second transmit array antenna 42B. The receiver circuit 48 may include a first receiver circuit and a second receiver circuit, with the first receiver circuit connected to the first receive array antenna 46A and the second receiver circuit connected to the second receive array antenna 46B. Each of the first transmitter circuit and the second transmitter circuit may include multiple transmitter circuits, with each transmitter circuit connected to multiple transmit antennas in the transmit array antenna. Each of the first receiver circuit and the second receiver circuit may include multiple receiver circuits, with each receiver circuit connected to multiple receive antennas in the receive array antenna. The number of transmitter array antennas 42 and transmitter circuits 44 and the number of receiver array antennas 46 and receiver circuits 48 may be set arbitrarily.

[0029] 4 is a diagram illustrating an example of a transmission circuit 44 and a reception circuit 48 according to the first embodiment. The transmission circuit 44 includes a signal generator 62. The radar unit 10 employs a linear frequency modulated continuous wave (L-FMCW) system in which the frequency increases linearly over time. The signal generator 62 generates an L-FMCW signal (also referred to as a chirp signal). The signal generator 62 generates the chirp signal using a reference signal, an RF synthesizer, and a frequency multiplier.

[0030] The chirp signal output from the signal generator 62 is supplied to multiple transmitting antennas Tx via multiple phase shifters 64 and multiple transmitting amplifiers 66. The chirp signal is also supplied to the receiving circuit 48. The phase shifters 64 adjust the phase of the transmitting signal. The transmitting amplifiers 66 adjust the transmitting power.

[0031] The transmitting circuit 44 may transmit chirp signals sequentially from one transmitting antenna, or may transmit chirp signals simultaneously from multiple transmitting antennas.

[0032] The receiving circuit 48 includes a plurality of receiving amplifiers 72, a plurality of mixers 74, a plurality of low-pass filters (LPFs) 76, and a plurality of A / D converters (ADCs) 78. A plurality of received signals output from a plurality of receiving antennas Rx are supplied to first input terminals of the plurality of mixers 74 via the plurality of receiving amplifiers 72. A chirp signal is supplied to second input terminals of the plurality of mixers 74.

[0033] The plurality of mixers 74 respectively multiply the plurality of received signals by the chirp signal to generate a plurality of received intermediate frequency (IF) signals. The plurality of received IF signals respectively output from the plurality of mixers 74 are supplied to the image generator 52 via a plurality of LPFs 76 and a plurality of ADCs 78.

[0034] The radar unit 10 causes all the receiving antennas Rx to simultaneously receive the reflected waves of the radio waves transmitted from one transmitting antenna Tx, and repeats this process for all the transmitting antennas Tx to perform one scan.

[0035] The signal generator 62 transmits a start pulse to the measurement unit 14 at the time when the chirp signal is transmitted, that is, at the time when the radar unit 10 starts one scan.

[0036] 5A and 5B are diagrams illustrating an example of a chirp signal according to the first embodiment. FIG. 5A shows a chirp signal that expresses amplitude A as a function of time t. FIG. 5B shows a chirp signal that expresses frequency f as a function of time t. As shown in FIG. 5B, the chirp signal is represented by a center frequency fc, a modulation bandwidth fb, and a signal time width Tb. The slope of the chirp signal is called the frequency change rate (chirp rate) γ.

[0037] The transmitted radar signal St(t) of the chirp signal is expressed by Equation 1.

[0038] St(t)=cos[2π(fc×t+γt 2 / 2)] Formula 1 The chirp rate γ is expressed by Equation 2.

[0039] γ=fb / Tb Equation 2 The reflected wave from an object a distance R away from the radar unit 10 is observed with a delay of Δt = 2R / c from the transmission timing, where c is the speed of light. The received signal Sr(t) is expressed by Equation 3, where a is the reflection intensity from the object.

[0040] Sr(t)=a×cos[2πfc(t-Δt)+πγ(t-Δt) 2 ] Formula 3 FIG. 6 is a diagram illustrating an example of the operation of the radar unit 10 according to the first embodiment. FIG. 6 shows the principle of detecting objects when there are multiple objects, for example, three objects. FIG. 6(a) shows the relationship between the transmitted signal and time and the relationship between the received signal and time. As shown in FIG. 6(a), the frequency of the transmitted signal (chirp signal) changes linearly over time. The received signal is delayed by Δt relative to the transmitted signal. When there are multiple objects, the reflected wave from the nearest object, indicated by the dashed line, is received first, and the reflected wave from the farthest object, indicated by the dashed line, is received last.

[0041] 4, the received signal is multiplied by the chirp signal in a mixer 74 to generate a received IF signal z(t). The received IF signal z(t) is expressed by Equation 4.

[0042] z(t)=a×cos(2πΔtγt) Equation 4 Figure 6(b) shows the relationship between the frequency of the received IF signal and time. In an ideal environment without noise, the frequency is constant for each reflected wave. The frequency of the received IF signal for the reflected wave from the nearest object, shown by the dashed line, is the lowest, and the frequency of the received IF signal for the reflected wave from the farthest object, shown by the dashed line, is the highest.

[0043] The processing unit 16 performs a fast Fourier transform (FFT) on the received IF signal z(t) in the time domain, as shown in Equation 4, to calculate the reflection intensity in the frequency domain. Therefore, the amplitude at each point in the frequency domain, which is the result of the FFT of the received IF signal, corresponds to the reflection intensity at each distance from the radar unit 10. if and the distance R are related by Equation 5.

[0044] f if =Δtγ=2Rγ / c Equation 5 The relationship between the reflection intensity and frequency obtained by performing FFT on the received IF signal in the time domain is shown in Figure 6(c). In this way, by calculating the amplitude of the frequency domain signal of the received IF signal, the reflection intensity for each distance from the radar unit 10 can be calculated.

[0045] The radio waves used in the embodiments may have a wavelength of 1 to 30 millimeters. Radio waves with a wavelength of 1 to 10 millimeters are called millimeter waves. Radio waves with a wavelength of 10 to 100 millimeters are called microwaves. Furthermore, the radio waves used in the embodiments may have a wavelength of 100 micrometers to 1 millimeter. Radio waves with a wavelength of 100 micrometers to 1 millimeter are called terahertz waves.

[0046] These radio waves are reflected by the subject's skin. They are also reflected by metal objects such as guns and knives. The reflectivity of metal is higher than that of skin. The intensity of the waves reflected by metal is higher than that of skin. These radio waves are absorbed by powder such as explosives. The reflectivity of powder is lower than that of skin. The intensity of the reflected waves is determined by the object located at the reflection point of the radio waves, such as skin, metal, or powder. Therefore, the type of object located at the reflection point can be determined from the intensity of the reflected waves.

[0047] Returning to the explanation of FIG. 1, the imaging unit 12 includes an optical sensor such as a camera. The system defines an inspection area in which the subject is located during scanning. The field of view of the imaging unit 12 is larger than the field of view of the radar unit 10. The field of view of the imaging unit 12 corresponds to, for example, the entire body of the subject. The imaging unit 12 is installed so that its field of view includes the entire body of the subject located in the inspection area. When the subject enters the inspection area, the imaging unit 12 captures an image of the subject's entire body once. The subject does not need to remain still during imaging. The imaging unit 12 can also capture images of moving subjects. The imaging unit 12 supplies one image of the subject's image information to the measurement unit 14 and the processing unit 16.

[0048] The measurement unit 14 is connected to the radar unit 10, the imaging unit 12, and the processing unit 16. In response to a start pulse from the transmission circuit 44, the measurement unit 14 determines the position of the scanner 30 when the radar unit 10 emits radio waves. The measurement unit 14 determines the position of the scanner 30 (specifically, the radar unit 10) by a self-position estimation method based on the object image output by the imaging unit 12. The position is a relative position with respect to the subject. The relative position is represented by coordinates in the coordinate system of the subject image information. The measurement unit 14 determines the position of the scanner 30 in real time, in addition to the position of the scanner 30 when radio wave emission begins.

[0049] The measurement unit 14 may determine the position using the self-position estimation method described in Non-Patent Document 1. The measurement unit 14 may be equipped with a gyro sensor and determine the position of the scanner 30 based on the output of the gyro sensor. The measurement unit 14 supplies the position of the scanner 30 to the processing unit 16.

[0050] The processing unit 16 is connected to the radar unit 10, the imaging unit 12, the measurement unit 14, and the display unit 18. The processing unit 16 includes an image generation unit 52, an unirradiated area detection unit 54, and a determination unit 56. The output signal of the imaging unit 12, the output signal of the receiving circuit 48, and the output signal of the measurement unit 14 are supplied to the image generation unit 52. The output signal of the imaging unit 12 and the output signal of the measurement unit 14 are supplied to the unirradiated area detection unit 54.

[0051] The non-irradiated area detection unit 54 detects a first area (referred to as a non-irradiated area) within the target that is not irradiated with radio waves and a second area (referred to as an irradiated area) within the target that is irradiated with radio waves, based on the position of the radar unit 10 when the radar unit 10 irradiated the radio waves and the viewing angle of the radar unit 10. For the part of the target that is irradiated with radio waves, the non-irradiated area detection unit 54 may determine, as the irradiated area, an area defined by the viewing angle of the radar unit 10, which is an area that falls within the viewing angles of all transmitting antennas and all receiving antennas. Alternatively, the non-irradiated area detection unit 54 may determine, as the irradiated area, an area that falls within the viewing angles of some transmitting antennas and some receiving antennas.

[0052] The non-irradiated area detection unit 54 determines the non-irradiated area by subtracting the irradiated area from the entire area of ​​the subject. The non-irradiated area detection unit 54 may detect an irradiated area where the level of the received signal is below a specified level as an unirradiated area rather than as an irradiated area. Therefore, the output signal of the receiving circuit 48 is also supplied to the non-irradiated area detection unit 54.

[0053] The non-irradiated region detection unit 54 supplies the coordinates of the contours of the irradiated region and the coordinates of the contours of the non-irradiated region to the image generation unit 52. The coordinates of the irradiated region and the non-irradiated region are coordinates in the coordinate system of the subject image information.

[0054] The output signal of the unirradiated area detection unit 54 is supplied to the image generation unit 52. The image generation unit 52 generates first image information representing the irradiated area and the unirradiated area based on the image of the irradiated area and the coordinates of the unirradiated area. The first image information may be image information representing only the unirradiated area. Here, the first image information is assumed to be image information representing the irradiated area and the unirradiated area.

[0055] The image generation unit 52 supplies the subject image information and the first image information to the display unit 18. The display unit 18 displays the first image based on the first image information superimposed on the subject image based on the subject image information. The first image includes an irradiated area image and an unirradiated area image. The display unit 18 may display the irradiated area image and the unirradiated area image as a rectangle (which may be filled in or hatched) indicating the entire area, or as a frame indicating the outline of the area. The display unit 18 may display the unirradiated area image in a hue, brightness, or saturation different from that of the irradiated area image. The display unit 18 may continuously display one of the irradiated area image and the unirradiated area image, and flash the other. This allows the staff member to easily recognize the irradiated area and the unirradiated area of ​​the subject.

[0056] The image generation unit 52 also generates second image information representing the area to be scanned by the scanner 30, i.e., the area to be irradiated with radio waves by the radar unit 10 (hereinafter referred to as the "intended irradiation area"), based on the real-time position of the scanner 30 output from the measurement unit 14. The image generation unit 52 also supplies the second image information to the display unit 18. The display unit 18 displays the second image based on the second image information. The display unit 18 may display the second image as a rectangle (which may be solid or hatched) representing the entire intended irradiation area, or as a frame indicating the outline of the intended irradiation area. The display unit 18 may also display the second image as a frame including a perspective view of the scanner 30. The display unit 18 may display the second image with a hue, brightness, or saturation different from that of the first image (the irradiated area image and / or the unirradiated area image). The display unit 18 may continuously display one of the first image and the second image, and flash the other.

[0057] The display unit 18 may display the first and second images superimposed on the subject image, allowing the staff member to position the second image on the image of the unexposed area so that the scanner 30 scans the unexposed area, thereby making it easier to position the scanner 30.

[0058] The image generation unit 52 also generates article image information of a specific article concealed by the subject based on the output signal of the receiving circuit 48. The image generation unit 52 supplies the article image information to the display unit 18. The display unit 18 displays an article image based on the article image information superimposed on the subject image. This allows an attendant to determine whether the subject is carrying a specific article based on the image.

[0059] The image generating unit 52 includes a buffer memory that stores the target person image information, the first image information, the second image information, and the item image information.

[0060] The determination unit 56 determines whether the subject is in possession of a specific item based on the item image information. The image generation unit 52 supplies the subject image information, item image information, and the determination result to the display unit 18. The display unit 18 displays the determination result in addition to the subject image and item image. An example of the display of the determination result is text representing the specific item. This allows the staff member to easily determine whether the subject is in possession of the specific item. The output form of the determination result is not limited to a display, but may also be the generation of an alarm sound, vibration of the scanner 30, etc.

[0061] The image generation unit 52, the unexposed region detection unit 54, and the determination unit 56 constituting the processing unit 16 may each be realized by multiple hardware blocks, or may be realized in software by one or more CPUs. For example, the processing unit 16 may be realized by one or more processing circuits such as a CPU, a microprocessor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or an electronic circuit including these circuits. The processing unit 16 may be realized by an information processing device such as a computer, a computer system in which multiple computers or servers communicate with each other via a network, or a PC cluster in which multiple computers cooperate to perform information processing. Instead of having a single CPU, the processing unit 16 may have multiple CPUs, each of which realizes at least some of the functions.

[0062] Fig. 7 is the first half of a flowchart for explaining an example of the operation of the system according to the first embodiment. Fig. 8 is the middle half of a flowchart for explaining an example of the operation of the system according to the first embodiment. Fig. 9 is the second half of a flowchart for explaining an example of the operation of the system according to the first embodiment.

[0063] The staff member guides the subject to the examination area (S102). The imaging unit 12 captures an image of the subject's entire body in the examination area (S104). The image generation unit 52 generates subject image information (S106).

[0064] The staff member aims the scanner 30 at the subject and moves the scanner 30 while transmitting and receiving radio waves, scanning multiple areas continuously (S108). The staff member can move the scanner 30 as desired. In the first embodiment, the subject may be scanned twice. The scanning in S108 is the first scanning, and is called the primary scanning. The receiving circuit 48 supplies the received signal to the image generating unit 52.

[0065] FIG. 10 is a diagram illustrating an example of the primary scanning according to the first embodiment, performed in S108. FIG. 10 shows the trajectory of the scanner 30 when the attendant scans almost the entire body of the subject 8 with the scanner 30. During scanning, the radar unit 10 irradiates an irradiation area according to the field of view angle and receives reflected radio waves from the irradiation area. The field of view angle depends on the number of antennas constituting the transmitting array antenna and the number of antennas constituting the receiving array antenna (aperture length). The image generation unit 52 creates an image of the object at the reflection point based on the received signal. The image resolution depends on the aperture length. To increase the aperture length of the radar unit 10, it is sufficient to create an image of the object based on reflected radio waves from multiple irradiation areas. The multiple irradiation areas must be arranged in a grid pattern. However, because the attendant's hand movements may meander or their movement speed may fluctuate, it is difficult to create multiple adjacent irradiation areas without gaps or overlaps.

[0066] The image generating section 52 generates first image information representing the irradiated and unirradiated regions based on the signal from the unirradiated region detecting section 54 (S110).

[0067] 11 is a diagram illustrating an example of the first image information according to the first embodiment generated in S110. As a result of the primary scan, a plurality of illumination areas 82 are set in the subject image 80. A gap may occur between a certain illumination area 82 and the surrounding illumination areas 82. Two adjacent illumination areas 82 may also partially overlap each other. The area other than the illumination areas 82 in the subject image 80 is an unilluminated area.

[0068] The image generating unit 52 supplies the subject image information generated in S106 to the display unit 18 (S112). The display unit 18 displays the subject image based on the subject image information (S114).

[0069] The staff member designates an inspection area in the image of the subject to be imaged to determine whether or not a specific item is present (S116).

[0070] 12 is a diagram for explaining an example of designating an inspection area according to the first embodiment in S116. When designating an inspection area, the image generation unit 52 generates a pointer 84 for designating the inspection target, and displays the pointer 84 superimposed on the subject image 80 on the display unit 18. An example of the pointer 84 is a star mark.

[0071] The staff member changes the position of the pointer 84 and sets it in a part where the subject is likely to hide a specific item, such as a pocket. An area of ​​a predetermined size centered on the pointer 84 is designated as the inspection area. The predetermined size may be smaller than the irradiation area.

[0072] The image generation unit 52 generates article image information of the inspection area based on the output signal of the receiving circuit 48. The image generation unit 52 can generate article image information from the received signal of each irradiation area, but can also generate article image information from the received signals of multiple adjacent irradiation areas to improve resolution.

[0073] The image generation unit 52 performs two types of imaging: low-resolution imaging and high-resolution imaging. First, the image generation unit 52 generates first article image information by synthetic aperture radar (SAR) imaging based on the received signals for each irradiation area (S118). The image generation unit 52 synthesizes multiple images based on the received signals for each irradiation area to generate the first article image information. The SAR imaging that generates the first article image information is also called low-resolution imaging.

[0074] The image generation unit 52 generates a virtual array antenna from the MIMO array antenna shown in FIG. 3, and generates first article image information based on the virtual array antenna.

[0075] 13 is a diagram illustrating an example of a virtual array antenna according to the first embodiment. A virtual antenna 88 is generated by a u-th transmitting antenna Txu and a v-th receiving antenna Rxv. The virtual antenna 88 is generated on a line connecting the transmitting antenna Txu and the receiving antenna Rxv. The distance in the x-axis direction between the transmitting antenna Txu and the receiving antenna Rxv is dxv. The distance in the y-axis direction between the transmitting antenna Txu and the receiving antenna Rxv is dyu. The path length between the target 86 and the transmitting antenna Txu is Ru. The path length between the target 86 and the receiving antenna Rxv is Rv. The path length between the target 86 and the virtual antenna 88 is R.

[0076] The multiple virtual antennas 88 generated by combining all the transmitting antennas Tx and all the receiving antennas Rx are arranged at equal intervals in both the x-axis and y-axis directions to form an equally spaced virtual array antenna, or the multiple virtual antennas 88 are arranged at uneven intervals in both the x-axis and y-axis directions to form an unequally spaced virtual array antenna.

[0077] The image generation unit 52 performs imaging on the virtual array antenna shown in Fig. 13 using a Range Migration Algorithm (RMA). An example of RMA imaging is described in Non-Patent Document 2. The antenna arrangement of the array antenna, the method of generating the virtual array antenna, and the imaging example are not limited to the example described here.

[0078] First, RMA imaging using a single (mono static) transmit / receive antenna that can be used for both transmission and reception will be explained, and then an example of expanding the single transmit / receive antenna into a MIMO antenna will be explained.

[0079] The modulation method of the radar unit 10 is the FMWC method, and the transmitted radar signal St is expressed by Equation 6.

[0080]

number

[0081] where fc is the center frequency and γ is the chirp rate. At this time, the received signal at the receiving antenna Rx is expressed by Equation 7.

[0082]

number

[0083] τ is the arrival time of the reflected wave from the target 86 to the receiving antenna Rx (δt in Equation 3). σ is the reflection coefficient (a in Equation 3). The received IF signal is expressed by Equation 8.

[0084]

number

[0085] The third term in the exponent of Equation 8 is called the residual video phase and is known to be negligible. If the time width of the FMWC pulse of the chirp signal is T, Equation 8 can be rewritten as Equation 9, where k is the wave number.

[0086]

number

[0087] When a radio wave reflected from a target 86 located at (x, y, z0) is received by a receiving antenna Rx located at (x', y', 0), the received wave s is expressed as in Equation 10.

[0088]

number

[0089] When Equation 10 is expanded into a plane wave with respect to exp(j2kR), Equation 11 is obtained.

[0090]

number

[0091] Σ is the two-dimensional Fourier transform of the reflection coefficient σ. Since the double integral is the inverse Fourier transform, Equation 11 can be rewritten as Equation 12.

[0092]

number

[0093] From equation 12, the reflection coefficient σ can be calculated using equation 13. Once the reflection coefficient σ is calculated, the target 86 is imaged.

[0094]

number

[0095] Next, RMA imaging using a MIMO array antenna will be explained. Here, imaging is performed using a virtual array antenna including a virtual antenna 88 generated from a transmitting antenna Tx and a receiving antenna Rx. The path length R from the transmitting antenna Tx to the receiving antenna Rx via the target 86 is u,v is shown in Equation 14.

[0096]

number

[0097] Taylor expansion of the path length gives Equation 15.

[0098]

number

[0099] The second term of Equation 15 is the path length R u,v is the difference ΔR between the propagation path length difference ΔR and the path length R due to the virtual array. By correcting for the propagation path length difference ΔR as in Equation 16, it can be used as the received signal from a single transmitting and receiving antenna. The division by ΔR in the exp term of Equation 16 corresponds to the correction. In a MIMO array antenna, the transmitting and receiving antennas are separate, but this correction makes it possible to approximate the received signal with the path length of a single transmitting and receiving antenna.

[0100]

number

[0101] By applying Equation 16 to RMA processing using a single transmitting and receiving antenna, imaging of a MIMO array antenna can be performed.

[0102] The resolutions δx and δy in the x-axis and y-axis directions of the first article image obtained by this imaging are given by Equations 17 and 18. Dx and Dy are the aperture lengths in the x-axis and y-axis directions of the MIMO array antenna consisting of the transmitting array antenna 42 and the receiving array antenna 46.

[0103]

number

[0104]

number

[0105] Equations 17 and 18 indicate that increasing the aperture lengths Dx and Dy will achieve higher resolution. It is difficult for an attendant to manually move the scanner 30 with precision on the order of millimeters, which is the wavelength of radio waves. In the first embodiment, as described below, during secondary scanning, the area requiring irradiation is displayed superimposed on the target image, allowing the attendant to correctly position the scanner 30 and scan the desired area.

[0106] Returning to the explanation of the flowchart, the image generation unit 52 performs high-resolution imaging based on the received signals of the set of multiple irradiation areas set in the primary scan. First, it is determined whether the aperture lengths of the multiple irradiation areas related to high-resolution imaging are equal to or greater than a threshold (S132). The threshold is determined based on the resolution required for the image of the item to be inspected.

[0107] If the aperture lengths of the multiple irradiation areas are less than the threshold (No in S132), the image generation unit 52 determines the irradiation areas (referred to as irradiation-required areas) required to achieve the aperture length corresponding to the required resolution, and generates third image information representing the irradiation-required areas (S134). The irradiation-required areas are selected from the unirradiated areas. The third image information is stored in the buffer memory of the image generation unit 52.

[0108] If the aperture length is equal to or greater than the threshold (Yes in S132), the image generation unit 52 determines whether or not a half-wavelength equally spaced virtual array antenna was generated in the SAR imaging of S118 (S136). If a half-wavelength equally spaced virtual array antenna was generated (Yes in S136), the image generation unit 52 performs SAR imaging based on the received signals of the multiple irradiation areas to generate second article image information (S138). By performing SAR imaging using the received signals of the multiple irradiation areas, the aperture length becomes longer, thereby achieving higher resolution. The resolution of the second article image information is higher than the resolution of the first article image information based on the received signals of a single irradiation area.

[0109] If the half-wavelength interval virtual array antenna has not been generated (No in S136), the image generation unit 52 performs imaging using compressed sensing or machine learning to generate second article image information (S140). Examples of compressed sensing are described in Non-Patent Document 3 or Non-Patent Document 4. Examples of machine learning are described in Non-Patent Document 5.

[0110] The image generation unit 52 determines whether the resolution of the second item image based on the second item image information generated in S140 is equal to or greater than a threshold (S142). The resolution threshold in S142 is higher than the resolution corresponding to the opening length threshold in S132.

[0111] If the image quality of the second product image is less than the threshold value (No in S142), the image generation unit 52 determines the area that needs to be irradiated to achieve the opening length corresponding to the resolution corresponding to the threshold value, and generates third image information representing the area that needs to be irradiated (S144).

[0112] If the resolution of the second item image is equal to or greater than the threshold (Yes in S142) or after S134, S138, and S144 have been executed, the image generation unit 52 supplies the first item image information and the second item image information to the determination unit 56 and receives a determination result from the determination unit 56 (S146). The determination unit 56 determines the condition of the subject based on the first item image information and the second item image information.

[0113] The image generating unit 52 determines whether the determination result indicates safety, danger, or uncertainty (S148).

[0114] If the determination result indicates safety, the image generation unit 52 supplies the subject image information, the first image information, and the second image information to the display unit 18 (S152). The display unit 18 displays the subject image based on the subject image information by superimposing the first image based on the first image information and the second image based on the second image information (S154).

[0115] 14 is a diagram for explaining an example of superimposed display of the subject image, the first image, and the second image by the display unit 18 according to the first embodiment in S154. The display unit 18 displays the first image 102 and the second image 108 superimposed on the subject image 80.

[0116] The first image 102 includes squares arranged two-dimensionally. The size of the squares is the same as the size of the illuminated area corresponding to the viewing angle of the radar unit 10. The squares include an illuminated area (area hatched with solid lines) 104 and an unilluminated area (area hatched with dashed lines) 106.

[0117] The second image 108 includes a frame indicating the outline of the region to be irradiated and a perspective view of the scanner 30. The size of the outline of the region to be irradiated is the same as the size of the square included in the first image 102.

[0118] The staff member looks at the first image 102 (FIG. 14) displayed on the display unit 18 and recognizes that the lower half of the subject's body is an unexposed region, i.e., an unexamined region. The staff member inputs to the image generation unit 52 via an input device (not shown) whether or not to scan the unexposed region. Based on this input, the image generation unit 52 determines whether or not further scanning is necessary (S156).

[0119] If further inspection is required (Yes in S156), the image generation unit 52 displays text prompting further scanning superimposed on the image displayed on the display unit 18 (FIG. 14) (S158). The staff member easily positions the scanner 30 by aligning the frame (region to be irradiated) of the second image 108 with one of the unirradiated regions 106 while looking at the image displayed on the display unit 18. The staff member then performs a primary scan of the unirradiated region 106 (S108). This allows additional inspection of regions not inspected in the initial primary scan.

[0120] The areas described above (irradiated area, unirradiated area, inspection area) are areas related to the xy plane. Inspection of an object is an inspection of the presence or absence of an object on the xy plane. The presence or absence of an object on a plane intersecting the xy plane, such as the yz plane or xz plane, is not inspected. When a subject conceals a specific object, it may be placed between the armpits or between the legs. This object is detected because it has a certain area on the yz plane, but may not have a certain area as it is only recognized as a point or line on the xy plane.

[0121] The image generation unit 52 performs image recognition on the subject image, and when the subject image on the xy plane includes an armpit or crotch extending in the depth direction (z-axis direction), the image generation unit 52 displays a mark 112 indicating the need for an inspection in the depth direction on the display unit 18, and prompts the staff member to perform further inspection (S160). In addition to or instead of the mark 112, the image generation unit 52 may display text prompting further inspection.

[0122] Fig. 15 is a diagram for explaining an example of a mark 112 indicating the need for depth-direction examination by the display unit 18 according to the first embodiment in S160. Fig. 15 shows an example in which the mark 112 is displayed superimposed on the subject image 80, and text 114 indicating the need for further examination of the armpits or crotch is also displayed.

[0123] Following the display in S160, the staff member carries out a primary scan of the area including the subject's armpits or crotch while the subject is in a state where the subject's arms are apart or legs are apart (S108). This allows inspection of items that are not imaged by the received signal from the irradiation area on one plane (xy plane).

[0124] If the judgment result indicates danger, the image generation unit 52 supplies the judgment result, the subject image information, and the first item image information or the second item image information to the display unit 18 (S164). The display unit 18 displays the subject image based on the subject image information, superimposed with the first item image based on the first item image information or the second item image based on the second item image information, and also displays the judgment result (S166).

[0125] 16 is a diagram for explaining an example of display of the determination result by the display unit 18 according to the first embodiment in S166. The display unit 18 displays the first item image or the second item image 116 superimposed on the subject image 80, and also displays text 118 indicating the determination result.

[0126] The officer sees the display shown in FIG. 16 and recognizes that the subject may be concealing a specific item such as a handgun. The officer executes specific item response processing (S168). Examples of specific item response processing include issuing an alarm or requesting security guard backup. The officer then guides the next subject into the inspection area (S102). This ensures that a subject carrying a specific item is detected.

[0127] If the determination result indicates unknown, the image generation unit 52 supplies the determination result, the subject image information, the first image information, the second image information, and the third image information to the display unit 18 (S170). The display unit 18 displays a subject image 80 based on the subject image information by superimposing a first image based on the first image information, a second image based on the second image information, and a third image based on the third image information (S172).

[0128] 17 is a diagram for explaining an example of the display of the display unit 18 according to the first embodiment in S172. The display unit 18 displays the first image 102, the second image 108, and the third image (region requiring irradiation) 120 superimposed on the subject image 80 (S172).

[0129] While viewing the display in FIG. 17 , the staff member easily positions the scanner 30 by aligning the frame (region to be irradiated) of the second image 108 with one of the regions 120 requiring irradiation. The staff member then scans the region 120 requiring irradiation (S174). The scanning in S174 is the second scanning of the same subject and is referred to as a secondary scanning. After the secondary scanning in S174, the image generation unit 52 performs imaging using compressed sensing or machine learning to generate second article image information (S140).

[0130] As a result, if the aperture lengths of the multiple irradiation areas set in the primary scan in S108 are less than the required size and the judgment result is unclear, the area that needs to be irradiated to achieve the required aperture length is accurately scanned in the secondary scan. Alternatively, if the resolution of the second article image obtained by compressed sensing or machine learning imaging in S140 is less than the required resolution and the judgment result is unclear, the area that needs to be irradiated to achieve the required resolution is accurately scanned in the secondary scan. In this way, scanning is divided into primary and secondary scans, and the primary scan is easily and quickly achieved. Only the areas that could not be inspected in the primary scan are inspected in the secondary scan.

[0131] According to the first embodiment, even if an unexposed area occurs when an attendant scans a subject while moving the scanner 30, the display image shows the unexposed area or the area requiring irradiation. The attendant can position the scanner 30 while looking at this display so that the unexposed area is scanned again, eliminating the unexposed area. SAR imaging is performed based on the received signals from the multiple illuminated areas, and a high-resolution image of the article is generated. This provides a system and processing device that can perform accurate inspections in a short time.

[0132] Second embodiment In the first embodiment, the subject of inspection was a human being, but the subject of inspection is not limited to a human being, and may be an animal or a suspicious object.

[0133] 18 is a diagram for explaining an example of the system according to the second embodiment inspecting baggage (suitcase). As in the first embodiment, an officer scans the suitcase 200 from the outside with the scanner 30.

[0134] The second embodiment is effective for non-destructive inspection of baggage or suspicious boxes. The configuration of the second embodiment is the same as that of the first embodiment. The processing flow of the second embodiment is almost the same as that of the first embodiment. In the flowcharts of FIGS. 7, 8, and 9, the subject person is replaced by the object, and guiding the subject to the inspection area in S102 is replaced by placing the object in the inspection area.

[0135] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0136] 10... radar unit, 12... imaging unit, 14... measurement unit, 16... processing unit, 18... display unit, 42... transmitting array antenna, 44... transmitting circuit, 46... receiving array antenna, 48... receiving circuit, 52... image generating unit, 54... unirradiated area detecting unit, 56... determining unit

Claims

1. a radar that irradiates a target with radio waves and receives radio waves reflected from the target; an imaging unit that captures an image of the target and outputs target image information; a measurement unit that measures the position of the radar; a processing unit that outputs first image information representing a first region in the target that is not irradiated with radio waves based on the position of the radar; A system comprising:

2. The system according to claim 1 , wherein the processing unit determines the first image information based on a position of the radar when the radar emits the radio waves.

3. 10. The system of claim 1, wherein the radar comprises a transmit antenna comprising a uniformly spaced array antenna, a non-uniformly spaced array antenna, or a minimally redundant array antenna.

4. 10. The system of claim 1, wherein the radar comprises a receiving antenna comprising a uniformly spaced array antenna, a non-uniformly spaced array antenna, or a minimally redundant array antenna.

5. The system according to claim 1 , wherein the measurement unit measures the position of the radar using an output signal of a gyro sensor or the target image information.

6. The system according to claim 1 , further comprising a display unit that displays a target image based on the target image information and a first image based on the first image information.

7. The operation period of the system is a primary scanning period in which the radar irradiates the radio waves onto an arbitrary area of ​​the target; 7. The system according to claim 6, further comprising a secondary scanning period in which the radar irradiates the first area with the radio waves after outputting the first image information.

8. The system of claim 7 , wherein the display modes of the display unit include a first display mode in which the first image is not displayed and a second display mode in which the first image is displayed.

9. 9. The system according to claim 8, wherein the display mode of the display unit is the first display mode during the primary scanning period, and changes from the first display mode to the second display mode after the primary scanning period.

10. The system according to claim 6 , wherein the processing unit also generates article image information representing a specific article included in the target using reflected radio waves from at least one region.

11. The system according to claim 10 , wherein the display unit displays an article image based on the article image information by superimposing it on the target image.

12. the processing unit determines whether the target includes a specific item using reflected radio waves from at least one area; The system according to claim 6 , wherein the display unit displays a determination result as to whether the target includes the specific item.

13. The system of claim 6 , wherein the display comprises a flat-panel display, an eyeglass display, or a goggle display.

14. the target image information is two-dimensional image information on a first surface, The system of claim 6, wherein the first region includes a region in the object on the first surface that is not irradiated with the radio waves and a region in the object on a second surface that intersects with the first surface that is not irradiated with the radio waves.

15. The system according to claim 14 , wherein the processing unit causes the display unit to display text that prompts the radar to irradiate the radio waves onto the area on the second surface that is not irradiated with the radio waves.

16. The processing unit determines an irradiation area within the target based on the position of the radar, and generates a second image representing the irradiation area; The system of claim 6 , wherein the display unit displays the target image, the first image, and the second image.

17. The system according to claim 6 , wherein the display unit displays the first image continuously or flashing in a specific hue, brightness, or saturation.

18. The processing unit also outputs second image information representing a second region in the object irradiated with radio waves; The system according to claim 6, wherein the display unit displays the first image based on the first image information and the second image based on the second image information superimposed on the target image as a rectangle showing the entire area or a frame showing the outline of the area.

19. The processing unit also outputs second image information representing a second region in the object irradiated with radio waves; The system according to claim 6 , wherein the display unit displays the first image based on the first image information in the target image with a hue, brightness, or saturation different from that of the second image based on the second image information.

20. The processing unit also outputs second image information representing a second region in the object irradiated with radio waves; The system according to claim 6 , wherein the display unit continuously displays one of the first image based on the first image information and the second image based on the second image information on the target image, and displays the other image in a blinking manner.

21. A radar that irradiates a target with radio waves and receives radio waves reflected from the target, an imaging unit that images the target and outputs a target image, a measurement unit that measures the position of the radar, and a display unit that displays the target image are connected to the radar. a processing device that outputs, to the display unit, first image information that represents a first region of the target that is not irradiated with the radio waves based on the position of the radar;

Images