Electronic apparatus, system, control method, and program

Abstract

To provide an electronic apparatus, a system, a control method, and a program that can accurately detect information on an object on the basis of transmission and reception of radio waves.SOLUTION: An electronic apparatus comprises a controller that acquires information on a predetermined detection area detected on the basis of transmission of a transmission wave and reception of a reflected wave that is the reflected transmission wave, acquires an image of a predetermined imaging area, and extracts, from the information on the predetermined detection area, information on a detection area corresponding to an area of a predetermined object in the image of the predetermined imaging area.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an electronic device, a system, a control method, and a program. [Background technology]

[0002] For example, in the field of the automobile industry, a technology for measuring the distance between a vehicle and a predetermined object is considered important. In particular, in recent years, various RADAR (Radio Detecting and Ranging) technologies have been researched, which transmit radio waves such as millimeter waves and receive the reflected waves from an object such as an obstacle to measure the distance between the vehicle and the object. The importance of such technology for measuring distance is expected to increase in the future with the development of technologies for assisting the driver in driving and technologies related to autonomous driving that automates part or all of driving.

[0003] In object detection technologies such as the above-mentioned millimeter wave radar, various attempts have been made to improve the detection accuracy. For example, Patent Document 1 proposes a technology that can improve the detection accuracy of a detection target in a space where radio wave reflection exists. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-196195 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, it would be extremely beneficial if information about an object could be detected with high accuracy based on the transmission and reception of radio waves using a sensor such as a millimeter wave radar.

[0006] An object of the present disclosure is to provide an electronic device, a system, a control method, and a program that can accurately detect information about an object based on the transmission and reception of radio waves. [Means for solving the problem]

[0007] The electronic device according to an embodiment includes: Acquire information on a predetermined detection area detected based on transmission of a transmission wave and reception of a reflected wave of the transmission wave; Acquire an image of a predetermined imaging area, The image capturing apparatus further includes a controller for extracting, from the information on the predetermined detection area, information on a detection area corresponding to a region of a predetermined object in an image of the predetermined imaging area.

[0008] In one embodiment, the system includes: a detection unit that detects a predetermined detection area based on transmission of a transmission wave and reception of a reflected wave of the transmission wave; An imaging unit that captures an image of a predetermined imaging area; a controller that extracts, from information of the predetermined detection area detected by the detection unit, information of a detection area corresponding to a region of a predetermined object in an image of the predetermined imaging area captured by the imaging unit; Equipped with.

[0009] A control method according to an embodiment is a control method for an electronic device, A step of acquiring information on a predetermined detection area detected based on transmitting a transmission wave and receiving a reflected wave of the transmission wave; acquiring an image of a predetermined imaging area; extracting information of a detection area corresponding to a region of a predetermined object in an image of the predetermined imaging area from the information of the predetermined detection area; Includes.

[0010] A program according to an embodiment includes: For electronic devices, A step of acquiring information on a predetermined detection area detected based on transmitting a transmission wave and receiving a reflected wave of the transmission wave; acquiring an image of a predetermined imaging area; extracting information of a detection area corresponding to a region of a predetermined object in an image of the predetermined imaging area from the information of the predetermined detection area; Execute the command. Effect of the Invention

[0011] According to one embodiment, it is possible to provide an electronic device, a system, a control method, and a program that can accurately detect information about an object based on the transmission and reception of radio waves. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating a schematic functional configuration of an electronic device according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a schematic functional configuration of a detection unit according to an embodiment. [Diagram 3] FIG. 4 illustrates an example of a transmission signal transmitted by a sensor according to an embodiment. [Figure 4A] FIG. 2 is a diagram illustrating an example of a configuration of a detection device according to an embodiment. [Figure 4B] FIG. 2 is a diagram illustrating an example of a configuration of a detection device according to an embodiment. [Diagram 5] 10 is a flowchart illustrating an operation of an electronic device according to an embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of a UI of the electronic device according to an embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a process performed by an electronic device according to an embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a process performed by an electronic device according to an embodiment. [Figure 9] FIG. 11 is a diagram illustrating an example of coordinate conversion of an electronic device according to an embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of coordinate conversion of an electronic device according to an embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of a process performed by an electronic device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In the present disclosure, an "electronic device" may be a device driven by electricity. Furthermore, a "system" may include a device driven by electricity. Furthermore, a "user" may be a person (typically a human) who uses a system and / or an electronic device according to an embodiment. A user may include a person who enjoys benefits from detecting various objects by using a system and / or an electronic device according to an embodiment. The electronic device, method, and program of the present disclosure can be used to detect people, objects, animals, etc. present in a specified space such as a room, a bed, a bathroom, a toilet, a car, a bus, a train, a passageway, or a street.

[0014] The electronic device according to the embodiment described below makes it possible to narrow down information on objects detected by a detection unit (sensor) based on a technology such as millimeter wave radar to a specific object specified in a two-dimensional image. When a specific object is specified by a user in an image, the electronic device according to the embodiment can detect the object using a radar sensor. Therefore, the electronic device according to the embodiment can detect, for example, vibrations of a specific object in a non-contact manner. The electronic device according to the embodiment will be described in detail below with reference to the drawings. The sensor used in the present disclosure is not limited to millimeter wave radar, and may be a sensor that detects radio wave radar, ultrasonic waves, or light using a frequency band other than millimeter waves.

[0015] FIG. 1 is a functional block diagram that shows a schematic configuration of an electronic device 1 according to an embodiment. As shown in FIG. 1, the electronic device 1 according to an embodiment includes a controller 10. In an embodiment, the electronic device 1 may include at least one of a storage unit 20, a communication unit 30, and an output unit 40, as appropriate. The controller 10, the storage unit 20, the communication unit 30, and the output unit 40 described above may be disposed or built into any location in the electronic device 1. In addition, at least one of the controller 10, the storage unit 20, the communication unit 30, and the output unit 40 described above may be disposed outside the electronic device 1 and connected to each other by a network that is a wired or wireless connection or a combination of these. The electronic device 1 according to an embodiment may omit at least a part of the functional units shown in FIG. 1, or may include other functional units other than the functional units shown in FIG. 1 as appropriate.

[0016] The electronic device 1 according to an embodiment may be various devices. For example, the electronic device according to an embodiment may be any device such as a general-purpose smartphone, tablet, phablet, notebook computer (notebook PC), computer, or server, in addition to a terminal designed for exclusive use. The electronic device according to an embodiment may have a function of communicating with other electronic devices, such as a mobile phone or smartphone. Here, the above-mentioned "other electronic device" may be, for example, an electronic device such as a mobile phone or smartphone, or may be any device such as a base station, a server, a dedicated terminal, or a computer. The "other electronic device" may be, for example, a detection unit 100 and / or an imaging unit 300 described later. The "other electronic device" in the present disclosure may also be a device or apparatus driven by power. When the electronic device according to an embodiment communicates with the other electronic device, the communication may be wired and / or wireless.

[0017] As shown in Fig. 1, the electronic device 1 according to an embodiment may be connected to the detection unit 100 by wire and / or wireless. With such a connection, the electronic device 1 according to an embodiment can acquire information detected by the detection unit 100. Furthermore, the electronic device 1 according to an embodiment may be connected to the imaging unit 300 by wire and / or wireless. With such a connection, the electronic device 1 according to an embodiment can acquire an image captured by the imaging unit 300. The detection unit 100 and the imaging unit 300 will be described further below.

[0018] The controller 10 controls and / or manages the entire electronic device 1, including each functional unit constituting the electronic device 1. The controller 10 may also control and / or manage the entire system including each functional unit constituting the system including the electronic device 1. The controller 10 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing power for executing various functions. The controller 10 may be realized as a single processor, as a number of processors, or as individual processors. The processor may be realized as a single integrated circuit. The integrated circuit is also called an IC (Integrated Circuit). The processor may be realized as a number of integrated circuits and discrete circuits connected to each other in a communicative manner. The processor may be realized based on various other known technologies.

[0019] In one embodiment, the controller 10 may be configured as, for example, a CPU or a DSP and a program executed by the CPU or the DSP. The program executed in the controller 10 and the results of the processing executed in the controller 10 may be stored in, for example, the storage unit 20. The controller 10 may include a memory necessary for the operation of the controller 10 as appropriate.

[0020] In the electronic device 1 according to an embodiment, the controller 10 can perform various processes on information output as a result of detection by the detection unit 100, for example. For this reason, in the electronic device 1, the controller 10 may be connected to the detection unit 100 by wire and / or wirelessly. Also, in the electronic device 1 according to an embodiment, the controller 10 can perform various processes on information (image) output as a result of imaging by the imaging unit 300, for example. For this reason, in the electronic device 1, the controller 10 may be connected to the imaging unit 300 by wire and / or wirelessly. The operation of the controller 10 of the electronic device 1 according to an embodiment will be described further below.

[0021] The storage unit 20 may function as a memory that stores various information. The storage unit 20 may store, for example, a program executed in the controller 10 and a result of a process executed in the controller 10. The storage unit 20 may store or accumulate a detection result by the detection unit 100 and / or an image captured by the imaging unit 300. The storage unit 20 may also function as a work memory for the controller 10. The storage unit 20 may be configured, for example, by a semiconductor memory or the like, but is not limited thereto, and may be any storage device. For example, the storage unit 20 may be a storage medium such as a memory card inserted into the electronic device 1 according to an embodiment. The storage unit 20 may also be configured to include, for example, a hard disk drive (HDD) and / or a solid state drive (SSD). The storage unit 20 may be an internal memory of a CPU used as the controller 10 described later, or may be connected to the controller 10 as a separate unit.

[0022] The communication unit 30 has an interface function for wired or wireless communication. The communication method performed by the communication unit 30 of an embodiment may be a wireless communication standard. For example, the wireless communication standard includes cellular phone communication standards such as 2G, 3G, 4G, and 5G. For example, the cellular phone communication standard includes LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (Global System for Mobile communications), and PHS (Personal Handy-phone System). For example, the wireless communication standard includes WiMAX (Worldwide Interoperability for Microwave Access), IEEE802.11, WiFi, Bluetooth (registered trademark), IrDA (Infrared Data Association), and NFC (Near Field Communication). The communication unit 30 may include a modem whose communication method is standardized by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). The communication unit 30 can support one or more of the above communication standards.

[0023] The communication unit 30 may be configured to include, for example, an antenna for transmitting and receiving radio waves and an appropriate RF unit. The communication unit 30 may perform wireless communication with, for example, a communication unit of another electronic device via, for example, an antenna. The communication unit 30 may also be configured as an interface such as a connector for wired connection to the outside. The communication unit 30 may be configured using known technology for performing wireless communication, and therefore a more detailed description of the hardware and the like will be omitted.

[0024] The various information received by the communication unit 30 may be supplied to, for example, the storage unit 20 and / or the controller 10. The various information received by the communication unit 30 may be stored in, for example, a memory built into the controller 10. In addition, the communication unit 30 may transmit, for example, the processing results by the controller 10 and / or the information stored in the storage unit 20 to the outside.

[0025] The output unit 40 may be any display device such as a liquid crystal display (LCD), an organic electro-luminescence panel, or an inorganic electro-luminescence panel. The output unit 40 may display various information such as characters, figures, or symbols. The output unit 40 may also display objects constituting various GUIs and icon images, for example, to prompt the user to operate the electronic device 1. Various data required for displaying on the output unit 40 may be supplied from, for example, the controller 10 or the storage unit 20. In addition, when the output unit 40 includes, for example, an LCD, it may be configured to include a backlight, etc., as appropriate. In one embodiment, the output unit 40 may display information based on the result of detection by the detection unit 100, for example. In one embodiment, the output unit 40 may display information based on the result of imaging by the imaging unit 300, for example (for example, an image or video (including still images and videos)).

[0026] Further, the output unit 40 may notify a predetermined warning for alerting the user of the electronic device 1 based on a predetermined signal output from the controller 10. The output unit 40 may include any functional unit that stimulates at least one of the user's hearing, vision, and touch, such as sound, voice, light, text, video, and vibration, as the predetermined warning. Specifically, the output unit 40 may include at least one of a sound output unit such as a buzzer or speaker, a light-emitting unit such as an LED, a display unit such as an LCD, and a tactile sensation providing unit such as a vibrator. In this way, the output unit 40 may notify a predetermined warning based on a predetermined signal output from the controller 10. In one embodiment, the output unit 40 may notify a predetermined warning as information that acts on at least one of hearing, vision, and touch.

[0027] 1 includes an output unit 40. However, in one embodiment, the output unit 40 may be provided outside the electronic device 1. In this case, the output unit 40 and the electronic device 1 may be connected to each other by wire or wirelessly, or by a combination of wire and wireless.

[0028] At least a part of each functional unit constituting the electronic device 1 according to an embodiment as shown in FIG. 1 may be constituted by a specific means in which software and hardware resources work together.

[0029] The detection unit 100 shown in Fig. 1 is configured to detect an object (target) such as an automobile, a human body, or a building or structure as point cloud information in a three-dimensional space. The detection unit 100 according to an embodiment will be described in more detail below.

[0030] FIG. 2 is a functional block diagram illustrating a schematic configuration of the detection unit 100 according to an embodiment. The detection unit 100 illustrated in FIG. 2 is, as an example, a millimeter wave radar sensor based on millimeter wave radar (RADAR (Radio Detecting and Ranging)) technology. However, the detection unit 100 according to an embodiment is not limited to a millimeter wave radar sensor. The detection unit 100 according to an embodiment may be, for example, a quasi-millimeter wave radar sensor. Furthermore, the detection unit 100 according to an embodiment is not limited to a millimeter wave radar sensor or a quasi-millimeter wave radar sensor, and may be various radar sensors that transmit and receive radio waves. Furthermore, the detection unit 100 according to an embodiment may be, for example, a microwave sensor, an ultrasonic sensor, or a sensor based on technology such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0031] When measuring the distance to an object by a millimeter wave radar, a frequency modulated continuous wave radar (hereinafter, FMCW radar) may be used. In an FMCW radar, a transmission signal is generated by sweeping the frequency of radio waves to be transmitted. Therefore, in a millimeter wave FMCW radar using radio waves in a frequency band of 79 GHz, for example, the frequency of the radio waves used has a frequency bandwidth of 4 GHz, such as 77 GHz to 81 GHz. The millimeter wave radar of the present disclosure may use radio waves in a frequency band other than 77 GHz to 81 GHz. A radar in the 79 GHz frequency band has a feature that a usable frequency bandwidth is wider than other millimeter wave / quasi-millimeter wave radars in frequency bands such as 24 GHz, 60 GHz, and 76 GHz. Hereinafter, a case where such an FMCW radar is used will be described as an example. The FMCW radar system used in the present disclosure may include an FCM (Fast-Chirp Modulation) system that transmits a chirp signal at a shorter period than normal. The signal generated by the detection unit 100 is not limited to an FMCW system signal. The signal generated by the detection unit 100 may be a signal of various systems other than the FMCW system. The transmission signal sequence stored as the signal to be transmitted may differ depending on these various systems. For example, in the case of the radar signal of the above-mentioned FMCW system, a signal whose frequency increases and decreases for each time sample may be used. Since the above-mentioned various systems can be appropriately applied using known technologies, a more detailed description will be omitted as appropriate.

[0032] As shown in Fig. 2, the detection unit 100 according to an embodiment may include a signal processing unit 110, a transmission unit 120, and a reception unit 130. The above-mentioned signal processing unit 110, the transmission unit 120, the reception unit 130, and the like may be disposed or built into any location in the detection unit 100. In addition, at least one of the above-mentioned signal processing unit 110, the transmission unit 120, the reception unit 130, and the like may be disposed outside the detection unit 100. The detection unit 100 according to an embodiment may omit at least a part of the functional units shown in Fig. 2, or may include other functional units as appropriate, other than the functional units shown in Fig. 2.

[0033] The signal processing unit 110 controls and / or manages the entire detection unit 100, including each functional unit constituting the detection unit 100. The signal processing unit 110 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing power for executing various functions. The signal processing unit 110 may be realized as one processor, several processors, or each individual processor. The processor may be realized as a single integrated circuit. The integrated circuit is also called an IC (Integrated Circuit). The processor may be realized as a plurality of communicatively connected integrated circuits and discrete circuits. The processor may be realized based on various other known technologies.

[0034] In one embodiment, the signal processing unit 110 may be configured as, for example, a CPU or a DSP and a program executed by the CPU or the DSP. The program executed in the signal processing unit 110 and the results of the processing executed in the signal processing unit 110 may be stored in, for example, an arbitrary storage unit built into the signal processing unit 110. The signal processing unit 110 may include a memory necessary for the operation of the signal processing unit 110 as appropriate.

[0035] In the detection unit 100 according to an embodiment, the signal processing unit 110 may appropriately perform various types of processing such as distance FFT (Fast Fourier Transform) processing, speed FFT processing, arrival angle estimation processing, clustering processing, etc. Since each of these types of processing performed by the signal processing unit 110 is known as a general radar technology, a detailed description thereof will be omitted as appropriate.

[0036] 2, the transmitting unit 120 may include a signal generating unit 121, a synthesizer 122, a phase control unit 123, an amplifier 124, and a transmitting antenna 125. The detecting unit 100 according to an embodiment may include a plurality of transmitting antennas 125. In this case, the detecting unit 100 may also include a plurality of phase control units 123 and amplifiers 124 corresponding to the plurality of transmitting antennas 125. When the detecting unit 100 according to an embodiment includes a plurality of transmitting antennas 125, the plurality of transmitting antennas 125 may configure a transmitting antenna array (transmitting array antenna).

[0037] 2, the receiving unit 130 may include a receiving antenna 131, an LNA 132, a mixer 133, an IF unit 134, and an AD conversion unit 135. The detection unit 100 according to an embodiment may include a plurality of receiving units 130 corresponding to the plurality of transmitting antennas 125, respectively.

[0038] In the detection unit 100 according to an embodiment, the signal processing unit 110 can control at least one of the transmission unit 120 and the reception unit 130. In this case, the signal processing unit 110 may control at least one of the transmission unit 120 and the reception unit 130 based on various information stored in an arbitrary storage unit. For example, an arbitrary storage unit built in the signal processing unit 110 may store various parameters for setting a range (area) in which an object is detected by a transmission wave transmitted from the transmission antenna 125 and a reflected wave received from the reception antenna 131. In addition, in the detection unit 100 according to an embodiment, the signal processing unit 110 may instruct the signal generation unit 121 to generate a signal or control the signal generation unit 121 to generate a signal.

[0039] The signal generating unit 121 generates a signal (transmission signal) to be transmitted as a transmission wave from the transmission antenna 125 under the control of the signal processing unit 110. When generating the transmission signal, the signal generating unit 121 may allocate a frequency of the transmission signal based on the control by the signal processing unit 110, for example. Specifically, the signal generating unit 121 may allocate a frequency of the transmission signal according to a parameter set by the signal processing unit 110, for example. For example, the signal generating unit 121 receives frequency information from the signal processing unit 110 or an arbitrary storage unit, and generates a signal of a predetermined frequency in a frequency band such as 77 to 81 GHz, for example. The signal generating unit 121 may be configured to include a functional unit such as a voltage controlled oscillator (VCO).

[0040] The signal generating unit 121 may be configured as hardware having the function, or may be configured as a microcomputer, or may be configured as a processor such as a CPU or DSP and a program executed by the processor, etc. Each functional unit described below may also be configured as hardware having the function, or may be configured as a microcomputer, or may be configured as a processor such as a CPU or DSP and a program executed by the processor, etc., if possible.

[0041] In the detection unit 100 according to an embodiment, the signal generating unit 121 may generate a transmission signal (transmission chirp signal) such as a chirp signal. In particular, the signal generating unit 121 may generate a signal (linear chirp signal) whose frequency changes periodically and linearly. For example, the signal generating unit 121 may generate a chirp signal whose frequency increases periodically and linearly from 77 GHz to 81 GHz over time. Also, for example, the signal generating unit 121 may generate a signal whose frequency periodically repeats a linear increase (up chirp) and decrease (down chirp) from 77 GHz to 81 GHz over time. The signal generated by the signal generating unit 121 may be set in advance in, for example, the signal processing unit 110. Also, the signal generated by the signal generating unit 121 may be stored in advance in, for example, an arbitrary storage unit. Since chirp signals used in technical fields such as radar are known, a more detailed description will be appropriately simplified or omitted. The signal generated by the signal generating unit 121 is supplied to a synthesizer 122 .

[0042] FIG. 3 is a diagram illustrating an example of a chirp signal generated by the signal generating unit 121. As shown in FIG.

[0043] In Fig. 3, the horizontal axis represents the elapsed time, and the vertical axis represents the frequency. In the example shown in Fig. 3, the signal generating unit 121 generates a linear chirp signal whose frequency changes periodically and linearly. In Fig. 3, each chirp signal is shown as c1, c2, ..., c8. As shown in Fig. 3, in each chirp signal, the frequency increases linearly with the passage of time.

[0044] In the example shown in FIG. 3, eight chirp signals such as c1, c2, ..., c8 are included in one subframe. That is, subframe 1 and subframe 2 shown in FIG. 3 are each configured to include eight chirp signals such as c1, c2, ..., c8. Also, in the example shown in FIG. 3, 16 subframes such as subframe 1 to subframe 16 are included in one frame. That is, frame 1 and frame 2 shown in FIG. 3 are each configured to include 16 subframes. Also, as shown in FIG. 3, a frame interval of a predetermined length may be included between frames. One frame shown in FIG. 3 may be, for example, about 30 milliseconds to 50 milliseconds long.

[0045] In Fig. 3, frame 2 and onwards may have the same configuration. Also, in Fig. 3, frame 3 and onwards may have the same configuration. In the detection unit 100 according to an embodiment, the signal generation unit 121 may generate a transmission signal as an arbitrary number of frames. Also, in Fig. 3, some chirp signals are omitted. In this way, the relationship between time and frequency of the transmission signal generated by the signal generation unit 121 may be stored in, for example, an arbitrary storage unit.

[0046] In this manner, the detector 100 according to an embodiment may transmit a transmission signal consisting of subframes including a plurality of chirp signals. Also, the detector 100 according to an embodiment may transmit a transmission signal consisting of a frame including a predetermined number of subframes.

[0047] Hereinafter, the detection unit 100 will be described as transmitting a transmission signal having a frame structure as shown in FIG. 3. However, the frame structure as shown in FIG. 3 is an example, and the number of chirp signals included in one subframe is not limited to eight. In one embodiment, the signal generation unit 121 may generate a subframe including any number of chirp signals (for example, any multiple). The subframe structure as shown in FIG. 3 is also an example, and the number of subframes included in one frame is not limited to 16. In one embodiment, the signal generation unit 121 may generate a frame including any number of subframes (for example, any multiple). The signal generation unit 121 may generate signals of different frequencies. The signal generation unit 121 may generate a plurality of discrete signals having different bandwidths, each having a frequency f.

[0048] Returning to FIG. 2, the synthesizer 122 increases the frequency of the signal generated by the signal generating unit 121 to a frequency in a predetermined frequency band. The synthesizer 122 may increase the frequency of the signal generated by the signal generating unit 121 to a frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125. The frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be set by, for example, the signal processing unit 110. In addition, the frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be stored in, for example, an arbitrary storage unit. The signal whose frequency has been increased by the synthesizer 122 is supplied to the phase control unit 123 and the mixer 133. When there are multiple phase control units 123, the signal whose frequency has been increased by the synthesizer 122 may be supplied to each of the multiple phase control units 123. Furthermore, when there are multiple receiving sections 130 , the signal whose frequency has been increased by the synthesizer 122 may be supplied to each of the mixers 133 in the multiple receiving sections 130 .

[0049] The phase control section 123 controls the phase of the transmission signal supplied from the synthesizer 122. Specifically, the phase control section 123 may adjust the phase of the transmission signal by appropriately advancing or delaying the phase of the signal supplied from the synthesizer 122 based on the control by, for example, the signal processing section 110. In this case, the phase control section 123 may adjust the phase of each transmission signal based on the path difference of each transmission wave transmitted from the multiple transmission antennas 125. By the phase control section 123 appropriately adjusting the phase of each transmission signal, the transmission waves transmitted from the multiple transmission antennas 125 reinforce each other in a predetermined direction to form a beam (beamforming). In this case, the correlation between the direction of beamforming and the phase amount to be controlled of the transmission signal transmitted by each of the multiple transmission antennas 125 may be stored in, for example, an arbitrary storage section. The transmission signal phase-controlled by the phase control section 123 is supplied to the amplifier 124.

[0050] The amplifier 124 amplifies the power of the transmission signal supplied from the phase control section 123, for example, based on the control by the signal processing section 110. When the detection section 100 includes a plurality of transmission antennas 125, the plurality of amplifiers 124 may respectively amplify the power of the transmission signal supplied from each of the plurality of phase control sections 123 corresponding thereto, for example, based on the control by the signal processing section 110. Since the technology itself for amplifying the power of the transmission signal is already known, a detailed description thereof will be omitted. The amplifier 124 is connected to the transmission antenna 125.

[0051] The transmitting antenna 125 outputs (transmits) the transmission signal amplified by the amplifier 124 as a transmission wave. When the detection unit 100 includes a plurality of transmitting antennas 125, the plurality of transmitting antennas 125 may output (transmit) the transmission signals amplified by corresponding ones of the plurality of amplifiers 124 as transmission waves. The transmitting antenna 125 can be configured in the same manner as a transmitting antenna used in known radar technology, and therefore a detailed description thereof will be omitted.

[0052] In this way, the detection unit 100 according to one embodiment includes a transmitting antenna 125, and can transmit a transmission signal (e.g., a transmitting chirp signal) as a transmission wave from the transmitting antenna 125. Here, at least one of the functional units constituting the detection unit 100 may be housed in a single housing. In this case, the single housing may have a structure that makes it difficult to open. For example, the transmitting antenna 125, the receiving antenna 131, and the amplifier 124 may be housed in a single housing, and this housing may have a structure that makes it difficult to open. Furthermore,

[0053] The detection unit 100 shown in FIG. 2 shows an example including one transmission antenna 125. However, in one embodiment, the detection unit 100 may include any number of transmission antennas 125. On the other hand, in one embodiment, the detection unit 100 may include multiple transmission antennas 125 when the transmission wave transmitted from the transmission antenna 125 forms a beam in a predetermined direction. In one embodiment, the detection unit 100 may include any multiple transmission antennas 125. In this case, the detection unit 100 may also include multiple phase control units 123 and amplifiers 124 corresponding to the multiple transmission antennas 125. The multiple phase control units 123 may each control the phase of the multiple transmission waves supplied from the synthesizer 122 and transmitted from the multiple transmission antennas 125. Furthermore, the multiple amplifiers 124 may each amplify the power of the multiple transmission signals transmitted from the multiple transmission antennas 125. Furthermore, in this case, the detection unit 100 may be configured to include multiple transmission antennas. In this way, when the detection unit 100 includes a plurality of transmission antennas 125, the detection unit 100 may also be configured to include a plurality of functional units required for transmitting transmission waves from the plurality of transmission antennas 125.

[0054] The receiving antenna 131 receives the reflected wave. The reflected wave may be a transmission wave reflected by a predetermined object 200. The receiving antenna 131 may be configured to include a plurality of antennas. When the detection unit 100 according to an embodiment includes a plurality of receiving antennas 131, the plurality of receiving antennas 131 may form a receiving antenna array (receiving array antenna). The receiving antenna 131 may be configured in the same manner as a receiving antenna used in a known radar technology, and therefore a detailed description will be omitted. The receiving antenna 131 is connected to the LNA 132. A reception signal based on the reflected wave received by the receiving antenna 131 is supplied to the LNA 132. In this way, when the detection unit 100 includes a plurality of receiving antennas 131, the detection unit 100 may be configured to include a plurality of functional units required for receiving and processing the reflected wave from the plurality of receiving antennas 131.

[0055] The detection unit 100 according to one embodiment can receive a reflected wave that is a result of a transmission wave transmitted as a transmission signal (transmission chirp signal) such as a chirp signal from a plurality of receiving antennas 131 and reflected by a predetermined object 200. In this way, when a transmission chirp signal is transmitted as a transmission wave, a reception signal based on the received reflected wave is also referred to as a reception chirp signal. That is, the detection unit 100 receives a reception signal (for example, a reception chirp signal) as a reflected wave from the receiving antenna 131.

[0056] The LNA 132 amplifies, with low noise, a received signal based on a reflected wave received by the receiving antenna 131. The LNA 132 may be a low noise amplifier, and amplifies, with low noise, the received signal supplied from the receiving antenna 131. The received signal amplified by the LNA 132 is supplied to the mixer 133.

[0057] Mixer 133 generates a beat signal by mixing (multiplying) the RF frequency reception signal supplied from LNA 132 with the transmission signal supplied from synthesizer 122. The beat signal mixed by mixer 133 is supplied to IF unit .

[0058] The IF unit 134 reduces the frequency of the beat signal supplied from the mixer 133 to an intermediate frequency (IF frequency). The beat signal whose frequency has been reduced by the IF unit 134 is supplied to an AD conversion unit 135.

[0059] The AD conversion unit 135 digitizes the analog beat signal supplied from the IF unit 134. The AD conversion unit 135 may be configured with any analog-to-digital conversion circuit (Analog to Digital Converter (ADC)). The beat signal digitized by the AD conversion unit 135 is supplied to the signal processing unit 110. When there are multiple receiving units 130, each of the beat signals digitized by the multiple AD conversion units 135 may be supplied to the signal processing unit 110.

[0060] The signal processing unit 110 may perform FFT processing on the beat signal digitized by the AD conversion unit 135 (hereinafter, appropriately referred to as "distance FFT processing"). For example, the signal processing unit 110 may perform FFT processing on the complex signal supplied from the AD conversion unit 135. The beat signal digitized by the AD conversion unit 135 can be expressed as a time change in signal strength (power). The signal processing unit 110 performs FFT processing on such a beat signal, thereby enabling it to be expressed as a signal strength (power) corresponding to each frequency. By performing distance FFT processing in the signal processing unit 110, a complex signal corresponding to distance can be obtained based on the beat signal digitized by the AD conversion unit 135.

[0061] When a peak is equal to or greater than a predetermined threshold in the result obtained by the distance FFT processing, the signal processing unit 110 may determine that a predetermined object 200 is present at the distance corresponding to the peak. For example, a method is known in which, when a peak value equal to or greater than a threshold is detected from the average power or amplitude of a disturbance signal, such as detection processing based on a constant false alarm rate (CFAR), an object that reflects a transmission wave (a reflecting object) is present.

[0062] In this manner, the detection unit 100 according to one embodiment can detect an object 200 reflecting a transmission wave as a target based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave.

[0063] The signal processing unit 110 can estimate the distance to a predetermined object based on one chirp signal (e.g., c1 shown in FIG. 3). That is, the detection unit 100 can measure (estimate) the distance between the detection unit 100 and the predetermined object 200 by performing distance FFT processing. Since the technology itself for measuring (estimating) the distance to a predetermined object by performing FFT processing on a beat signal is well known, a more detailed description will be appropriately simplified or omitted.

[0064] Furthermore, the signal processing unit 110 may further perform FFT processing (hereinafter, appropriately referred to as "velocity FFT processing") on the beat signal that has been subjected to the distance FFT processing. For example, the signal processing unit 110 may perform FFT processing on the complex signal that has been subjected to the distance FFT processing. The signal processing unit 110 can estimate the relative velocity with respect to a predetermined object based on a subframe of the chirp signal (for example, subframe 1 shown in FIG. 3). By performing velocity FFT processing on multiple chirp signals in the signal processing unit 110, a complex signal corresponding to the relative velocity can be obtained based on the complex signal corresponding to the distance obtained by the distance FFT processing.

[0065] As described above, when the beat signal is subjected to distance FFT processing, a plurality of vectors can be generated. The relative velocity with respect to a predetermined object can be estimated by obtaining the phase of the peak in the result of performing velocity FFT processing on these plurality of vectors. That is, the electronic device 1 can measure (estimate) the relative velocity between the detection unit 100 and the predetermined object 200 by performing velocity FFT processing. In this way, the signal processing unit 110 may perform a fast Fourier transform (Two Dimensional Fast Fourier Transform: 2D-FFT) in the range direction and the velocity direction based on the beat signal. Since the technology itself of measuring (estimating) the relative velocity with respect to a predetermined object by performing velocity FFT processing on the result of distance FFT processing is well known, a more detailed description will be appropriately simplified or omitted.

[0066] In general FMCW radar technology, it is possible to determine whether or not a target is present based on the results of performing fast Fourier transform processing on beat frequencies extracted from a received signal. Here, the results of performing fast Fourier transform processing on beat frequencies extracted from a received signal also contain noise components due to clutter (unwanted reflected components). Therefore, a process may be performed to remove noise components from the results of processing the received signal and extract only the target signal.

[0067] Furthermore, the signal processing unit 110 may estimate (arrival direction estimation) the direction (arrival angle) from which the reflected wave arrives from the predetermined object 200 based on the determination of whether or not the target exists. The signal processing unit 110 may perform arrival direction estimation for the point where it is determined that the target exists. The detection unit 100 can estimate the direction from which the reflected wave arrives by receiving the reflected wave from the multiple receiving antennas 131. For example, it is assumed that the multiple receiving antennas 131 are arranged at a predetermined interval. In this case, the transmission wave transmitted from the transmission antenna 125 is reflected by the predetermined object 200 to become a reflected wave, and the multiple receiving antennas 131 arranged at a predetermined interval each receive the reflected wave R. Then, the signal processing unit 110 can estimate the direction from which the reflected wave arrives at the receiving antenna 131 based on the phase of the reflected wave received by each of the multiple receiving antennas 131 and the path difference of each reflected wave. That is, the detection unit 100 can measure (estimate) the arrival angle θ, which indicates the direction from which the reflected wave reflected by the target arrives, based on the result of the velocity FFT process.

[0068] Various techniques have been proposed for estimating the direction of arrival of the reflected wave R based on the results of velocity FFT processing. For example, known algorithms for estimating the direction of arrival include MUSIC (MUltiple SIgnal Classification) and ESPRIT (Estimation of Signal Parameters via Rotational Invariance Technique). Therefore, detailed descriptions of known techniques will be appropriately simplified or omitted.

[0069] The signal processing unit 110 detects an object present in an area where a transmission wave is transmitted based on at least one of distance FFT processing, speed FFT processing, and arrival angle estimation. The signal processing unit 110 may perform object detection by, for example, clustering processing based on the supplied distance information, speed information, and angle information. As an algorithm used for clustering data, for example, DBSCAN (Density-based spatial clustering of applications with noise) is known. In the clustering processing, for example, the average power of points constituting a detected object may be calculated.

[0070] As described above, the detection unit 100 can detect an object reflecting a transmission wave in a three-dimensional space as point cloud information. That is, in one embodiment, based on the detection result output from the detection unit 100, it can determine (detect) whether or not an object reflecting a transmission wave exists at a certain coordinate in the three-dimensional space. Also, in one embodiment, the detection unit 100 can detect the signal strength and speed of each point in the three-dimensional space. As described above, the detection unit 100 according to one embodiment may detect an object reflecting a transmission wave as point cloud information in the three-dimensional space based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave.

[0071] 1, an electronic device 1 according to an embodiment may include an imaging section 300. The electronic device 1 and the imaging section 300 may be connected to each other by wire or wirelessly, or by a combination of wire and wireless connection.

[0072] The imaging unit 300 may be configured to include an image sensor that electronically captures an image, such as a digital camera. The imaging unit 300 may be configured to include an imaging element that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 300 may capture an image of an object that exists in front of the imaging unit 300. Here, the object that exists in front of the imaging unit 300 may be, for example, a car, a person, and / or any object that exists in the surroundings. The imaging unit 300 may convert the captured image into a signal and transmit it to the electronic device 1. For example, the imaging unit 300 may transmit a signal based on the captured image to the storage unit 20, the communication unit 30, the output unit 40, and / or the controller 10 of the electronic device 1. The imaging unit 300 is not limited to an imaging device such as a digital camera, and may be any device that can capture an object. The imaging unit 300 may be, for example, a LIDAR (Light Detection And Ranging).

[0073] In one embodiment, the imaging section 300 may capture still images at predetermined time intervals (for example, 15 frames per second). Also, in one embodiment, the imaging section 300 may capture continuous moving images, for example.

[0074] Next, the arrangement of the detection unit 100 and the imaging unit 300 connected to the electronic device 1 according to an embodiment will be described.

[0075] 4A and 4B are diagrams showing an example of the configuration of a detection device in which the detection section 100 and the imaging section 300 are arranged.

[0076] Fig. 4A is a front view showing an example of the detector 3 according to an embodiment as viewed from the front. Fig. 4B is a side view showing an example of the detector 3 according to an embodiment as viewed from the side (left). The coordinate axes shown in Fig. 4A and Fig. 4B are aligned with the coordinate axes showing the propagation direction of the transmission wave and / or the reflected wave of the detector 100 shown in Fig. 2.

[0077] 4A and 4B, the detection device 3 according to an embodiment may include a detection section 100 and an imaging section 300. Also, as shown in Fig. 4A and 4B, the detection device 3 according to an embodiment may include at least one of a holder section 5 and an attachment section 7, etc. as appropriate. Also, the detection device 3 according to an embodiment may be disposed in any other device or a housing of another device, etc., without including at least one of the holder section 5 and the attachment section 7.

[0078] The detection unit 100 shown in FIG. 4A and FIG. 4B may be the detection unit 100 described in FIG. 1 and / or FIG. 2. As shown in FIG. 4A and FIG. 4B, the detection unit 100 may include a radio wave input unit 101 that receives a reflected wave of a transmitted wave reflected by an object. In the detection unit 100, at least one of the transmitting antenna 125 and the receiving antenna 131 may be disposed at a position corresponding to the radio wave input unit 101. As shown in FIG. 4B, the radio wave input unit 101 may be oriented along the optical axis Ra of the detection unit 100. Here, the optical axis of the detection unit 100 may be, for example, a direction perpendicular to a surface on which at least one of the transmitting antenna 125 and the receiving antenna 131 of the detection unit 100 is installed. In addition, when at least one of the transmitting antenna 125 and the receiving antenna 131 is included in a plurality of antennas, the optical axis of the detection unit 100 may be a direction perpendicular to a surface on which at least one of the plurality of antennas is installed. With this configuration, the detection unit 100 can transmit a transmission wave and / or receive a reflected wave centered on the optical axis Ra. That is, the detection unit 100 can detect an object as a point cloud in an area centered on the optical axis Ra.

[0079] The detection unit 100 according to an embodiment may have directivity. That is, the detection unit 100 may detect an object by directional radio waves. Here, the directivity may be a relationship between the radiation direction and radiation intensity of radio waves as a characteristic of an antenna. The presence or absence of directivity is related to the use of the antenna. An antenna with strong directivity strongly radiates radio waves in a specific direction. The directivity may have the same characteristics in the case of transmission and the case of reception. The electric field intensity of the radio waves radiated by the antenna can be expressed in decibels (dB) as the gain of the antenna. The detection unit 100 according to an embodiment may have directivity and have a main lobe (main beam) in the direction of the optical axis Ra shown in FIG. 4B, for example. That is, the detection unit 100 according to an embodiment may have the strongest radiation level in the direction of the optical axis Ra, for example.

[0080] 4A and 4B may be the imaging section 300 described in FIG. 1. As shown in FIG. 4A and 4B, the imaging section 300 may include a light input section 301 that receives light reflected by an object. In the imaging section 300, an image sensor such as a light receiving element or an imaging element may be disposed at a position corresponding to the light input section 301. As shown in FIG. 4B, the light input section 301 may face the optical axis La of the imaging section 300. Also, the light input section 301 may be a position where a lens is disposed in the imaging section 300. Here, the optical axis of the imaging section 300 may be, for example, a direction perpendicular to a surface on which a light receiving element (or an imaging element) used for imaging in the imaging section 300 is disposed. With such a configuration, the imaging section 300 can capture an image centered on the optical axis La.

[0081] As shown in FIG. 4A and FIG. 4B, the holder unit 5 maintains the detection unit 100 and the imaging unit 300 in a fixed state in the detection device 3. The holder unit 5 may maintain the detection unit 100 at a height where the detection unit 100 can easily detect a predetermined detection area. The holder unit 5 may maintain the detection unit 100 at a height where the imaging unit 300 can easily obtain an image of a predetermined imaging area. The detection unit 100 and the imaging unit 300 may be screwed to the holder unit 5, or may be fixed to the holder unit 5 with an adhesive or the like, or may be fixed by being fitted into the holder unit 5. The holder unit 5 may include a mechanism capable of adjusting the optical axes of the detection unit 100 and the imaging unit 300 in the detection device 3.

[0082] 4A and 4B, the attachment unit 7 enables the detection device 3 in which the detection unit 100 and the imaging unit 300 are installed to be attached and fixed to, for example, another member or another device. The attachment unit 7 may be fixed to, for example, another member or another device by screws, or may be fixed by adhesive or the like, or may be fixed by being fitted in. The attachment unit 7 may have various configurations, such as a pedestal shape, in order to stabilize the detection device 3 including the detection unit 100 and the imaging unit 300.

[0083] The holder part 5 and the attachment part 7 may be made of a metal such as an aluminum plate, or may be made of a material such as an acrylic plate. The holder part 5 and the attachment part 7 may be made of an aluminum plate or an acrylic plate with a thickness of about 5 mm.

[0084] As shown in Figures 4A and 4B, in the detection device 3 according to an embodiment, the detection unit 100 and the imaging unit 300 may be disposed adjacent to each other in the vicinity of each other. In the example shown in Figures 4A and 4B, the detection unit 100 and the imaging unit 300 are disposed adjacent to each other in the up-down direction (Z-axis direction). In the detection device 3 according to an embodiment, the detection unit 100 and the imaging unit 300 may be disposed adjacent to each other in the left-right direction or in an oblique direction, for example.

[0085] 4B, in the detection device 3 according to an embodiment, the detection unit 100 and the imaging unit 300 may be arranged so that their optical axes Ra and La are parallel (or nearly parallel). That is, in the electronic device 1 according to an embodiment, in a state in which the optical axis La of the imaging unit 300 is installed so as to be parallel (or nearly parallel) to the optical axis Ra of the detection unit 100, the point cloud information from the detection unit 100 and the image from the imaging unit 300 may be used.

[0086] 4B, in the detection device 3 according to one embodiment, the detection unit 100 and the imaging unit 300 may be disposed such that the distance between their respective optical axes Ra and La is kept at a distance G. By disposing them in this manner, the point cloud information from the detection unit 100 and the image information from the imaging unit 300 are shifted from each other by the distance G. Therefore, the detection unit 100 and the imaging unit 300 may be disposed such that the distance G between them is relatively small, for example, 2 to 3 cm.

[0087] In one embodiment, the controller 10 may correct at least one of the point cloud information obtained by the detection unit 100 detecting an object (target) and the image obtained by the imaging unit 300 capturing an image of the object (target). In one embodiment, the controller 10 may use information obtained by aligning the point cloud information from the detection unit 100 and the image from the imaging unit 300 through correction.

[0088] As described above, the electronic device 1 according to one embodiment can utilize an image obtained by the imaging unit 300 capturing an image of an object and point cloud information obtained by the detection unit 100 detecting the object.

[0089] Next, the operation of the electronic device 1 according to the embodiment will be described.

[0090] For example, the detection unit 100, such as a radar sensor, can detect a predetermined detection area based on transmitting a transmission wave and receiving a reflected wave of the transmission wave. More specifically, the detection unit 100 can detect (or estimate) the position, speed, and / or arrival angle of a predetermined object by receiving a reflected wave of the transmission wave reflected by the object.

[0091] The detection unit 100 can also detect the vibration of an object (such as the presence or absence of vibration or the degree of vibration) by continuing to detect the position and / or speed of the object for a predetermined period of time. The detection unit 100 can detect the vibration of an object in a non-contact manner, for example, by using millimeter waves. For example, the detection unit 100 can detect the vibration of various facilities such as buildings or structures. In recent years, extending the life and strengthening infrastructure has become an important social issue. Vibrations of infrastructure and the like are important physical information for determining the soundness of the infrastructure and the like. For this reason, there is a demand for a means of efficiently inspecting various buildings or structures (such as buildings, bridges, or tunnels) including various infrastructure facilities. For example, if there is a sensing system that can measure the vibration of an object in a non-contact manner using a radar sensor such as a millimeter wave radar, it is promising as a means of efficiently inspecting infrastructure and the like.

[0092] However, a technology such as millimeter wave radar detects responses from objects present in a predetermined detection area based on the transmission of a transmission wave and the reception of a reflected wave. For this reason, when it is desired to detect vibrations of a building present in a predetermined detection area using a millimeter wave radar, it is assumed that in addition to the building, for example, a car is also present in the detection area. In such a case, the millimeter wave radar detects not only the vibrations of the building, but also the vibrations of the car. Therefore, with such detection, it may be difficult to detect the vibrations of only the building with good accuracy.

[0093] For example, in the above-mentioned example, if the user specifies a building, vibration information corresponding to the building is extracted and the vibration of the building is detected, not only is the convenience high, but the vibration of the building can be detected with high accuracy. Therefore, the electronic device 1 according to an embodiment extracts vibration information corresponding to an object, for example, when the object is specified by the user. Therefore, the electronic device 1 according to an embodiment can detect the vibration of the object (for example, even if an object other than the object exists in the detection area). Therefore, according to the electronic device 1 according to an embodiment, not only the convenience can be improved, but also the vibration of the object can be detected with high accuracy. Hereinafter, such an embodiment will be described.

[0094] Fig. 5 is a flowchart showing an example of the operation of the electronic device 1 according to an embodiment. The operation shown in Fig. 5 may be started, for example, when a user of the electronic device 1 according to an embodiment detects vibration of a predetermined object. At the time when the operation shown in Fig. 5 starts, the detection unit 100 is assumed to be ready to detect a predetermined detection area based on the transmission of a transmission wave and the reception of a reflected wave of the transmission wave by transmitting and receiving radio waves such as millimeter waves. Also, at the time when the operation shown in Fig. 5 starts, the imaging unit 300 is assumed to be ready to capture an image of a predetermined imaging area.

[0095] When the operation shown in FIG. 5 starts, the controller 10 of the electronic device 1 according to the embodiment acquires an image of a predetermined imaging area captured by the imaging section 300 (step S11).

[0096] In one embodiment, the controller 10 may display the image acquired in step S11 on, for example, the output unit 40 (display). FIG. 6 is a diagram showing an example in which an image captured by the imaging unit 300 is displayed on the output unit 40 (display). In FIG. 6, the output unit 40 displays an image 40e of the inside of a facility captured by the imaging unit 300. A loudspeaker 40f placed on a desk is displayed near the center of the lower part of the image 40e of the inside of the facility shown in FIG. 6. In FIG. 6, this loudspeaker 40f is the object. Of course, the object of the present disclosure is not limited to the loudspeaker 40f, and may be any appropriate object such as a person, an object, or an animal. As shown in FIG. 6, the output unit 40 may display a bounding box 40d. The bounding box 40d may be used to set a predetermined object (or object area) in the acquired image. In addition, the output unit 40 may display a setting box 40a that sets the left end of the bounding box 40d. The output unit 40 may also display a setting box 40b for setting the right edge of the bounding box 40d. The output unit 40 may also display a setting box 40c for setting the distance of a target (object). In the example shown in Fig. 6, the bounding box 40d includes an object 40f (loudspeaker 40f) in the image 40e.

[0097] Next, the controller 10 sets a predetermined object (or target area) in the image acquired in step S11 (step S12). In step S12, the controller 10 may set the predetermined object (or target area) in the image based on an operation by a user. For example, the controller 10 may prompt the user to set the predetermined object (or target area) via a user interface (UI) as shown in FIG. 6. The UI shown in FIG. 6 prompts the user to set the predetermined object (or target area) in the image by a bounding box 40d. In the UI shown in FIG. 6, the user can adjust (set) the horizontal range of the loudspeaker shown in FIG. 6 by specifying the horizontal range (Left Edge and Right Edge) of the bounding box by the setting boxes 40a and / or 40b.

[0098] Fig. 6 shows a case where the detection unit 100 performs arrival direction estimation in the horizontal direction. That is, the UI shown in Fig. 6 prompts the user to set the horizontal range of a predetermined object (or target area) in the image by a bounding box. On the other hand, if the detection unit 100 can also perform arrival direction estimation in the vertical direction, for example, the UI shown in Fig. 6 may prompt the user to set the vertical range of a predetermined object (or target area) in the image by a bounding box.

[0099] Furthermore, in the case where the output unit 40 (display) includes a touch panel, the controller 10 may set a predetermined object (or target area) based on a touch (tap) operation on the touch panel by the user. In this case, when the user touches (tap) a predetermined object in the image of the UI shown in Fig. 6, the object (or target area) corresponding to the touched image may be automatically set.

[0100] Furthermore, the controller 10 may set a predetermined object (or object area) in an image without being based on an operation by a user. For example, the controller 10 may perform image recognition or the like on an image such as that shown in FIG. 6, so that when a predetermined object is included in the image, the predetermined object (or object area) is automatically set. For example, it is assumed that a loudspeaker has been set in advance as an object to be recognized by image recognition. In this case, when a loudspeaker is included in the image of the UI shown in FIG. 6, the object (or object area) corresponding to the image of the loudspeaker may be automatically set without a user operation.

[0101] Once the target object is set in step S12, the controller 10 acquires information on a predetermined detection area detected by the detection unit 100 (step S13). In step S13, the controller 10 may acquire information on a predetermined detection area detected based on the transmission of a transmission wave and the reception of a reflected wave of the transmission wave. The process of step S13 does not necessarily have to be performed after steps S11 and S12, and may be performed, for example, before steps S11 and S12.

[0102] The predetermined detection area detected by the detection unit 100 in step S13 needs to include the object in the image set in step S12. As described above, the detection unit 100 and the imaging unit 300 may be arranged so that their respective optical axes Ra and La are parallel (or nearly parallel). With this configuration, the imaging unit 300 can capture an image centered on the optical axis La. Therefore, the predetermined detection area detected by the detection unit 100 can be made to include the object set in the image.

[0103] After acquiring the information of the predetermined detection area in step S13, the controller 10 acquires a point cloud near the designated distance (step S14). In step S14, the controller 10 may extract a point cloud near the designated distance from information corresponding to coordinates (point cloud) in three-dimensional space. Here, the "designated distance" may represent the distance to an object to be detected by the detection unit 100.

[0104] For example, in the image shown in Fig. 6, the distance (from the detection device 3) to the loudspeaker is about 70 cm. In this case, the controller 10 may prompt the user to input the value of 70 cm as the distance (Target Distance) to the object (target) in the UI shown in Fig. 6. In step S14, the controller 10 may extract a point cloud near the specified 70 cm from information corresponding to coordinates (point cloud) in three-dimensional space.

[0105] Furthermore, the detection unit 100 can detect the distance to the target object. Therefore, in step S14, the controller 10 may specify, for example, the distance to the target object detected by the detection unit 100 as the above-mentioned "specified distance". In this case, the controller 10 does not need to prompt the user to input the distance to the target object (target). In step S14, the controller 10 may extract a point cloud near the distance to the target object detected by the detection unit 100 from information corresponding to coordinates (point cloud) in the three-dimensional space.

[0106] In a simplified embodiment, the process of step S14 may be omitted.

[0107] Next, the controller 10 performs an arrival direction estimation for the points extracted as points near the specified distance from the information corresponding to the coordinates (points) in the three-dimensional space (step S15). In step S15, the controller 10 may perform a process based on the same concept as the process performed by the signal processor 110 as a normal basic radar process.

[0108] The information of the predetermined detection area obtained at the time of completing the process of step S15 may be information on which basic radar processing has been performed by the signal processing unit 110 of the detection unit 100 or the controller 10. For example, the process up to step S14 may include the above-mentioned 2D-FFT process and CFAR process. In the 2D-FFT process, 2D-FF process may be performed on the IF signal received by the detection unit 100. In the CFAR process, CFAR process may be performed in a range near a specified distance on the result calculated by the 2D-FFT process to obtain information on a point cloud. In addition, for example, the process up to step S15 may include the above-mentioned process of estimating the arrival direction. In the arrival direction estimation process, a process of estimating the arrival direction may be performed on the point cloud obtained by the CFAR process.

[0109] FIG. 7 is a diagram showing an example of the result of distance FFT processing performed by the signal processing unit 110 of the detection unit 100 in the situation shown in FIG. 6. The horizontal axis represents the distance (range) detected by the detection unit 100, and the vertical axis represents the intensity of the FFT-processed signal. As shown in FIG. 7, a peak of the FFT intensity, which is thought to be due to the detection of a loudspeaker, can be confirmed near a distance (range) of 0.7 m (70 cm). FIG. 8 is a diagram showing an example of the result of velocity FT processing performed by the signal processing unit 110 of the detection unit 100 in the situation shown in FIG. 6. The horizontal axis represents the distance (range) detected by the detection unit 100, and the vertical axis represents the velocity component of the FFT-processed signal. As shown in FIG. 7, a peak of the velocity component, which is thought to be due to the detection of a loudspeaker, can be confirmed near a distance (range) of 0.7 m (70 cm).

[0110] When the process in step S15 is completed, the controller 10 can obtain information corresponding to coordinates (point group) in the three-dimensional space as a point group near the specified distance. Next, the controller 10 converts the obtained coordinates of the points in the three-dimensional space into coordinates on a two-dimensional plane (step S16). That is, in step S16, the controller 10 makes the coordinates of the points in the three-dimensional space detected by the detection unit 100 correspond to the coordinates on the image captured by the imaging unit 300. In step S16, the controller 10 may make the coordinates of the points in the three-dimensional space detected by the detection unit 100 correspond to the coordinates on the image captured by the imaging unit 300 by any method.

[0111] An example of a method for associating the coordinates of a point group in a three-dimensional space detected by the detection section 100 with coordinates on an image captured by the imaging section 300 will be described below.

[0112] The point cloud data acquired in step S15 is information indicating the three-dimensional position of the object detected in the three-dimensional space. Therefore, the controller 10 converts the three-dimensional point cloud data acquired in step S15 into two-dimensional data (step S16). In step S16, the controller 10 may generate point cloud information that can be processed two-dimensionally from the point cloud information in the three-dimensional space output by the detection unit 100.

[0113] Fig. 9 is a diagram illustrating information on a group of points in a three-dimensional space output by the detection unit 100. Fig. 9 is a diagram illustrating an example of a situation in which an object Tm located at a certain location is three-dimensionally detected by the detection unit 100 installed at the origin O.

[0114] As shown in FIG. 9, the position where the detection unit 100 is installed is taken as the reference (origin O), and the direction approaching the object Tm is taken as the X-axis positive direction, and the direction moving away from the object Tm is taken as the X-axis negative direction. Also, as shown in FIG. 9, the position where the detection unit 100 is installed is taken as the reference (origin O), and the right side of the detection unit 100 is taken as the Y-axis positive direction, and the left side of the detection unit 100 is taken as the Y-axis negative direction. Also, as shown in FIG. 9, the position where the detection unit 100 is installed is taken as the reference (origin O), and the upper side of the detection unit 100 is taken as the Z-axis positive direction, and the lower side of the detection unit 100 is taken as the Z-axis negative direction. That is, in FIG. 9, lx indicates the distance in the depth direction, ly indicates the distance in the horizontal direction, and lz indicates the distance in the vertical (height) direction. In the situation shown in FIG. 9, the output of the detection unit 100 may be four-channel data of data elements (signal strength and speed) of positions (X, Y, Z) in three-dimensional space at a certain moment. In step S16 shown in FIG. 5, the controller 10 converts the information detected in such a three-dimensional space into two-dimensional information.

[0115] Fig. 10 is a diagram for explaining an example of generating point cloud information that can be processed two-dimensionally from point cloud information in a three-dimensional space output by the detection unit 100. Fig. 10 is a diagram showing an example in which an object Tm located at a certain location is detected three-dimensionally (spatially) by the detection unit 100 installed at the origin O, and converted into two-dimensionally (planarly) by the electronic device 1.

[0116] As shown in FIG. 10, with the position of the origin O as a reference, the right side of the origin O is the X-axis positive direction, and the lower side of the origin O is the Y-axis positive direction. That is, in FIG. 10, px indicates the horizontal coordinate, and py indicates the vertical coordinate. In the situation shown in FIG. 10, the output of the detection unit 100 is converted at a certain moment from the above-mentioned four-channel data to three-channel data of data elements (signal strength and speed) of position (X, Y) on a two-dimensional plane. In this way, in step S16 shown in FIG. 5, the electronic device 1 converts the information detected in the three-dimensional space into information on a two-dimensional plane.

[0117] When converting the information detected in the three-dimensional space as described above into information on a two-dimensional plane, the coordinates of the two-dimensional plane may be calculated based on, for example, the following formulas (1) and (2).

[0118]

number

[0119]

number

[0120] In the above formula (1) and formula (2), lx i ,ly i ,lz i indicates the output based on the result of detection by the detection unit 100, that is, the information of the point group in the three-dimensional space. i indicates the distance in the x direction of the i-th piece of information detected by the detection unit 100. i indicates the distance in the y direction of the i-th piece of information detected by the detection unit 100. i indicates the distance in the z direction of the i-th piece of information detected by the detection unit 100.

[0121] In addition, in the above formulas (1) and (2), px i ,py i indicates the coordinates of the point group converted into two-dimensional plane information by the controller 10 of the electronic device 1. In particular, px i indicates the x-coordinate of the i-th piece of information detected by the detection unit 100. i indicates the y coordinate of the i-th piece of information detected by the detection unit 100.

[0122] Furthermore, in the above formula (1), M indicates the number of pixels in the horizontal direction when the image is assumed to be a two-dimensional plane, and αx indicates the horizontal angle of view when the image is assumed to be a two-dimensional plane. Also, in the above formula (2), N indicates the number of pixels in the vertical direction when the image is assumed to be a two-dimensional plane, and αy indicates the vertical angle of view when the image is assumed to be a two-dimensional plane.

[0123] In the above formula (1) and formula (2), px i ,py i In order to function as a coordinate value, px may be rounded off to an integer. i ,py i is set so that it fits within the size of the image after it is converted to a 2D plane, e.g., 0≦px i ≦M, or 0≦py i Data that does not satisfy N may be discarded.

[0124] In this way, the electronic device 1 according to an embodiment may generate two-dimensionally processable point cloud information from the output of the detection unit 100. In particular, the electronic device 1 according to an embodiment may convert the point cloud information detected by the detection unit 100 into two-dimensionally processable point cloud information based on at least one of a predetermined number of pixels in a two-dimensional image and a predetermined angle of view. The electronic device 1 according to an embodiment may generate two-dimensionally processable point cloud information from the output of the detection unit 100 based on a conversion formula other than the above-mentioned formula (1) and formula (2). Each point constituting the two-dimensional point cloud data generated in step S16 may include three-channel information of reflection intensity, distance, and speed at the point.

[0125] Once the three-dimensional coordinates of the point are converted to two dimensions in step S14, the controller 10 determines whether or not the point is within a detection area corresponding to an object in the image (step S17). That is, in step S17, the controller 10 determines whether or not the point detected by the detection unit 100 is at a position corresponding to the object in the image captured by the imaging unit 300. For example, in step S17, the controller 10 determines whether or not the point detected by the detection unit 100 is located within the area of ​​a loudspeaker in the image displayed on the output unit 40 in FIG. 6.

[0126] For example, if a point detected by the detection unit 100 is not located within the area of ​​a loudspeaker in the image displayed on the output unit 40 in Fig. 6, the point is considered not to be a response based on the detection of a loudspeaker. Therefore, if the determination in step S17 is negative (NO), the controller 10 excludes, ignores, discards, etc. the point, and then proceeds to the next step S19.

[0127] On the other hand, for example, if a point detected by the detection unit 100 is located within the area of ​​a loudspeaker in the image displayed on the output unit 40 in Fig. 6, the point is considered to be a response based on the detection of a loudspeaker. Therefore, if the determination in step S17 is positive (YES), the controller 10 proceeds to the next step S19 without excluding, ignoring, or discarding the point.

[0128] In step S19, the controller 10 determines whether or not the determination in step S17 has been performed for all points in the detection area detected by the detection unit 100. In step S19, if the determination in step S17 has not been performed for all points in the detection area (NO), the controller 10 returns to step S15 and repeats the processes in steps S15 to S18 for the remaining points. On the other hand, in step S19, if the determination in step S17 has been performed for all points in the detection area (YES), the controller 10 may proceed to step S20.

[0129] In step S20, the controller 10 outputs the remaining point cloud (the point cloud that was not discarded in step S18). In this manner, the electronic device 1 can extract and output the detection result at the position corresponding to the predetermined object in the image captured by the imaging unit 300 from the information detected by the detection unit 100.

[0130] Furthermore, in step S20, the controller 10 may continue to detect time changes in the remaining point cloud (point cloud that has not been discarded in step S18) to extract and output the vibration waveform of the object. The controller 10 may display the vibration waveform of the object thus output on, for example, the output unit 40 (display).

[0131] Fig. 11 is a graph showing a result of detecting vibration of a loudspeaker set in an image displayed on the output unit 40 in Fig. 6, for example, as an example of an operation of the electronic device 1 according to an embodiment. The vertical axis of the graph shown in Fig. 11 indicates the amplitude of vibration, and the horizontal axis indicates time.

[0132] As described above, in the electronic device 1, the controller 10 extracts information on a detection area corresponding to a region of a predetermined object in an image of a predetermined imaging area from information on the predetermined detection area (for example, steps S17 and S18 shown in FIG. 5). As information on the predetermined detection area, the controller 10 may obtain, for example, information detected by the detection unit 100 based on transmission of a transmission wave and reception of a reflected wave of the transmission wave. The controller 10 may also obtain an image of the predetermined imaging area from, for example, the imaging unit 300.

[0133] In one embodiment, the controller 10 may extract information of a cloud of points within a detection area corresponding to the area of ​​the specified object from information of a cloud of points detected as information of the specified detection area (for example, steps S17 and S18 shown in FIG. 5).

[0134] In addition, in one embodiment, the controller 10 may extract information on a cloud of points within the detection area corresponding to the area of ​​the specified object from information on a cloud of points detected near a specified distance among information on the specified detection area (for example, step S14 shown in FIG. 5).

[0135] In addition, in one embodiment, the controller 10 may extract information of a detection area corresponding to the area of ​​the specified object specified in the image of the specified imaging area from the information of the specified detection area (for example, step S12 shown in FIG. 5).

[0136] In addition, in one embodiment, the controller 10 may extract information of the point cloud within the detection area corresponding to the area of ​​the specified object from the direction-estimated information of the point cloud detected as information of the specified detection area (for example, step S15 shown in FIG. 5).

[0137] In addition, in one embodiment, the controller 10 may convert the three-dimensional coordinates of the point cloud detected as information of the specified detection area into two-dimensional coordinates, and determine whether or not the two-dimensional coordinates of the point cloud are within the area of ​​the specified object in the image of the specified imaging area (for example, step S16 shown in FIG. 5).

[0138] In one embodiment, the controller 10 may generate information regarding vibration of the predetermined object based on information of a detection area corresponding to the area of ​​the predetermined object (for example, step S20 shown in FIG. 5).

[0139] In addition, in one embodiment, the controller 10 may obtain information on the predetermined detection region from a detection unit 100 that is arranged so that the optical axis faces a predetermined direction. In this case, the controller 10 may obtain an image of the predetermined imaging area from an imaging section 300 that is disposed so that the optical axis faces a direction substantially parallel to the predetermined direction.

[0140] According to the electronic device 1 of the embodiment, for example, when a target object is specified by a user, vibration information corresponding to the target object can be extracted. Therefore, the electronic device 1 of the embodiment can detect the vibration of the target object (for example, even if an object other than the target object is present in the detection area). Therefore, according to the electronic device 1 of the embodiment, not only can the convenience be improved, but the vibration of the target object can be detected with high accuracy.

[0141] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically contradictory, and multiple components or steps can be combined into one or divided. Although the embodiments according to the present disclosure have been described mainly with respect to the device, the embodiments according to the present disclosure can also be realized as a method including steps executed by each component of the device. The embodiments according to the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, executed by a processor included in the device. It should be understood that these are also included in the scope of the present disclosure.

[0142] The above-described embodiment is not limited to implementation as the electronic device 1. For example, the above-described embodiment may be implemented as a system including the electronic device 1. The above-described embodiment may also be implemented as a control method for an apparatus such as the electronic device 1 or a system including such an apparatus. Furthermore, the above-described embodiment may also be implemented as a program executed by an apparatus such as the electronic device 1 or an information processing device (e.g., a computer), or a system, or may be implemented as a storage medium or recording medium on which such a program is recorded. [Explanation of symbols]

[0143] 1 Electronic equipment 3. Detection equipment 5 Holder part 7 Mounting part 10 Controller 20 Memory section 30 Communications Department 40 Output section 100 Detector 101 Radio wave input section 110 Signal processing section 120 Transmitter 121 Signal Generator 122 Synthesizer 123 Phase control section 124 Amplifier 125 Transmitting Antenna 130 Receiving unit 131 Receiving antenna 132 LNA 133 Mixer 134 IF Section 135 AD conversion section 300 Imaging unit 301 Optical input section

Claims

1. Acquire information on a predetermined detection area detected based on transmission of a transmission wave and reception of a reflected wave of the transmission wave; Acquire an image of a predetermined imaging area, An electronic device comprising: a controller that extracts, from the information of the predetermined detection area, information of a detection area corresponding to a region of a predetermined object in an image of the predetermined imaging area.

2. The electronic device according to claim 1 , wherein the controller extracts information of a cloud of points within a detection region corresponding to a region of the predetermined object from information of a cloud of points detected as information of the predetermined detection region.

3. 2 . The electronic device according to claim 1 , wherein the controller extracts information of a cloud of points within the detection area corresponding to a region of the specified object from information of a cloud of points detected within the specified detection area near a specified distance.

4. The electronic device according to claim 1 , wherein the controller extracts information of a detection area corresponding to a region of the predetermined object specified in the image of the predetermined imaging area from the information of the predetermined detection area.

5. The electronic device according to claim 1 , wherein the controller extracts information of the point cloud within the detection area corresponding to the area of ​​the specified object from information of a direction estimated of the point cloud detected as information of the specified detection area.

6. 2. The electronic device of claim 1, wherein the controller converts three-dimensional coordinates of a point cloud detected as information of the specified detection area into two-dimensional coordinates and determines whether the two-dimensional coordinates of the point cloud are within the area of ​​the specified object in the image of the specified imaging area.

7. The electronic device according to claim 1 , wherein the controller generates information about vibration of the predetermined object based on information about a detection area corresponding to an area of ​​the predetermined object.

8. The controller: acquiring information on the predetermined detection area from a detection unit arranged such that an optical axis faces a predetermined direction; The electronic device according to claim 1 , wherein the image of the predetermined imaging area is obtained from an imaging section arranged such that an optical axis of the imaging section is oriented in a direction substantially parallel to the predetermined direction.

9. a detection unit that detects a predetermined detection area based on transmission of a transmission wave and reception of a reflected wave of the transmission wave; An imaging unit that captures an image of a predetermined imaging area; a controller that extracts, from information of the predetermined detection area detected by the detection unit, information of a detection area corresponding to a region of a predetermined object in an image of the predetermined imaging area captured by the imaging unit; A system comprising:

10. The optical axis of the detection unit is arranged to face a predetermined direction, The system according to claim 9 , wherein an optical axis of the imaging unit is arranged to be oriented in a direction substantially parallel to the predetermined direction.

11. A step of acquiring information on a predetermined detection area detected based on transmitting a transmission wave and receiving a reflected wave of the transmission wave; acquiring an image of a predetermined imaging area; extracting information of a detection area corresponding to a region of a predetermined object in an image of the predetermined imaging area from the information of the predetermined detection area; A method for controlling an electronic device, comprising:

12. For electronic devices, A step of acquiring information on a predetermined detection area detected based on transmitting a transmission wave and receiving a reflected wave of the transmission wave; acquiring an image of a predetermined imaging area; extracting information of a detection area corresponding to a region of a predetermined object in an image of the predetermined imaging area from the information of the predetermined detection area; A program to execute.

Images