Electronic apparatus, system, control method, and program

Abstract

An electronic apparatus (1) is provided with a controller (10) which: acquires information relating to a predetermined detection region, the information being detected on the basis of the transmission of a transmitted wave and the reception of a reflected wave resulting from the transmission wave being reflected; acquires an image (40e) of a predetermined imaging region; and extracts, from the information relating to the predetermined detection region, information relating to a detection region corresponding to a region of a predetermined object (40f) in the image (40e) of the predetermined imaging region.

Description

Electronic device, system, control method, and program CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2023-144771, filed on September 6, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to an electronic device, a system, a control method, and a program.

[0003] For example, in fields such as the automobile industry, technologies for measuring the distance between a vehicle and a predetermined object have become increasingly important. In particular, in recent years, various RADAR (Radio Detecting and Ranging) technologies have been researched, which measure the distance between a vehicle and an object by transmitting radio waves such as millimeter waves and receiving the waves reflected by the object, such as an obstacle. The importance of such technologies for measuring distance is expected to increase in the future along with the development of technologies for assisting drivers in driving and technologies related to autonomous driving, which automates driving partially or completely.

[0004] In technologies for detecting objects such as the above-mentioned millimeter wave radar, various attempts have been made to improve the detection accuracy. For example, Patent Document 1 proposes a technology that can improve the detection accuracy of a detection target in a space where radio wave reflection exists.

[0005] Japanese Patent Application Laid-Open No. 2021-196195

[0006] An electronic device according to one embodiment includes a controller that acquires information about a predetermined detection area detected based on the transmission of a transmission wave and the reception of a reflected wave from the transmission wave, acquires an image of a predetermined imaging area, and extracts, from the information about the predetermined detection area, information about a detection area corresponding to a region of a predetermined object in the image of the predetermined imaging area.

[0007] A system according to one embodiment includes a detection unit that detects a predetermined detection area based on the transmission of a transmission wave and the reception of a reflected wave of the transmission wave, an imaging unit that captures an image of the predetermined imaging area, and a controller that extracts, from information about the predetermined detection area detected by the detection unit, information about a detection area corresponding to an area of ​​a predetermined object in the image of the predetermined imaging area captured by the imaging unit.

[0008] A control method according to one embodiment is a method for controlling an electronic device, and includes the steps of: acquiring information about a predetermined detection area that is detected based on transmitting a transmission wave and receiving a reflected wave from the transmission wave; acquiring an image of a predetermined imaging area; and extracting, from the information about the predetermined detection area, information about a detection area that corresponds to a region of a predetermined object in the image of the predetermined imaging area.

[0009] A program according to one embodiment causes an electronic device to perform the following steps: acquiring information about a predetermined detection area detected based on the transmission of a transmission wave and the reception of a reflected wave from the transmission wave; acquiring an image of a predetermined imaging area; and extracting, from the information about the predetermined detection area, information about a detection area corresponding to a region of a predetermined object in the image of the predetermined imaging area.

[0010] 1 is a block diagram schematically illustrating a functional configuration of an electronic device according to an embodiment; FIG. 2 is a block diagram schematically illustrating a functional configuration of a detection unit according to an embodiment; FIG. 3 is a diagram illustrating an example of a transmission signal transmitted by a sensor according to an embodiment; FIG. 4 is a diagram illustrating an example of a configuration of a detection device according to an embodiment; FIG. 5 is a flowchart illustrating the operation of an electronic device according to an embodiment; FIG. 6 is a diagram illustrating an example of a UI of an electronic device according to an embodiment; FIG. 7 is a diagram illustrating an example of processing by an electronic device according to an embodiment; FIG. 8 is a diagram illustrating an example of processing by an electronic device according to an embodiment; FIG. 9 is a diagram illustrating an example of coordinate transformation of an electronic device according to an embodiment; FIG. 10 is a diagram illustrating an example of coordinate transformation of an electronic device according to an embodiment; FIG. 11 is a diagram illustrating an example of processing by an electronic device according to an embodiment;

[0011] It would be extremely beneficial if information about an object could be accurately detected based on the transmission and reception of radio waves using a sensor such as a millimeter-wave radar. An object of the present disclosure is to provide an electronic device, a system, a control method, and a program that can accurately detect information about an object based on the transmission and reception of radio waves. According to one embodiment, it is possible to provide an electronic device, a system, a control method, and a program that can accurately detect information about an object based on the transmission and reception of radio waves.

[0012] In the present disclosure, an "electronic device" may refer to a device powered by electricity. Furthermore, a "system" may include a device powered by electricity. Furthermore, a "user" may refer to a person (typically a human) who uses a system and / or an electronic device according to an embodiment. A user may include a person who benefits from detecting various objects by using a system and / or an electronic device according to an embodiment. The electronic device, method, and program according to the present disclosure may be used to detect people, objects, animals, etc. present in a predetermined space such as a room, a bed, a bathroom, a toilet, a car, a bus, a train, a corridor, or a street.

[0013] An electronic device according to an embodiment described below enables information about objects detected by a detection unit (sensor) based on technology such as millimeter-wave radar to be narrowed down to a specific object specified in a two-dimensional image. When a specific object is specified in an image by a user, the electronic device according to an embodiment can detect the object using a radar sensor. Therefore, the electronic device according to an embodiment can detect, for example, vibrations of a specific object in a non-contact manner. The electronic device according to an embodiment will be described in detail below with reference to the drawings. The sensor used in the present disclosure is not limited to millimeter-wave radar, but may also be a sensor that detects radio wave radar, ultrasound, light, or the like, using a frequency band other than millimeter waves.

[0014] FIG. 1 is a functional block diagram illustrating a schematic configuration of an electronic device 1 according to an embodiment. As illustrated in FIG. 1 , the electronic device 1 according to an embodiment includes a controller 10. In an embodiment, the electronic device 1 may include at least one of, for example, a storage unit 20, a communication unit 30, and an output unit 40, as appropriate. The controller 10, storage unit 20, communication unit 30, and output unit 40 may be located at any location in the electronic device 1 or may be built into the electronic device 1. Furthermore, at least one of the controller 10, storage unit 20, communication unit 30, and output unit 40 may be located outside the electronic device 1 and connected to each other via a network, either wired or wireless, or a combination thereof. The electronic device 1 according to an embodiment may omit at least some of the functional units illustrated in FIG. 1 or may include functional units other than the functional units illustrated in FIG. 1 as appropriate.

[0015] The electronic device 1 according to an embodiment may be various types of devices. For example, the electronic device according to an embodiment may be any device, such as a dedicated terminal, a general-purpose smartphone, a tablet, a phablet, a notebook PC (notebook PC), a computer, or a server. Furthermore, the electronic device according to an embodiment may have a function for communicating with other electronic devices, such as a mobile phone or a smartphone. Here, the "other electronic device" may be, for example, an electronic device such as a mobile phone or a smartphone, or any device, such as a base station, a server, a dedicated terminal, or a computer. The "other electronic device" may be, for example, the detection unit 100 and / or the imaging unit 300 described below. Furthermore, the "other electronic device" in the present disclosure may also be a device or apparatus powered by electricity. When the electronic device according to an embodiment communicates with the other electronic device, the communication may be wired and / or wireless.

[0016] 1 , the electronic device 1 according to an embodiment may be connected to the detection unit 100 by wire and / or wirelessly. Through such a connection, the electronic device 1 according to an embodiment can acquire information detected by the detection unit 100. Furthermore, the electronic device 1 according to an embodiment may be connected to the imaging unit 300 by wire and / or wirelessly. Through such a connection, the electronic device 1 according to an embodiment can acquire an image captured by the imaging unit 300. The detection unit 100 and the imaging unit 300 will be described in further detail below.

[0017] The controller 10 controls and / or manages the entire electronic device 1, including each functional unit constituting the electronic device 1. The controller 10 may also control and / or manage the entire system, including each functional unit constituting the system including the electronic device 1. The controller 10 may include at least one processor, such as a central processing unit (CPU) or a digital signal processor (DSP), to provide control and processing capabilities for executing various functions. The controller 10 may be implemented as a single processor, several processors, or individual processors. The processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (integrated circuit). The processor may be implemented as multiple integrated circuits and discrete circuits connected to each other in a communicative manner. The processor may also be implemented based on various other known technologies.

[0018] In one embodiment, the controller 10 may be configured as, for example, a CPU or a DSP and a program executed by the CPU or DSP. The program executed by the controller 10 and the results of processing executed by the controller 10 may be stored in, for example, the storage unit 20. The controller 10 may include a memory necessary for the operation of the controller 10, as appropriate.

[0019] In the electronic device 1 according to an embodiment, the controller 10 can perform various processes on information output as a result of detection by the detection unit 100, for example. Therefore, in the electronic device 1, the controller 10 may be connected to the detection unit 100 by wire and / or wirelessly. Also, in the electronic device 1 according to an embodiment, the controller 10 can perform various processes on information (images) output as a result of image capture by the imaging unit 300, for example. Therefore, in the electronic device 1, the controller 10 may be connected to the imaging unit 300 by wire and / or wirelessly. The operation of the controller 10 of the electronic device 1 according to an embodiment will be described further below.

[0020] The storage unit 20 may function as a memory for storing various types of information. The storage unit 20 may store, for example, programs executed by the controller 10 and results of processing executed by the controller 10. The storage unit 20 may store or accumulate detection results by the detection unit 100 and / or images captured by the imaging unit 300. The storage unit 20 may also function as a work memory for the controller 10. The storage unit 20 may be configured, for example, using a semiconductor memory or the like, but is not limited thereto and may be any storage device. For example, the storage unit 20 may be a storage medium such as a memory card inserted into the electronic device 1 according to an embodiment. The storage unit 20 may also include, for example, a hard disk drive (HDD) and / or a solid state drive (SSD). The storage unit 20 may be an internal memory of a CPU used as the controller 10 (described below), or may be a separate device connected to the controller 10.

[0021] The communication unit 30 has an interface function for wired or wireless communication. In one embodiment, the communication method performed by the communication unit 30 may be a wireless communication standard. For example, the wireless communication standard includes cellular phone communication standards such as 2G, 3G, 4G, and 5G. For example, the cellular phone communication standard includes Long Term Evolution (LTE), Wideband Code Division Multiple Access (W-CDMA), CDMA2000, Personal Digital Cellular (PDC), Global System for Mobile communications (GSM (registered trademark)), and Personal Handy-phone System (PHS). For example, the wireless communication standard includes Worldwide Interoperability for Microwave Access (WiMAX), IEEE 802.11, Wi-Fi, Bluetooth (registered trademark), Infrared Data Association (IrDA), and Near Field Communication (NFC). The communication unit 30 may include, for example, a modem whose communication method is standardized by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). The communication unit 30 can support one or more of the above communication standards.

[0022] The communication unit 30 may be configured to include, for example, an antenna for transmitting and receiving radio waves and an appropriate RF unit. The communication unit 30 may communicate wirelessly with, for example, a communication unit of another electronic device via an antenna. The communication unit 30 may also be configured as an interface such as a connector for wired connection to an external device. The communication unit 30 can be configured using known technology for wireless communication, and therefore a more detailed description of the hardware and the like will be omitted.

[0023] The various types of information received by the communication unit 30 may be supplied to, for example, the storage unit 20 and / or the controller 10. The various types of information received by the communication unit 30 may be stored in, for example, a memory built into the controller 10. Furthermore, the communication unit 30 may transmit, for example, processing results by the controller 10 and / or information stored in the storage unit 20 to the outside.

[0024] The output unit 40 may be any display device, such as a liquid crystal display (LCD), an organic electroluminescence (EL) display (organic electroluminescence) panel, or an inorganic electroluminescence (EL) display. The output unit 40 may display various types of information, such as characters, figures, or symbols. The output unit 40 may also display various objects and icon images constituting a GUI, for example, to prompt a user to operate the electronic device 1. Various data required for display on the output unit 40 may be supplied from, for example, the controller 10 or the storage unit 20. If the output unit 40 includes, for example, an LCD, it may also be configured to include a backlight, as appropriate. In one embodiment, the output unit 40 may display information based on the results of detection by the detection unit 100. In another embodiment, the output unit 40 may display information based on the results of imaging by the imaging unit 300, for example, an image or video (including still images and videos).

[0025] Furthermore, the output unit 40 may issue a predetermined warning to alert a user of the electronic device 1, based on a predetermined signal output from the controller 10. The output unit 40 may include any functional unit that stimulates at least one of the user's hearing, vision, and touch senses, such as sound, voice, light, text, video, and vibration, as the predetermined warning. Specifically, the output unit 40 may include at least one of an audio output unit such as a buzzer or speaker, a light-emitting unit such as an LED, a display unit such as an LCD, and a tactile sensation providing unit such as a vibrator. In this way, the output unit 40 may issue a predetermined warning based on a predetermined signal output from the controller 10. In one embodiment, the output unit 40 may issue the predetermined warning as information that stimulates at least one of the hearing, vision, and touch senses.

[0026] 1 includes a built-in output unit 40. However, in one embodiment, the output unit 40 may be provided outside the electronic device 1. In this case, the output unit 40 and the electronic device 1 may be connected by wire, wirelessly, or a combination of wire and wirelessly.

[0027] At least some of the functional units constituting the electronic device 1 according to one embodiment as shown in FIG. 1 may be configured by specific means in which software and hardware resources work together.

[0028] 1 is configured to detect an object (target) such as a vehicle, a human body, a building, or a structure as point cloud information in a three-dimensional space. The detection unit 100 according to one embodiment will be described in more detail below.

[0029] FIG. 2 is a functional block diagram illustrating a schematic configuration of a detection unit 100 according to an embodiment. The detection unit 100 illustrated in FIG. 2 is, as an example, a millimeter-wave radar sensor based on millimeter-wave radar (RADAR (Radio Detecting and Ranging)) technology. However, the detection unit 100 according to an embodiment is not limited to a millimeter-wave radar sensor. The detection unit 100 according to an embodiment may be, for example, a quasi-millimeter-wave radar sensor. Furthermore, the detection unit 100 according to an embodiment is not limited to a millimeter-wave radar sensor or a quasi-millimeter-wave radar sensor, and may be any type of radar sensor that transmits and receives radio waves. Furthermore, the detection unit 100 according to an embodiment may be, for example, a microwave sensor, an ultrasonic sensor, or a sensor based on technology such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0030] When measuring the distance to an object using a millimeter-wave radar, a frequency-modulated continuous wave radar (hereinafter referred to as FMCW radar) is sometimes used. FMCW radar generates a transmission signal by sweeping the frequency of the radio waves it transmits. Therefore, in a millimeter-wave FMCW radar that uses radio waves in the 79 GHz frequency band, for example, the frequency of the radio waves used has a frequency bandwidth of 4 GHz, such as 77 GHz to 81 GHz. The millimeter-wave radar of the present disclosure may also use radio waves in a frequency band other than 77 GHz to 81 GHz. Radar using the 79 GHz frequency band has a feature that it has a wider usable frequency bandwidth than other millimeter-wave / quasi-millimeter-wave radars, such as those using the 24 GHz, 60 GHz, and 76 GHz frequency bands. Below, we will explain an example of using such an FMCW radar. The FMCW radar system used in the present disclosure may include the FCM (Fast-Chirp Modulation) system, which transmits chirp signals at a shorter period than normal. The signal generated by the detection unit 100 is not limited to an FMCW signal. The signal generated by the detection unit 100 may be a signal of various systems other than the FMCW system. The transmission signal sequence stored as the signal to be transmitted may differ depending on these various systems. For example, in the case of the above-mentioned FMCW radar signal, a signal whose frequency increases and decreases with each time sample may be used. Since known technologies can be applied as appropriate to the above-mentioned various systems, further detailed description will be omitted as appropriate.

[0031] As shown in Fig. 2, the detection unit 100 according to an embodiment may be configured to include a signal processing unit 110, a transmission unit 120, and a reception unit 130. The above-described signal processing unit 110, transmission unit 120, reception unit 130, etc. may be disposed at any location in the detection unit 100 or may be built-in. Furthermore, at least one of the above-described signal processing unit 110, transmission unit 120, reception unit 130, etc. may be disposed outside the detection unit 100. The detection unit 100 according to an embodiment may omit at least some of the functional units shown in Fig. 2, or may include other functional units as appropriate in addition to the functional units shown in Fig. 2.

[0032] The signal processing unit 110 controls and / or manages the entire detection unit 100, including each functional unit constituting the detection unit 100. The signal processing unit 110 may include at least one processor, such as a central processing unit (CPU) or a digital signal processor (DSP), to provide control and processing power for executing various functions. The signal processing unit 110 may be implemented as a single processor, several processors, or individual processors. The processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (integrated circuit). The processor may be implemented as multiple communicatively connected integrated circuits and discrete circuits. The processor may also be implemented based on various other known technologies.

[0033] In one embodiment, the signal processing unit 110 may be configured as, for example, a CPU or a DSP and a program executed by the CPU or DSP. The program executed in the signal processing unit 110 and the results of the processing executed in the signal processing unit 110 may be stored in, for example, an arbitrary storage unit built into the signal processing unit 110. The signal processing unit 110 may include a memory necessary for the operation of the signal processing unit 110, as appropriate.

[0034] In the detection unit 100 according to one embodiment, the signal processing unit 110 may appropriately perform various types of processing, such as distance FFT (Fast Fourier Transform) processing, velocity FFT processing, arrival angle estimation processing, clustering processing, etc. These types of processing performed by the signal processing unit 110 are known as general radar techniques, and therefore detailed descriptions thereof will be omitted as appropriate.

[0035] 2 , the transmitting unit 120 may include a signal generating unit 121, a synthesizer 122, a phase control unit 123, an amplifier 124, and a transmitting antenna 125. The detecting unit 100 according to an embodiment may include a plurality of transmitting antennas 125. In this case, the detecting unit 100 may also include a plurality of phase control units 123 and amplifiers 124 corresponding to the plurality of transmitting antennas 125, respectively. When the detecting unit 100 according to an embodiment includes a plurality of transmitting antennas 125, the plurality of transmitting antennas 125 may form a transmitting antenna array (transmitting array antenna).

[0036] 2 , the receiving unit 130 may include a receiving antenna 131, an LNA 132, a mixer 133, an IF unit 134, and an AD conversion unit 135. The detecting unit 100 according to one embodiment may include a plurality of receiving units 130 corresponding to the plurality of transmitting antennas 125, respectively.

[0037] In the detection unit 100 according to an embodiment, the signal processing unit 110 can control at least one of the transmission unit 120 and the reception unit 130. In this case, the signal processing unit 110 may control at least one of the transmission unit 120 and the reception unit 130 based on various information stored in an arbitrary storage unit. For example, an arbitrary storage unit built into the signal processing unit 110 may store various parameters for setting a range (area) in which an object is detected by a transmission wave transmitted from the transmission antenna 125 and a reflected wave received from the reception antenna 131. Furthermore, in the detection unit 100 according to an embodiment, the signal processing unit 110 may instruct the signal generation unit 121 to generate a signal or control the signal generation unit 121 to generate a signal.

[0038] The signal generating unit 121 generates a signal (transmission signal) to be transmitted as a transmission wave from the transmitting antenna 125 under the control of the signal processing unit 110. When generating the transmission signal, the signal generating unit 121 may assign a frequency to the transmission signal, for example, based on the control of the signal processing unit 110. Specifically, the signal generating unit 121 may assign a frequency to the transmission signal, for example, in accordance with parameters set by the signal processing unit 110. For example, the signal generating unit 121 receives frequency information from the signal processing unit 110 or an arbitrary storage unit, and generates a signal of a predetermined frequency in a frequency band such as 77 to 81 GHz. The signal generating unit 121 may be configured to include a functional unit such as a voltage-controlled oscillator (VCO).

[0039] The signal generating unit 121 may be configured as hardware having the function, or may be configured as a microcomputer, or may be configured as a processor such as a CPU or DSP and a program executed by the processor, etc. Each functional unit described below may also be configured as hardware having the function, or may be configured as a microcomputer, if possible, or may be configured as a processor such as a CPU or DSP and a program executed by the processor, etc.

[0040] In the detection unit 100 according to an embodiment, the signal generation unit 121 may generate a transmission signal (transmission chirp signal), such as a chirp signal. In particular, the signal generation unit 121 may generate a signal whose frequency periodically changes linearly (linear chirp signal). For example, the signal generation unit 121 may generate a chirp signal whose frequency periodically increases linearly from 77 GHz to 81 GHz over time. Alternatively, the signal generation unit 121 may generate a signal whose frequency periodically repeats a linear increase (up chirp) and decrease (down chirp) from 77 GHz to 81 GHz over time. The signal generated by the signal generation unit 121 may be preset in, for example, the signal processing unit 110. The signal generated by the signal generation unit 121 may also be stored in advance in, for example, an arbitrary storage unit. Chirp signals used in technical fields such as radar are well known, and therefore further detailed description thereof will be appropriately simplified or omitted. The signal generated by the signal generating unit 121 is supplied to the synthesizer 122 .

[0041] FIG. 3 is a diagram illustrating an example of a chirp signal generated by the signal generating unit 121. In FIG.

[0042] In Fig. 3, the horizontal axis represents elapsed time, and the vertical axis represents frequency. In the example shown in Fig. 3, the signal generator 121 generates linear chirp signals whose frequencies change periodically and linearly. In Fig. 3, each chirp signal is denoted as c1, c2, ..., c8. As shown in Fig. 3, the frequency of each chirp signal increases linearly over time.

[0043] In the example shown in FIG. 3 , one subframe includes eight chirp signals, such as c1, c2, ..., c8. That is, subframe 1 and subframe 2 shown in FIG. 3 each include eight chirp signals, such as c1, c2, ..., c8. Also, in the example shown in FIG. 3 , one frame includes 16 subframes, such as subframe 1 to subframe 16. That is, frame 1 and frame 2 shown in FIG. 3 each include 16 subframes. Also, as shown in FIG. 3 , a frame interval of a predetermined length may be included between frames. One frame shown in FIG. 3 may be, for example, 30 to 50 milliseconds long.

[0044] In Fig. 3, the same configuration may be used for frames 2 and onward. Also, in Fig. 3, the same configuration may be used for frames 3 and onward. In the detection unit 100 according to one embodiment, the signal generation unit 121 may generate a transmission signal as any number of frames. Also, in Fig. 3, some chirp signals are omitted from the illustration. In this way, the relationship between time and frequency of the transmission signal generated by the signal generation unit 121 may be stored in, for example, any storage unit.

[0045] In this way, the detector 100 according to an embodiment may transmit a transmission signal consisting of subframes each including a plurality of chirp signals. Also, the detector 100 according to an embodiment may transmit a transmission signal consisting of a frame each including a predetermined number of subframes.

[0046] In the following description, the detector 100 is assumed to transmit a transmission signal having a frame structure as shown in FIG. 3 . However, the frame structure as shown in FIG. 3 is merely an example, and the number of chirp signals included in one subframe is not limited to eight. In one embodiment, the signal generator 121 may generate subframes including any number of chirp signals (e.g., any plural number). The subframe structure as shown in FIG. 3 is also merely an example, and the number of subframes included in one frame is not limited to 16. In one embodiment, the signal generator 121 may generate frames including any number of subframes (e.g., any plural number). The signal generator 121 may generate signals of different frequencies. The signal generator 121 may generate multiple discrete signals, each having a frequency f with a different bandwidth.

[0047] Returning to FIG. 2 , the synthesizer 122 increases the frequency of the signal generated by the signal generating unit 121 to a frequency in a predetermined frequency band. The synthesizer 122 may increase the frequency of the signal generated by the signal generating unit 121 to a frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125. The frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be set by, for example, the signal processing unit 110. Furthermore, the frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be stored in, for example, an arbitrary storage unit. The signal whose frequency has been increased by the synthesizer 122 is supplied to the phase control unit 123 and the mixer 133. When there are multiple phase control units 123, the signal whose frequency has been increased by the synthesizer 122 may be supplied to each of the multiple phase control units 123. Furthermore, if there are multiple receiving sections 130 , the signal whose frequency has been increased by the synthesizer 122 may be supplied to each mixer 133 in the multiple receiving sections 130 .

[0048] The phase control unit 123 controls the phase of the transmission signal supplied from the synthesizer 122. Specifically, the phase control unit 123 may adjust the phase of the transmission signal by appropriately advancing or delaying the phase of the signal supplied from the synthesizer 122, for example, based on control by the signal processing unit 110. In this case, the phase control unit 123 may adjust the phase of each transmission signal based on the path difference of each transmission wave transmitted from the multiple transmission antennas 125. By the phase control unit 123 appropriately adjusting the phase of each transmission signal, the transmission waves transmitted from the multiple transmission antennas 125 constructively interact with each other in a predetermined direction to form a beam (beamforming). In this case, the correlation between the beamforming direction and the phase amount to be controlled of the transmission signal transmitted from each of the multiple transmission antennas 125 may be stored in, for example, an arbitrary storage unit. The transmission signal phase-controlled by the phase control unit 123 is supplied to the amplifier 124.

[0049] The amplifier 124 amplifies the power of the transmission signal supplied from the phase control section 123, for example, based on control by the signal processing section 110. If the detection section 100 includes a plurality of transmission antennas 125, the plurality of amplifiers 124 may each amplify the power of the transmission signal supplied from a corresponding one of the plurality of phase control sections 123, for example, based on control by the signal processing section 110. Since the technology itself for amplifying the power of the transmission signal is already known, a detailed description thereof will be omitted. The amplifier 124 is connected to the transmission antenna 125.

[0050] The transmitting antenna 125 outputs (transmits) as a transmission wave the transmission signal amplified by the amplifier 124. When the detection unit 100 includes a plurality of transmitting antennas 125, the plurality of transmitting antennas 125 may output (transmit) as a transmission wave the transmission signal amplified by a corresponding one of the plurality of amplifiers 124. The transmitting antenna 125 can be configured in the same manner as a transmitting antenna used in known radar technology, and therefore a detailed description thereof will be omitted.

[0051] In this way, the detection unit 100 according to one embodiment includes a transmitting antenna 125, and can transmit a transmission signal (e.g., a transmission chirp signal) as a transmission wave from the transmitting antenna 125. Here, at least one of the functional units constituting the detection unit 100 may be housed in a single housing. In this case, the single housing may have a structure that makes it difficult to open. For example, the transmitting antenna 125, the receiving antenna 131, and the amplifier 124 may be housed in a single housing, and this housing may have a structure that makes it difficult to open. Furthermore,

[0052] The detecting unit 100 shown in FIG. 2 is an example including one transmitting antenna 125. However, in one embodiment, the detecting unit 100 may include any number of transmitting antennas 125. On the other hand, in one embodiment, the detecting unit 100 may include multiple transmitting antennas 125 when the transmission waves transmitted from the transmitting antennas 125 form beams in a predetermined direction. In one embodiment, the detecting unit 100 may include any number of transmitting antennas 125. In this case, the detecting unit 100 may also include multiple phase control units 123 and amplifiers 124 corresponding to the multiple transmitting antennas 125. The multiple phase control units 123 may each control the phases of the multiple transmission waves supplied from the synthesizer 122 and transmitted from the multiple transmitting antennas 125. The multiple amplifiers 124 may each amplify the power of the multiple transmission signals transmitted from the multiple transmitting antennas 125. In this case, the detecting unit 100 may be configured to include multiple transmitting antennas. In this way, when the detection unit 100 includes a plurality of transmission antennas 125, the detection unit 100 may also be configured to include a plurality of functional units required for transmitting transmission waves from the plurality of transmission antennas 125.

[0053] The receiving antenna 131 receives the reflected wave. The reflected wave may be a transmitted wave reflected by a predetermined object 200. The receiving antenna 131 may be configured to include multiple antennas. When the detection unit 100 according to an embodiment includes multiple receiving antennas 131, the multiple receiving antennas 131 may form a receiving antenna array (receiving array antenna). The receiving antenna 131 may be configured in the same manner as a receiving antenna used in known radar technology, and therefore a detailed description thereof will be omitted. The receiving antenna 131 is connected to the LNA 132. A received signal based on the reflected wave received by the receiving antenna 131 is supplied to the LNA 132. In this way, when the detection unit 100 includes multiple receiving antennas 131, the detection unit 100 may also be configured to include multiple functional units required to receive and process reflected waves from the multiple receiving antennas 131.

[0054] The detection unit 100 according to one embodiment can receive reflected waves that are formed by transmitting transmission signals (transmission chirp signals), such as chirp signals, from a plurality of reception antennas 131 and reflecting the transmitted waves from a predetermined object 200. When transmission chirp signals are transmitted as transmission waves, reception signals based on the received reflected waves are also referred to as reception chirp signals. That is, the detection unit 100 receives reception signals (e.g., reception chirp signals) as reflected waves from the reception antennas 131.

[0055] The LNA 132 amplifies, with low noise, the received signal based on the reflected wave received by the receiving antenna 131. The LNA 132 may be a low noise amplifier, and amplifies, with low noise, the received signal supplied from the receiving antenna 131. The received signal amplified by the LNA 132 is supplied to the mixer 133.

[0056] The mixer 133 generates a beat signal by mixing (multiplying) the RF frequency reception signal supplied from the LNA 132 with the transmission signal supplied from the synthesizer 122. The beat signal mixed by the mixer 133 is supplied to the IF unit 134.

[0057] The IF unit 134 performs frequency conversion on the beat signal supplied from the mixer 133, thereby lowering the frequency of the beat signal to an intermediate frequency (IF). The beat signal whose frequency has been lowered by the IF unit 134 is supplied to the AD conversion unit 135.

[0058] The AD conversion unit 135 digitizes the analog beat signal supplied from the IF unit 134. The AD conversion unit 135 may be configured with any analog-to-digital conversion circuit (Analog to Digital Converter (ADC)). The beat signal digitized by the AD conversion unit 135 is supplied to the signal processing unit 110. When there are multiple receiving units 130, the beat signals digitized by the multiple AD conversion units 135 may each be supplied to the signal processing unit 110.

[0059] The signal processing unit 110 may perform FFT processing (hereinafter referred to as "distance FFT processing") on the beat signal digitized by the AD conversion unit 135. For example, the signal processing unit 110 may perform FFT processing on the complex signal supplied from the AD conversion unit 135. The beat signal digitized by the AD conversion unit 135 can be expressed as a time change in signal strength (power). By performing FFT processing on such a beat signal, the signal processing unit 110 can express it as a signal strength (power) corresponding to each frequency. By performing distance FFT processing in the signal processing unit 110, a complex signal corresponding to distance can be obtained based on the beat signal digitized by the AD conversion unit 135.

[0060] When a peak in the result obtained by the distance FFT processing is equal to or greater than a predetermined threshold, the signal processing unit 110 may determine that a predetermined object 200 is present at the distance corresponding to the peak. For example, a method is known in which, when a peak value equal to or greater than a threshold is detected from the average power or amplitude of a disturbance signal, an object that reflects a transmitted wave (a reflecting object) is determined to be present, such as in detection processing based on a constant false alarm rate (CFAR).

[0061] In this way, the detection unit 100 of one embodiment can detect an object 200 that reflects a transmission wave as a target based on a transmission signal transmitted as a transmission wave and a received signal received as a reflected wave.

[0062] The signal processing unit 110 can estimate the distance to a predetermined object based on one chirp signal (e.g., c1 shown in FIG. 3 ). That is, the detection unit 100 can measure (estimate) the distance between the detection unit 100 and a predetermined object 200 by performing distance FFT processing. The technology for measuring (estimating) the distance to a predetermined object by performing FFT processing on a beat signal is well known, so a more detailed description thereof will be appropriately simplified or omitted.

[0063] Furthermore, the signal processing unit 110 may further perform FFT processing (hereinafter referred to as "velocity FFT processing") on the beat signal that has undergone distance FFT processing. For example, the signal processing unit 110 may perform FFT processing on the complex signal that has undergone distance FFT processing. The signal processing unit 110 can estimate the relative velocity with respect to a predetermined object based on a subframe of the chirp signal (e.g., subframe 1 shown in FIG. 3). By performing velocity FFT processing on multiple chirp signals in the signal processing unit 110, a complex signal corresponding to the relative velocity can be obtained based on the complex signal corresponding to the distance obtained by the distance FFT processing.

[0064] As described above, performing distance FFT processing on the beat signal can generate multiple vectors. By determining the phase of the peak in the result of performing velocity FFT processing on these multiple vectors, the relative velocity with respect to a predetermined object can be estimated. That is, by performing velocity FFT processing, the electronic device 1 can measure (estimate) the relative velocity between the detection unit 100 and the predetermined object 200. In this manner, the signal processing unit 110 may perform a two-dimensional fast Fourier transform (2D-FFT) in the range direction and the velocity direction based on the beat signal. Because the technology for measuring (estimating) the relative velocity with respect to a predetermined object by performing velocity FFT processing on the result of distance FFT processing is well known, a detailed description thereof will be simplified or omitted as appropriate.

[0065] In general FMCW radar technology, the presence or absence of a target can be determined based on the results of extracting beat frequencies from a received signal and performing a fast Fourier transform on the extracted beat frequencies. However, the results of extracting beat frequencies from a received signal and performing a fast Fourier transform on the extracted beat frequencies also contain noise components due to clutter (unwanted reflection components). Therefore, a process may be performed to remove the noise components from the processed received signal and extract only the target signal.

[0066] Furthermore, the signal processing unit 110 may estimate (DOA estimation) the direction in which a reflected wave arrives (angle of arrival) from a predetermined object 200 based on the determination of whether or not a target is present. The signal processing unit 110 may perform DOA estimation for a point where it is determined that a target is present. The detection unit 100 can estimate the DOA of the reflected wave by receiving reflected waves from multiple receiving antennas 131. For example, the multiple receiving antennas 131 are assumed to be arranged at a predetermined interval. In this case, a transmission wave transmitted from the transmitting antenna 125 is reflected by the predetermined object 200 and becomes a reflected wave, and the multiple receiving antennas 131 arranged at a predetermined interval each receive the reflected wave R. The signal processing unit 110 can estimate the DOA of the reflected wave at the receiving antenna 131 based on the phase of the reflected wave received by each of the multiple receiving antennas 131 and the path difference between the reflected waves. That is, the detection unit 100 can measure (estimate) the angle of arrival θ, which indicates the direction from which the reflected wave reflected by the target arrives, based on the result of the velocity FFT processing.

[0067] Various techniques have been proposed for estimating the direction of arrival of a reflected wave R based on the results of velocity FFT processing. For example, known algorithms for estimating direction of arrival include MUSIC (MUltiple SIgnal Classification) and ESPRIT (Estimation of Signal Parameters via Rotational Invariance Technique). Therefore, detailed descriptions of known techniques will be appropriately simplified or omitted.

[0068] The signal processing unit 110 detects an object present in the area where the transmission wave is transmitted based on at least one of distance FFT processing, velocity FFT processing, and arrival angle estimation. The signal processing unit 110 may perform object detection by, for example, performing clustering processing based on the supplied distance information, velocity information, and angle information. Known algorithms used for clustering data include DBSCAN (Density-based spatial clustering of applications with noise). In the clustering processing, for example, the average power of points constituting the detected object may be calculated.

[0069] As described above, the detection unit 100 can detect an object that reflects a transmission wave in three-dimensional space as point cloud information. That is, in one embodiment, based on the detection result output from the detection unit 100, it can determine (detect) whether or not an object that reflects a transmission wave exists at a certain coordinate in three-dimensional space. Furthermore, in one embodiment, the detection unit 100 can detect the signal strength and velocity of each point in three-dimensional space. As described above, the detection unit 100 according to one embodiment may detect an object that reflects a transmission wave as point cloud information in three-dimensional space based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave.

[0070] 1, the electronic device 1 according to an embodiment may include an imaging unit 300. The electronic device 1 and the imaging unit 300 may be connected to each other by wire, wirelessly, or by a combination of wire and wirelessly.

[0071] The imaging unit 300 may be configured to include an image sensor that electronically captures an image, such as a digital camera. The imaging unit 300 may be configured to include an imaging element that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 300 may capture an image of an object present in front of the imaging unit 300. Here, the object present in front of the imaging unit 300 may be, for example, a car, a person, and / or any object present in the surrounding area. The imaging unit 300 may convert the captured image into a signal and transmit it to the electronic device 1. For example, the imaging unit 300 may transmit a signal based on the captured image to the storage unit 20, communication unit 30, output unit 40, and / or controller 10 of the electronic device 1. The imaging unit 300 is not limited to an imaging device such as a digital camera, and may be any device that can capture an image of an object. The imaging unit 300 may be, for example, a light detection and ranging (LIDAR) device.

[0072] In one embodiment, the imaging unit 300 may capture still images at predetermined time intervals (for example, 15 frames per second). Also, in one embodiment, the imaging unit 300 may capture continuous moving images, for example.

[0073] Next, the arrangement of the detection unit 100 and the imaging unit 300 connected to the electronic device 1 according to the embodiment will be described.

[0074] 4A and 4B are diagrams showing an example of the configuration of a detection device in which the detection unit 100 and the imaging unit 300 are arranged.

[0075] 4A is a front view showing an example of the detector 3 according to one embodiment as viewed from the front. FIG. 4B is a side view showing an example of the detector 3 according to one embodiment as viewed from the side (left). The coordinate axes shown in FIG. 4A and FIG. 4B are aligned with the coordinate axes showing the propagation directions of the transmitted waves and / or reflected waves of the detector 100 shown in FIG.

[0076] 4A and 4B, the detection device 3 according to an embodiment may include a detection unit 100 and an imaging unit 300. Also, as shown in FIGS. 4A and 4B, the detection device 3 according to an embodiment may include at least one of a holder unit 5 and an attachment unit 7, as appropriate. Also, the detection device 3 according to an embodiment may be disposed in any other device or the housing of another device, without including at least one of the holder unit 5 and the attachment unit 7.

[0077] The detection unit 100 shown in FIGS. 4A and 4B may be the detection unit 100 described in FIGS. 1 and / or 2. As shown in FIGS. 4A and 4B, the detection unit 100 may include a radio wave input unit 101 that receives a reflected wave of a transmitted wave reflected by an object. In the detection unit 100, at least one of a transmitting antenna 125 and a receiving antenna 131 may be disposed at a position corresponding to the radio wave input unit 101. As shown in FIG. 4B, the radio wave input unit 101 may be oriented along the optical axis Ra of the detection unit 100. Here, the optical axis of the detection unit 100 may be, for example, a direction perpendicular to a surface on which at least one of the transmitting antennas 125 and the receiving antennas 131 of the detection unit 100 is installed. Furthermore, when the detection unit 100 includes a plurality of at least one of the transmitting antennas 125 and the receiving antennas 131, the optical axis of the detection unit 100 may be a direction perpendicular to a surface on which at least one of the plurality of antennas is installed. With this configuration, the detection unit 100 can transmit a transmission wave and / or receive a reflected wave around the optical axis Ra. That is, the detection unit 100 can detect an object as a point cloud in an area centered around the optical axis Ra.

[0078] The detection unit 100 according to an embodiment may be directional. That is, the detection unit 100 may detect an object using directional radio waves. Here, directivity may be a characteristic of an antenna, i.e., the relationship between the radiation direction and radiation intensity of radio waves. The presence or absence of directivity is related to the application of the antenna. A highly directional antenna strongly radiates radio waves in a specific direction. The directivity may be the same for both transmission and reception. The electric field intensity of radio waves radiated by an antenna can be expressed in decibels (dB) as the antenna gain. The detection unit 100 according to an embodiment may have directivity, and thus may have a main lobe (main beam) in the direction of the optical axis Ra shown in FIG. 4B , for example. That is, the detection unit 100 according to an embodiment may have the strongest radiation level in the direction of the optical axis Ra, for example.

[0079] 4A and 4B may be the same as the imaging unit 300 described in FIG. 1. As shown in FIGS. 4A and 4B, the imaging unit 300 may include a light input unit 301 that receives light reflected by an object. In the imaging unit 300, an image sensor such as a light receiving element or an imaging element may be disposed at a position corresponding to the light input unit 301. As shown in FIG. 4B, the light input unit 301 may be oriented along the optical axis La of the imaging unit 300. Alternatively, the light input unit 301 may be disposed at a position where a lens is disposed in the imaging unit 300. Here, the optical axis of the imaging unit 300 may be, for example, a direction perpendicular to a surface on which a light receiving element (or an imaging element) used for imaging in the imaging unit 300 is disposed. With this configuration, the imaging unit 300 can capture an image centered on the optical axis La.

[0080] 4A and 4B , the holder unit 5 maintains the detection unit 100 and the imaging unit 300 in a fixed state in the detection device 3. The holder unit 5 may maintain the detection unit 100 at a height that makes it easy for the detection unit 100 to detect a predetermined detection area. The holder unit 5 may also maintain the detection unit 100 at a height that makes it easy for the imaging unit 300 to acquire an image of a predetermined imaging area. The detection unit 100 and the imaging unit 300 may be fastened to the holder unit 5 with screws, or may be fixed to the holder unit 5 with an adhesive, or may be fixed by being fitted into the holder unit 5. The holder unit 5 may include a mechanism that can adjust the optical axes of the detection unit 100 and the imaging unit 300 in the detection device 3.

[0081] 4A and 4B , the mounting unit 7 enables the detection device 3, in which the detection unit 100 and the imaging unit 300 are installed, to be attached and fixed to, for example, another member or another device. The mounting unit 7 may be fastened to, for example, the other member or another device by screws, or may be fixed with an adhesive or may be fixed by being fitted in. The mounting unit 7 can have various configurations, such as a pedestal shape, in order to stabilize the detection device 3 including the detection unit 100 and the imaging unit 300.

[0082] The holder 5 and the mounting part 7 may be made of a metal such as an aluminum plate, or may be made of a material such as an acrylic plate. The holder 5 and the mounting part 7 may be made of an aluminum plate or an acrylic plate with a thickness of about 5 mm, for example.

[0083] 4A and 4B, in the detection device 3 according to one embodiment, the detection unit 100 and the imaging unit 300 may be disposed adjacent to each other in the vicinity thereof. In the example shown in FIGS. 4A and 4B, the detection unit 100 and the imaging unit 300 are disposed adjacent to each other in the vertical direction (Z-axis direction). In the detection device 3 according to one embodiment, the detection unit 100 and the imaging unit 300 may be disposed adjacent to each other in the horizontal direction or diagonally.

[0084] 4B , in the detection device 3 according to an embodiment, the detection unit 100 and the image capture unit 300 may be arranged so that their optical axes Ra and La are parallel (or nearly parallel). That is, in the electronic device 1 according to an embodiment, the point cloud information from the detection unit 100 and the image from the image capture unit 300 may be used in a state where the optical axis La of the image capture unit 300 is installed so that it is parallel (or nearly parallel) to the optical axis Ra of the detection unit 100.

[0085] 4B , in the detection device 3 according to one embodiment, the detection unit 100 and the image capture unit 300 may be disposed so that the distance between their respective optical axes Ra and La is maintained at a distance G. By disposing them in this manner, the point cloud information from the detection unit 100 and the image information from the image capture unit 300 are shifted from each other by the distance G. Therefore, the detection unit 100 and the image capture unit 300 may be disposed so that the distance G between them is relatively small, for example, 2 to 3 cm.

[0086] In one embodiment, the controller 10 may correct at least one of the point cloud information obtained by the detection unit 100 detecting an object (target) and the image obtained by the imaging unit 300 capturing an image of the object (target). In one embodiment, the controller 10 may use information obtained by aligning the point cloud information from the detection unit 100 and the image from the imaging unit 300 through correction.

[0087] As described above, the electronic device 1 according to one embodiment can utilize an image obtained by the imaging unit 300 capturing an image of an object and point cloud information obtained by the detection unit 100 detecting the object.

[0088] Next, the operation of the electronic device 1 according to the embodiment will be described.

[0089] The detection unit 100, such as a radar sensor, can detect a predetermined detection area based on transmitting a transmission wave and receiving a reflected wave of the transmission wave. More specifically, the detection unit 100 can detect (or estimate) the position, speed, and / or angle of arrival of the received reflected wave of a predetermined object by receiving a reflected wave of the transmission wave reflected by the object.

[0090] Furthermore, the detection unit 100 can detect the vibration of an object (such as the presence or absence of vibration or the degree of vibration) by continuously detecting the position and / or velocity of the object for a predetermined period of time. The detection unit 100 can detect the vibration of an object non-contact, for example, using millimeter waves. For example, the detection unit 100 can detect the vibration of various facilities, such as buildings or structures. In recent years, extending the lifespan and strengthening infrastructure has become an important social issue. Vibrations of infrastructure and the like are important physical information for determining the soundness of such infrastructure and the like. For this reason, there is a demand for a means of efficiently inspecting various buildings or structures (e.g., buildings, bridges, tunnels, etc.) including various infrastructure facilities. For example, a sensing system that can measure the vibration of an object non-contact using a radar sensor such as a millimeter-wave radar would be promising as a means of efficiently inspecting infrastructure and the like.

[0091] However, technologies such as millimeter-wave radar detect responses from objects present within a predetermined detection area based on the transmission of transmitted waves and the reception of reflected waves. For example, when a millimeter-wave radar is used to detect vibrations from a building within a predetermined detection area, it is assumed that a vehicle, for example, is also present within the detection area. In such a case, the millimeter-wave radar will detect not only the vibrations of the building but also the vibrations of the vehicle. Therefore, with such detection, it may be difficult to accurately detect the vibrations of only the building.

[0092] For example, in the above-described example, if a user specifies a building, vibration information corresponding to the building is extracted and vibrations of the building are detected, not only is convenience high but vibrations of the building can be detected with high accuracy. Therefore, the electronic device 1 according to an embodiment extracts vibration information corresponding to an object, for example, when the user specifies the object. Therefore, the electronic device 1 according to an embodiment can detect vibrations of the object (even if an object other than the object is present in the detection area). Therefore, the electronic device 1 according to an embodiment can not only improve convenience but also accurately detect vibrations of the object. Such an embodiment will be described below.

[0093] Fig. 5 is a flowchart showing an example of the operation of the electronic device 1 according to an embodiment. The operation shown in Fig. 5 may be started, for example, when a user of the electronic device 1 according to an embodiment detects vibrations of a predetermined object. At the time when the operation shown in Fig. 5 starts, the detection unit 100 is assumed to be ready to detect a predetermined detection area based on the transmission of a transmission wave and the reception of a reflected wave of the transmission wave by transmitting and receiving radio waves such as millimeter waves. Also, at the time when the operation shown in Fig. 5 starts, the imaging unit 300 is assumed to be ready to capture an image of the predetermined imaging area.

[0094] When the operation shown in FIG. 5 starts, the controller 10 of the electronic device 1 according to the embodiment acquires an image of a predetermined imaging area captured by the imaging unit 300 (step S11).

[0095] In one embodiment, the controller 10 may display the image acquired in step S11, for example, on the output unit 40 (display). FIG. 6 is a diagram showing an example in which an image captured by the imaging unit 300 is displayed on the output unit 40 (display). In FIG. 6, the output unit 40 displays an image 40e of the interior of a facility captured by the imaging unit 300. A loudspeaker 40f placed on a desk is displayed near the center of the bottom of the image 40e of the interior of the facility shown in FIG. 6. In FIG. 6, this loudspeaker 40f is the target object. Of course, the target object in the present disclosure is not limited to the loudspeaker 40f, and may be any appropriate object, such as a person, an object, or an animal. As shown in FIG. 6, the output unit 40 may display a bounding box 40d. The bounding box 40d may be used to set a predetermined target object (or target area) in the acquired image. In addition, the output unit 40 may display a setting box 40a that sets the left edge of the bounding box 40d. The output unit 40 may also display a setting box 40b for setting the right edge of the bounding box 40d. The output unit 40 may also display a setting box 40c for setting the distance to a target (object). In the example shown in Fig. 6, the bounding box 40d includes an object 40f (loudspeaker 40f) in the image 40e.

[0096] Next, the controller 10 sets a predetermined object (or target area) in the image acquired in step S11 (step S12). In step S12, the controller 10 may set the predetermined object (or target area) in the image based on a user operation. For example, the controller 10 may prompt the user to set the predetermined object (or target area) via a user interface (UI) such as that shown in FIG. 6. The UI shown in FIG. 6 prompts the user to set the predetermined object (or target area) in the image using a bounding box 40d. In the UI shown in FIG. 6, the user can adjust (set) the horizontal range of the loudspeaker shown in FIG. 6 by specifying the horizontal range (left edge and right edge) of the bounding box using setting boxes 40a and / or 40b.

[0097] 6 illustrates a case where the detection unit 100 performs direction of arrival estimation in the horizontal direction. That is, the UI illustrated in FIG. 6 prompts the user to set the horizontal range of a predetermined object (or target area) in the image using a bounding box. On the other hand, if the detection unit 100 can also perform direction of arrival estimation in the vertical direction, for example, the UI illustrated in FIG. 6 may prompt the user to set the vertical range of a predetermined object (or target area) in the image using a bounding box.

[0098] Furthermore, if the output unit 40 (display) includes a touch panel, the controller 10 may set a predetermined object (or target area) based on a touch (tap) operation on the touch panel by the user. In this case, when the user touches (tap) a predetermined object in the UI image shown in FIG. 6 , the object (or target area) corresponding to the touched image may be automatically set.

[0099] Furthermore, the controller 10 may set a predetermined object (or target area) in an image without a user operation. For example, the controller 10 may perform image recognition on an image such as that shown in FIG. 6 , and when a predetermined object is included in the image, the predetermined object (or target area) may be automatically set. For example, assume that a loudspeaker is set in advance as an object to be recognized by image recognition. In this case, when a loudspeaker is included in the UI image shown in FIG. 6 , an object (or target area) corresponding to the image of the loudspeaker may be automatically set without a user operation.

[0100] Once the target object is set in step S12, the controller 10 acquires information about a predetermined detection area detected by the detection unit 100 (step S13). In step S13, the controller 10 may acquire information about the predetermined detection area detected based on the transmission of a transmission wave and the reception of a reflected wave of the transmission wave. The processing of step S13 does not necessarily have to be performed after steps S11 and S12, and may be performed, for example, before steps S11 and S12.

[0101] The predetermined detection area detected by the detection unit 100 in step S13 must include the object in the image set in step S12. As described above, the detection unit 100 and the imaging unit 300 may be arranged so that their respective optical axes Ra and La are parallel (or nearly parallel). With this configuration, the imaging unit 300 can capture an image centered on the optical axis La. Therefore, the predetermined detection area detected by the detection unit 100 can be made to include the object set in the image.

[0102] After acquiring the information on the predetermined detection area in step S13, the controller 10 acquires a point cloud near the specified distance (step S14). In step S14, the controller 10 may extract a point cloud near the specified distance from information corresponding to coordinates (point cloud) in three-dimensional space. Here, the "specified distance" may represent the distance to the object to be detected by the detection unit 100.

[0103] For example, assume that the distance (from the detector 3) to the loudspeaker is approximately 70 cm in the image shown in Fig. 6. In this case, the controller 10 may prompt the user to input the value of 70 cm as the distance to the object (target) (Target Distance) in the UI shown in Fig. 6. In step S14, the controller 10 may extract a point cloud near the specified 70 cm from information corresponding to coordinates (point cloud) in three-dimensional space.

[0104] Furthermore, the detection unit 100 can detect the distance to the target object. Therefore, in step S14, the controller 10 may specify, for example, the distance to the target object detected by the detection unit 100 as the above-mentioned "specified distance." In this case, the controller 10 does not need to prompt the user to input the distance to the target object (target). In step S14, the controller 10 may extract a point cloud near the distance to the target object detected by the detection unit 100 from information corresponding to coordinates (point cloud) in three-dimensional space.

[0105] In a simplified embodiment, the process of step S14 may be omitted.

[0106] Next, the controller 10 performs direction-of-arrival estimation for the point cloud extracted as a point cloud near the specified distance from the information corresponding to the coordinates (point cloud) in the three-dimensional space (step S15). In step S15, the controller 10 may perform processing based on the same concept as the processing performed by the signal processing unit 110 as normal basic radar processing.

[0107] The information on the predetermined detection area obtained upon completion of the processing of step S15 may be information obtained by performing basic radar processing by the signal processing unit 110 of the detection unit 100 or the controller 10. For example, the processing up to step S14 may include the above-described 2D-FFT processing and CFAR processing. In the 2D-FFT processing, 2D-FF processing may be performed on the IF signal received by the detection unit 100. In the CFAR processing, point cloud information may be obtained by performing CFAR processing on the results calculated by the 2D-FFT processing within a range near a specified distance. In addition, for example, the processing up to step S15 may include the above-described processing for estimating the direction of arrival. In the direction of arrival estimation processing, a process for estimating the direction of arrival may be performed on the point cloud obtained by the CFAR processing.

[0108] FIG. 7 is a diagram showing an example of the results of distance FFT processing performed by the signal processing unit 110 of the detection unit 100 in the situation shown in FIG. 6. The horizontal axis represents the distance (range) detected by the detection unit 100, and the vertical axis represents the intensity of the FFT-processed signal. As shown in FIG. 7, a peak in the FFT intensity, which is likely due to the detection of a loudspeaker, can be seen near a distance (range) of 0.7 m (70 cm). FIG. 8 is a diagram showing an example of the results of velocity FFT processing performed by the signal processing unit 110 of the detection unit 100 in the situation shown in FIG. 6. The horizontal axis represents the distance (range) detected by the detection unit 100, and the vertical axis represents the velocity component of the FFT-processed signal. As shown in FIG. 7, a peak in the velocity component, which is likely due to the detection of a loudspeaker, can be seen near a distance (range) of 0.7 m (70 cm).

[0109] Upon completion of the processing in step S15, the controller 10 can obtain information corresponding to coordinates in three-dimensional space (point cloud) as a point cloud near the specified distance. Next, the controller 10 converts the obtained coordinates of the points in three-dimensional space into coordinates on a two-dimensional plane (step S16). That is, in step S16, the controller 10 corresponds the coordinates of the points in three-dimensional space detected by the detection unit 100 to coordinates on the image captured by the imaging unit 300. In step S16, the controller 10 may use any method to correspond the coordinates of the points in three-dimensional space detected by the detection unit 100 to coordinates on the image captured by the imaging unit 300.

[0110] An example of a method for associating the coordinates of a point cloud in three-dimensional space detected by the detection unit 100 with coordinates on an image captured by the image capture unit 300 will be described below.

[0111] The point cloud data acquired in step S15 is information indicating the three-dimensional positions of the detected objects in three-dimensional space. Therefore, the controller 10 converts the three-dimensional point cloud data acquired in step S15 into two-dimensional data (step S16). In step S16, the controller 10 may generate point cloud information that can be processed two-dimensionally from the point cloud information in three-dimensional space output by the detection unit 100.

[0112] Fig. 9 is a diagram illustrating point cloud information in a three-dimensional space output by the detection unit 100. Fig. 9 is a diagram illustrating an example of a situation in which an object Tm located at a certain location is three-dimensionally detected by the detection unit 100 installed at the origin O.

[0113] As shown in FIG. 9 , the position where the detection unit 100 is installed is taken as the reference (origin O), with the direction toward the object Tm being the positive X-axis direction, and the direction away from the object Tm being the negative X-axis direction. Also, as shown in FIG. 9 , the position where the detection unit 100 is installed is taken as the reference (origin O), with the right side of the detection unit 100 being the positive Y-axis direction, and the left side of the detection unit 100 being the negative Y-axis direction. Also, as shown in FIG. 9 , the position where the detection unit 100 is installed is taken as the reference (origin O), with the upper side of the detection unit 100 being the positive Z-axis direction, and the lower side of the detection unit 100 being the negative Z-axis direction. That is, in FIG. 9 , lx indicates the distance in the depth direction, ly indicates the distance in the horizontal direction, and lz indicates the distance in the vertical (height) direction. In the situation shown in FIG. 9 , the output of the detection unit 100 may be four-channel data of data elements (signal strength and velocity) of position (X, Y, Z) in three-dimensional space at a certain moment. In step S16 shown in FIG. 5, the controller 10 converts the information detected in such a three-dimensional space into two-dimensional information.

[0114] 10 is a diagram illustrating an example of generating point cloud information that can be processed two-dimensionally from point cloud information in three-dimensional space output by the detection unit 100. Fig. 10 is a diagram illustrating an example of an object Tm located at a certain location that is detected three-dimensionally (spatially) by the detection unit 100 installed at the origin O and converted into two-dimensional (planar) data by the electronic device 1.

[0115] As shown in Fig. 10, with the position of origin O as the reference, the right side of origin O is the positive X-axis direction, and the bottom side of origin O is the positive Y-axis direction. That is, in Fig. 10, px indicates the horizontal coordinate, and py indicates the vertical coordinate. In the situation shown in Fig. 10, the output of detection unit 100 is converted at a certain moment from the above-mentioned four-channel data into three-channel data of data elements (signal strength and speed) of position (X, Y) on a two-dimensional plane. In this way, in step S16 shown in Fig. 5, electronic device 1 converts information detected in three-dimensional space into information on a two-dimensional plane.

[0116] When converting the information detected in three-dimensional space as described above into information on a two-dimensional plane, the coordinates of the two-dimensional plane may be calculated based on, for example, the following equations (1) and (2).

[0117]

[0118]

[0119] In the above formulas (1) and (2), lx ｉ ,ly ｉ , lz ｉ indicates the output based on the detection result by the detection unit 100, that is, the information of the point cloud in the three-dimensional space. ｉ indicates the distance in the x direction of the i-th piece of information detected by the detection unit 100. ｉ indicates the distance in the y direction of the i-th piece of information detected by the detection unit 100. ｉ indicates the distance in the z direction of the i-th piece of information detected by the detection unit 100.

[0120] In addition, in the above formulas (1) and (2), px ｉ , py ｉ indicates the coordinates of the point cloud converted into two-dimensional plane information by the controller 10 of the electronic device 1. In particular, px ｉ indicates the x-coordinate of the i-th piece of information detected by the detection unit 100. ｉ indicates the y coordinate of the i-th piece of information detected by the detection unit 100.

[0121] Furthermore, in the above formula (1), M represents the number of pixels in the horizontal direction when a two-dimensional planar image is assumed, and αx represents the horizontal angle of view when a two-dimensional planar image is assumed, and in the above formula (2), N represents the number of pixels in the vertical direction when a two-dimensional planar image is assumed, and αy represents the vertical angle of view when a two-dimensional planar image is assumed.

[0122] In the above formulas (1) and (2), px ｉ , py ｉIn order to function as a coordinate value, px may be rounded off to the nearest integer. ｉ , py ｉ is set to fit within the size of the image after conversion to a two-dimensional plane, for example, 0≦px ｉ ≦M, or 0≦py ｉ Data that does not satisfy N may be discarded.

[0123] In this manner, the electronic device 1 according to an embodiment may generate point cloud information that can be processed two-dimensionally from the output of the detection unit 100. In particular, the electronic device 1 according to an embodiment may convert the point cloud information detected by the detection unit 100 into point cloud information that can be processed two-dimensionally based on at least one of a predetermined number of pixels in a two-dimensional image and a predetermined angle of view. When generating the point cloud information that can be processed two-dimensionally from the output of the detection unit 100, the electronic device 1 according to an embodiment may use a conversion formula other than the above-described formulas (1) and (2). Each point constituting the two-dimensional point cloud data generated in step S16 may include three channels of information: reflection intensity, distance, and velocity at that point.

[0124] After the three-dimensional coordinates of the point are converted into two dimensions in step S14, the controller 10 determines whether the point is within a detection area corresponding to the object in the image (step S17). That is, in step S17, the controller 10 determines whether the point detected by the detection unit 100 is at a position corresponding to the object in the image captured by the imaging unit 300. For example, in step S17, the controller 10 determines whether the point detected by the detection unit 100 is located within the area of ​​a loudspeaker in the image displayed on the output unit 40 in FIG. 6 .

[0125] For example, if a point detected by the detection unit 100 is not located within the area of ​​a loudspeaker in the image displayed on the output unit 40 in Fig. 6, the point is not considered to be a response based on the detection of a loudspeaker. Therefore, if the determination in step S17 is negative (NO), the controller 10 excludes, ignores, or discards the point, and then proceeds to the next step S19.

[0126] On the other hand, for example, if a point detected by the detection unit 100 is located within the area of ​​a loudspeaker in the image displayed on the output unit 40 in Fig. 6, the point is considered to be a response based on the detection of a loudspeaker. Therefore, if the determination in step S17 is positive (YES), the controller 10 proceeds to the next step S19 without excluding, ignoring, or discarding the point.

[0127] In step S19, the controller 10 determines whether or not the determination in step S17 has been made for all points within the detection area detected by the detection unit 100. If the determination in step S17 has not been made for all points within the detection area (NO in step S19), the controller 10 returns to step S15 and repeats the processes in steps S15 to S18 for the remaining points. On the other hand, if the determination in step S17 has been made for all points within the detection area (YES in step S19), the controller 10 may proceed to step S20.

[0128] In step S20, the controller 10 outputs the remaining point cloud (the point cloud that was not discarded in step S18). In this way, the electronic device 1 can extract and output the detection result at a position corresponding to a predetermined object in the image captured by the imaging unit 300 from the information detected by the detection unit 100.

[0129] Furthermore, in step S20, the controller 10 may continue to detect changes over time in the remaining point cloud (the point cloud that was not discarded in step S18) to extract and output a vibration waveform or the like of the object. The controller 10 may display the vibration waveform of the object thus output on, for example, the output unit 40 (display).

[0130] Fig. 11 is a graph showing the results of detecting vibrations of a loudspeaker set in the image displayed on the output unit 40 in Fig. 6, as an example of the operation of the electronic device 1 according to an embodiment. The vertical axis of the graph shown in Fig. 11 represents the amplitude of vibration, and the horizontal axis represents time.

[0131] As described above, in the electronic device 1, the controller 10 extracts information about a detection area corresponding to a region of a predetermined object in an image of a predetermined imaging area from information about the predetermined detection area (for example, steps S17 and S18 shown in FIG. 5 ). As the information about the predetermined detection area, the controller 10 may obtain, for example, information detected by the detection unit 100 based on transmission of a transmission wave and reception of a reflected wave of the transmission wave. The controller 10 may also obtain an image of the predetermined imaging area from, for example, the imaging unit 300.

[0132] In one embodiment, the controller 10 may extract information on a point cloud within a detection area corresponding to the area of ​​the specified object from information on the point cloud detected as information on the specified detection area (for example, steps S17 and S18 shown in Figure 5).

[0133] In addition, in one embodiment, the controller 10 may extract information on a point cloud within the detection area corresponding to the area of ​​the specified object from information on a point cloud detected near a specified distance from information on the specified detection area (for example, step S14 shown in Figure 5).

[0134] In addition, in one embodiment, the controller 10 may extract information about a detection area corresponding to the area of ​​the specified object specified in the image of the specified imaging area from the information about the specified detection area (for example, step S12 shown in Figure 5).

[0135] In addition, in one embodiment, the controller 10 may extract information on the point cloud within the detection area corresponding to the area of ​​the specified object from the direction-estimated information on the point cloud detected as information on the specified detection area (e.g., step S15 shown in Figure 5).

[0136] In addition, in one embodiment, the controller 10 may convert the three-dimensional coordinates of the point cloud detected as information of the specified detection area into two-dimensional coordinates, and determine whether the two-dimensional coordinates of the point cloud are within the area of ​​the specified object in the image of the specified imaging area (e.g., step S16 shown in Figure 5).

[0137] In one embodiment, the controller 10 may generate information regarding the vibration of the predetermined object based on information about the detection area corresponding to the area of ​​the predetermined object (for example, step S20 shown in FIG. 5).

[0138] In one embodiment, the controller 10 may acquire information about the predetermined detection area from the detection unit 100 that is arranged so that the optical axis faces a predetermined direction. In this case, the controller 10 may acquire an image of the predetermined imaging area from the imaging unit 300 that is arranged so that the optical axis faces a direction substantially parallel to the predetermined direction.

[0139] According to the electronic device 1 of one embodiment, for example, when a user specifies an object, it is possible to extract vibration information corresponding to the object. Therefore, the electronic device 1 of one embodiment can detect the vibration of the object (even if an object other than the object is present in the detection area). Therefore, the electronic device 1 of one embodiment can not only improve convenience but also accurately detect the vibration of the object.

[0140] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present disclosure. For example, functions included in each component or step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined or divided into one. Although the embodiments of the present disclosure have been described mainly in terms of an apparatus, the embodiments of the present disclosure can also be realized as a method including steps executed by each component of the apparatus. The embodiments of the present disclosure can also be realized as a method, a program executed by a processor included in the apparatus, or a storage medium on which a program is recorded. It should be understood that these are also encompassed within the scope of the present disclosure.

[0141] The above-described embodiment is not limited to implementation as the electronic device 1. For example, the above-described embodiment may be implemented as a system including the electronic device 1. The above-described embodiment may also be implemented as a control method for a device such as the electronic device 1 or a system including such a device. Furthermore, the above-described embodiment may also be implemented as a program executed by a device such as the electronic device 1 or an information processing device (e.g., a computer), or a system, or as a storage medium or recording medium on which such a program is recorded.

[0142] REFERENCE SIGNS LIST 1 electronic device 3 detection device 5 holder section 7 mounting section 10 controller 20 storage section 30 communication section 40 output section 100 detection section 101 radio wave input section 110 signal processing section 120 transmission section 121 signal generation section 122 synthesizer 123 phase control section 124 amplifier 125 transmission antenna 130 reception section 131 reception antenna 132 LNA 133 mixer 134 IF section 135 AD conversion section 300 imaging section 301 optical input section

Claims

1. An electronic device comprising a controller that acquires information about a predetermined detection area detected based on the transmission of a transmission wave and the reception of a reflected wave of the transmission wave, acquires an image of a predetermined imaging area, and extracts, from the information about the predetermined detection area, information about a detection area corresponding to a region of a predetermined object in the image of the predetermined imaging area.

2. The electronic device according to claim 1, wherein the controller extracts information of a cloud of points within a detection area corresponding to a region of the specified object from information of a cloud of points detected as information of the specified detection area.

3. The electronic device of claim 1, wherein the controller extracts information of a cloud of points within the detection area corresponding to the area of ​​the specified object from information of a cloud of points detected within the specified detection area near a specified distance.

4. The electronic device according to claim 1, wherein the controller extracts information of a detection area corresponding to an area of ​​the specified object specified in an image of the specified imaging area from the information of the specified detection area.

5. The electronic device of claim 1, wherein the controller extracts information of the point cloud within the detection area corresponding to the area of ​​the specified object from directional estimated information of the point cloud detected as information of the specified detection area.

6. The electronic device of claim 1, wherein the controller converts the three-dimensional coordinates of the point cloud detected as information of the specified detection area into two-dimensional coordinates, and determines whether the two-dimensional coordinates of the point cloud are within the area of ​​the specified object in the image of the specified imaging area.

7. The electronic device according to claim 1, wherein the controller generates information relating to vibration of the predetermined object based on information of a detection area corresponding to an area of ​​the predetermined object.

8. The electronic device of claim 1, wherein the controller obtains information of the specified detection area from a detection unit arranged so that its optical axis faces a specified direction, and obtains an image of the specified imaging area from an imaging unit arranged so that its optical axis faces in a direction approximately parallel to the specified direction.

9. A system comprising: a detection unit that detects a predetermined detection area based on transmitting a transmission wave and receiving a reflected wave of the transmission wave; an imaging unit that captures an image of a predetermined imaging area; and a controller that extracts, from the information of the predetermined detection area detected by the detection unit, information of a detection area corresponding to a region of a predetermined object in the image of the predetermined imaging area captured by the imaging unit.

10. The system according to claim 9, wherein the optical axis of the detection unit is arranged to face a predetermined direction, and the optical axis of the imaging unit is arranged to face a direction substantially parallel to the predetermined direction.

11. A method for controlling an electronic device, comprising the steps of: acquiring information about a predetermined detection area detected based on transmitting a transmission wave and receiving a reflected wave of the transmission wave; acquiring an image of a predetermined imaging area; and extracting, from the information about the predetermined detection area, information about a detection area corresponding to a region of a predetermined object in the image of the predetermined imaging area.

12. A program that causes an electronic device to execute the steps of: acquiring information about a predetermined detection area detected based on transmitting a transmission wave and receiving a reflected wave of the transmission wave; acquiring an image of a predetermined imaging area; and extracting, from the information about the predetermined detection area, information about a detection area corresponding to a region of a predetermined object in the image of the predetermined imaging area.

Images