Electronic device, method for controlling electronic device, and program

Abstract

To provide an electronic apparatus that improves the accuracy of object detection performed by a sensor with directivity, a method for controlling an electronic apparatus, and a program.SOLUTION: An electronic apparatus comprises a control unit that detects an object based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave being the transmission wave reflected on the object. The control unit adjusts the signal strength of the reception signal according to the directivity of at least one of the transmission wave and the reflected wave.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to electronic equipment, methods for controlling electronic equipment, and programs. [Background technology]

[0002] For example, in fields such as the automotive industry, technologies for measuring the distance between a vehicle and a designated object are considered important. In particular, in recent years, various radar (RADAR (Radio Detecting and Ranging)) technologies have been researched, which measure the distance to an object by transmitting radio waves such as millimeter waves and receiving reflected waves reflected from obstacles and other objects. The importance of such distance measurement technologies is expected to increase even further in the future with the development of technologies that assist drivers and technologies related to autonomous driving that automate part or all of the driving. Such technologies for measuring the distance to an object are not limited to fields such as transportation, but are expected to be used in various fields. For example, in nursing homes or medical settings, if the location of a person requiring care or nursing can be detected, it can be useful for tracking or monitoring the actions of that person.

[0003] Incidentally, sensors that detect objects by transmitting and receiving radio waves may have a detection intensity that changes depending on the distance to the object to be detected. For example, Patent Document 1 discloses a method for creating a heat map of detection intensity by quantifying the detection intensity according to the distance from multiple distance measuring sensors (ToF (Time of Flight) sensors). [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-19047 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] For example, when detecting objects using directional radio waves with sensors such as millimeter-wave radar, the detection intensity may change depending on the object's orientation relative to the sensor. If the detection intensity differs depending on the object's orientation relative to the sensor, this can affect the detection accuracy of the object, depending on its orientation.

[0006] The purpose of this disclosure is to provide electronic equipment, a control method for electronic equipment, and a program that improve the accuracy of object detection using directional sensors. [Means for solving the problem]

[0007] An electronic device according to one embodiment is The system includes a control unit that detects an object based on a transmission signal transmitted as a transmission wave and a received signal received as a reflected wave when the transmission wave is reflected by an object. The control unit adjusts the signal strength of the received signal according to the directivity of at least one of the transmitted wave and the reflected wave. The control unit, Based on the transmitted signal and the received signal which is received as a reflected wave after the transmitted wave has been reflected by the first object, the point cloud information obtained by detecting the first object, At each position in the point cloud, the signal strength of the received signal is adjusted according to the directivity of at least one of the transmitted wave and the reflected wave, The correct position information of the first object, Machine learning is performed based on this.

[0008] A control method for electronic equipment according to one embodiment is: A step of detecting an object based on a transmitted signal that is transmitted as a transmitted wave, and a received signal that is received as a reflected wave when the transmitted wave is reflected by an object, The steps include adjusting the signal strength of the received signal according to the directivity of at least one of the transmitted wave and the reflected wave, Based on the transmitted signal and the received signal which is received as a reflected wave after the transmitted wave has been reflected by the first object, the point cloud information obtained by detecting the first object, At each position in the point cloud, the signal strength of the received signal is adjusted according to the directivity of at least one of the transmitted wave and the reflected wave, The correct position information of the first object, The steps to perform machine learning based on this, Includes.

[0009] A program according to an embodiment causes a computer to detect the object based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflection wave obtained by reflecting the transmission wave from the object, and adjust the signal strength of the reception signal according to the directivity of at least one of the transmission wave and the reflection wave. Based on the transmitted signal and the received signal which is received as a reflected wave after the transmitted wave has been reflected by the first object, the point cloud information obtained by detecting the first object, At each position in the point cloud, the signal strength of the received signal is adjusted according to the directivity of at least one of the transmitted wave and the reflected wave, The correct position information of the first object, The steps to perform machine learning based on this, perform the above.

Advantage of the Invention

[0010] According to one embodiment, it is possible to provide an electronic device, a control method for an electronic device, and a program that improve the accuracy of object detection by a sensor having directivity.

Brief Description of the Drawings

[0011] [Figure 1] FIG. is a block diagram showing a functional configuration of an electronic device according to an embodiment. [Figure 2] FIG. is a block diagram showing a functional configuration of a sensor according to an embodiment. [Figure 3] FIG. is a diagram showing an example of a transmission signal transmitted by a sensor according to an embodiment. [Figure 4] FIG. is a diagram showing an example of a configuration of a detection device according to an embodiment. [Figure 5] FIG. is a diagram showing an example of object detection by a sensor according to an embodiment. [Figure 6] FIG. is a flowchart for explaining an operation (learning phase) of an electronic device according to an embodiment. [Figure 7] FIG. is a diagram for explaining an example of processing by an electronic device according to an embodiment. [Figure 8] FIG. is a diagram for explaining an example of processing by an electronic device according to an embodiment. [Figure 9]This figure shows an example of a two-dimensional map of adjustment values ​​for an electronic device according to one embodiment. [Figure 10] This figure shows an example of a two-dimensional map of adjustment values ​​for an electronic device according to one embodiment. [Figure 11] This is a flowchart illustrating the operation (inference phase) of an electronic device according to one embodiment. [Modes for carrying out the invention]

[0012] In this disclosure, “electronic device” may mean an electric-powered device. “System” may mean a device that includes an electric-powered device. “User” may mean a person (typically a human) who uses the system and / or electronic device according to one embodiment. A user may include a person who benefits from detecting various objects by using the system and / or electronic device according to one embodiment. The electronic devices, methods, and programs of this disclosure can be used, for example, to detect people, objects, or animals present in a given space such as a room, bed, bathroom, toilet, car, bus, train, corridor, or street.

[0013] The electronic device according to one embodiment described below can generate point cloud information that can be processed in two dimensions from point cloud information in three-dimensional space detected by a sensor based on technology such as millimeter-wave radar. The electronic device according to one embodiment can detect the presence and position of an object based on the point cloud information that can be processed in two dimensions by employing technology such as image recognition. Furthermore, the electronic device according to one embodiment can reduce the influence of the directivity of the transmitted wave and the reflected wave when the transmitted wave is reflected by an object on the detection of the object. The electronic device according to one embodiment will be described in detail below with reference to the drawings.

[0014] Figure 1 is a functional block diagram schematically showing the configuration of an electronic device 1 according to one embodiment. As shown in Figure 1, the electronic device 1 according to one embodiment includes a controller 10. In one embodiment, the electronic device 1 may appropriately include at least one of, for example, a storage unit 20, a communication unit 30, a display unit 40, and a notification unit 50. The controller 10, storage unit 20, communication unit 30, display unit 40, and notification unit 50 described above may be arranged or built into any part of the electronic device 1. Furthermore, at least one of the controller 10, storage unit 20, communication unit 30, display unit 40, and notification unit 50 described above may be arranged outside the electronic device 1 and connected to each other by a wired, wireless, or a combination thereof network. In the electronic device 1 according to one embodiment, at least some of the functional units shown in Figure 1 may be omitted, and other functional units other than those shown in Figure 1 may be appropriately included.

[0015] The electronic device 1 according to one embodiment may be various types of devices. For example, the electronic device according to one embodiment may be any device, such as a specially designed terminal, a general-purpose smartphone, tablet, phablet, notebook PC, computer, or server. Furthermore, the electronic device according to one embodiment may have the function of communicating with other electronic devices, such as a mobile phone or smartphone. Here, the "other electronic devices" mentioned above may be electronic devices such as a mobile phone or smartphone, or any device such as a base station, server, dedicated terminal, or computer. The "other electronic devices" may be, for example, the sensor 100 and / or imaging unit 300 described later. Furthermore, the "other electronic devices" in this disclosure may also be devices or devices that are powered by electricity. When the electronic device according to one embodiment communicates with other electronic devices, it may communicate by wire and / or wirelessly.

[0016] As shown in Figure 1, the electronic device 1 according to one embodiment may be connected to the sensor 100 by wire and / or wirelessly. Through such a connection, the electronic device 1 according to one embodiment can acquire information about the results detected by the sensor 100. Furthermore, the electronic device 1 according to one embodiment may be connected to the imaging unit 300 by wire and / or wirelessly. Through such a connection, the electronic device 1 according to one embodiment can acquire information about the image captured by the imaging unit 300. The sensor 100 and the imaging unit 300 will be described further later.

[0017] The controller 10 controls and / or manages the entire electronic device 1, including each functional part that constitutes the electronic device 1. The controller 10 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing capabilities for performing various functions. The controller 10 may be implemented as a single processor, as several processors, or as separate processors. The processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be implemented as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented based on various other known technologies.

[0018] In one embodiment, the controller 10 may be configured as, for example, a CPU or DSP and a program executed by the CPU or DSP. The program executed by the controller 10, and the results of the processing performed by the controller 10, may be stored in, for example, a storage unit 20. The controller 10 may also include memory necessary for the operation of the controller 10 as appropriate.

[0019] In one embodiment of the electronic device 1, the controller 10 can perform various processes on information output as a result of detection by, for example, the sensor 100. For this reason, in the electronic device 1, the controller 10 may be connected to the sensor 100 by wire and / or wirelessly. Also, in one embodiment of the electronic device 1, the controller 10 can perform various processes on information (images) output as a result of imaging by, for example, the imaging unit 300. For this reason, in the electronic device 1, the controller 10 may be connected to the imaging unit 300 by wire and / or wirelessly. The operation of the controller 10 in one embodiment of the electronic device 1 will be described further later.

[0020] The storage unit 20 may function as a memory for storing various types of information. For example, the storage unit 20 may store programs executed in the controller 10 and the results of processes executed in the controller 10. The storage unit 20 may also store or accumulate detection results from the sensor 100 and / or images captured by the imaging unit 300. Furthermore, the storage unit 20 may function as the work memory of the controller 10. The storage unit 20 can be made of, for example, a semiconductor memory, but is not limited to this, and can be any storage device. For example, the storage unit 20 may be a storage medium such as a memory card inserted into the electronic device 1 according to one embodiment. Furthermore, the storage unit 20 may be made up of, for example, a hard disk drive (HDD) and / or a solid state drive (SSD). Furthermore, the storage unit 20 may be the internal memory of the CPU used as the controller 10 described later, or it may be connected to the controller 10 as a separate unit.

[0021] The memory unit 20 may store, for example, machine learning data. Here, machine learning data may be data generated by machine learning. Furthermore, machine learning may be based on AI (Artificial Intelligence) technology that enables the execution of specific tasks through training. More specifically, machine learning may be a technology in which an information processing device, such as a computer, learns a large amount of data and automatically builds an algorithm or model to perform tasks such as classification and / or prediction. In this specification, machine learning may be included as a part of AI.

[0022] In this specification, machine learning may include supervised learning, which learns the features or rules of input data based on correct answer data. It may also include unsupervised learning, which learns the features or rules of input data without correct answer data. Furthermore, machine learning may include reinforcement learning, which learns the features or rules of input data by providing rewards or punishments. In this specification, machine learning may be an arbitrary combination of supervised learning, unsupervised learning, and reinforcement learning. The concept of machine learning data in this embodiment may include an algorithm that outputs a predetermined inference (estimation) result using an algorithm learned on input data. This embodiment can use, for example, linear regression to predict the relationship between dependent and independent variables, a neural network (NN) that mathematically models the neurons of the human brain, the least squares method which calculates by squaring the error, a decision tree which structures problem solving in a tree structure, and regularization which transforms data in a predetermined way, as well as other appropriate algorithms. This embodiment may utilize deep learning, a type of neural network. Deep learning is a type of neural network, specifically a neural network with a deep network hierarchy.

[0023] The communication unit 30 has the function of an interface for communication by wire or wireless. The communication method performed by the communication unit 30 in one embodiment may be a wireless communication standard. For example, wireless communication standards include cellular phone communication standards such as 2G, 3G, 4G, and 5G. For example, cellular phone communication standards include LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (Registered Trademark) (Global System for Mobile communications), and PHS (Personal Handy-phone System). For example, wireless communication standards include WiMAX (Worldwide Interoperability for Microwave Access), IEEE802.11, WiFi, Bluetooth (Registered Trademark), IrDA (Infrared Data Association), and NFC (Near Field Communication). The communication unit 30 may include a modem whose communication method has been standardized by, for example, the ITU-T (International Telecommunication Union Telecommunication Standardization Sector). The communication unit 30 can support one or more of the above-mentioned communication standards.

[0024] The communication unit 30 may include, for example, an antenna for transmitting and receiving radio waves and a suitable RF unit. The communication unit 30 may also communicate wirelessly with, for example, the communication unit of another electronic device via the antenna. The communication unit 30 may also be configured as an interface such as a connector for wired connection to the outside. Since the communication unit 30 can be configured using known technologies for wireless communication, a more detailed explanation of the hardware will be omitted.

[0025] The various types of information received by the communication unit 30 may be supplied, for example, to the storage unit 20 and / or the controller 10. The various types of information received by the communication unit 30 may be stored, for example, in the memory built into the controller 10. The communication unit 30 may also transmit, for example, the processing results by the controller 10 and / or the information stored in the storage unit 20 to an external source.

[0026] The display unit 40 may be any display device, such as a liquid crystal display (LCD), an organic electro-luminescence panel (OLED), or an inorganic electro-luminescence panel (IEL). The display unit 40 may display various types of information, such as characters, graphics, or symbols. The display unit 40 may also display various GUI objects and icon images to prompt the user to operate the electronic device 1. Various data necessary for displaying information in the display unit 40 may be supplied from, for example, the controller 10 or the storage unit 20. Furthermore, if the display unit 40 includes, for example, an LCD, it may be configured to include a backlight as appropriate. In one embodiment, the display unit 40 may display information based on the results of detection by, for example, the sensor 100. In another embodiment, the display unit 40 may display information based on the results of imaging by, for example, the imaging unit 300.

[0027] The notification unit 50 may notify a predetermined warning to alert the user of the electronic device 1, etc., based on a predetermined signal output from the controller 10. The notification unit 50 may be any functional unit that stimulates at least one of the user's hearing, sight, or touch as a predetermined warning, such as sound, voice, light, text, images, and vibration. Specifically, the notification unit 50 may be at least one of the following: an audio output unit such as a buzzer or speaker, a light-emitting unit such as an LED, a display unit such as an LCD, and a tactile presentation unit such as a vibrator. In this way, the notification unit 50 may notify a predetermined warning based on a predetermined signal output from the controller 10. In one embodiment, the notification unit 50 may notify a predetermined alarm as information that acts on at least one of the hearing, sight, and touch.

[0028] The electronic device 1 shown in Figure 1 incorporates a notification unit 50. However, in one embodiment, the notification unit 50 may be provided outside the electronic device 1. In this case, the notification unit 50 and the electronic device 1 may be connected by wire, wireless, or a combination of wire and wireless.

[0029] As shown in Figure 1, at least a portion of each functional part constituting the electronic device 1 according to one embodiment may be configured by specific means in which software and hardware resources cooperate.

[0030] The sensor 100 shown in Figure 1 is configured to detect objects (targets), such as automobiles or human bodies, as point cloud information in three-dimensional space. A sensor 100 according to one embodiment will be described in more detail below.

[0031] Figure 2 is a functional block diagram schematically showing the configuration of a sensor 100 according to one embodiment. The sensor 100 shown in Figure 2 is, as an example, based on millimeter-wave radar (RADAR (Radio Detecting and Ranging)) technology (millimeter-wave radar sensor). However, the sensor 100 according to one embodiment is not limited to a millimeter-wave radar sensor. For example, the sensor 100 according to one embodiment may be a quasi-millimeter-wave radar sensor. Furthermore, the sensor 100 according to one embodiment is not limited to a millimeter-wave radar sensor or a quasi-millimeter-wave radar sensor, but may be various types of radar sensors that transmit and receive radio waves. In addition, the sensor 100 according to one embodiment may be, for example, a microwave sensor, an ultrasonic sensor, or a sensor based on technology such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0032] When measuring distance and other parameters using millimeter-wave radar, frequency-modulated continuous wave radar (hereinafter referred to as FMCW radar) is sometimes used. FMCW radar generates a transmitted signal by sweeping the frequency of the transmitted radio waves. Therefore, for example, in a millimeter-wave FMCW radar using radio waves in the 79 GHz frequency band, the radio wave frequencies used will have a frequency bandwidth of 4 GHz, such as 77 GHz to 81 GHz. Radar in the 79 GHz frequency band has the advantage of a wider usable frequency bandwidth than other millimeter-wave / quasi-millimeter-wave radars, such as those in the 24 GHz, 60 GHz, and 76 GHz frequency bands. The following describes an example of employing such an FMCW radar. The FMCW radar system used in this disclosure may include a Fast-Chirp Modulation (FCM) system that transmits chirp signals at a shorter period than usual. The signal generated by sensor 100 is not limited to an FMCW signal. The signal generated by sensor 100 may be a signal of various types other than the FMCW method. The transmitted signal sequence stored as the transmitted signal may differ depending on these various methods. For example, in the case of the FMCW radar signal described above, signals whose frequency increases and decreases with each time sample may be used. Since known techniques can be appropriately applied to the various methods described above, more detailed explanations will be omitted as appropriate.

[0033] As shown in Figure 2, the sensor 100 according to one embodiment may include a radar control unit 110, a transmitting unit 120, and a receiving unit 130. The radar control unit 110, transmitting unit 120, and receiving unit 130 described above may be placed or built into any location in the sensor 100. Furthermore, at least one of the radar control unit 110, transmitting unit 120, and receiving unit 130 described above may be placed outside the sensor 100. The sensor 100 according to one embodiment may omit at least some of the functional units shown in Figure 2, or may appropriately include other functional units other than those shown in Figure 2.

[0034] The radar control unit 110 controls and / or manages the entire sensor 100, including each functional part that constitutes the sensor 100. The radar control unit 110 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing capabilities for performing various functions. The radar control unit 110 may be implemented as a single processor, as several processors, or as separate processors. The processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be implemented as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented based on various other known technologies.

[0035] In one embodiment, the radar control unit 110 may be configured as, for example, a CPU or DSP and a program executed by the CPU or DSP. The program executed in the radar control unit 110, and the results of the processing performed in the radar control unit 110, may be stored in, for example, any storage unit built into the radar control unit 110. The radar control unit 110 may also include memory necessary for the operation of the radar control unit 110 as appropriate.

[0036] In the sensor 100 according to one embodiment, the radar control unit 110 may perform various processes as appropriate, such as distance FFT (Fast Fourier Transform) processing, velocity FFT processing, angle of arrival estimation processing, and clustering processing. Since each of these processes performed by the radar control unit 110 is known as general radar technology, a more detailed explanation will be omitted.

[0037] As shown in Figure 2, the transmitting unit 120 may include a signal generation unit 121, a synthesizer 122, a phase control unit 123, an amplifier 124, and a transmitting antenna 125. The sensor 100 according to one embodiment may include a plurality of transmitting antennas 125. In this case, the sensor 100 may also include a plurality of phase control units 123 and amplifiers 124 corresponding to each of the plurality of transmitting antennas 125. When the sensor 100 according to one embodiment includes a plurality of transmitting antennas 125, the plurality of transmitting antennas 125 may constitute a transmitting antenna array (transmitting array antenna).

[0038] As shown in Figure 2, the receiving unit 130 may include a receiving antenna 131, an LNA 132, a mixer 133, an IF unit 134, and an AD conversion unit 135. In one embodiment, the sensor 100 may include multiple receiving units 130, each corresponding to one of the multiple transmitting antennas 125.

[0039] In one embodiment of the sensor 100, the radar control unit 110 can control at least one of the transmitting unit 120 and the receiving unit 130. In this case, the radar control unit 110 may control at least one of the transmitting unit 120 and the receiving unit 130 based on various information stored in any storage unit. For example, any storage unit built into the radar control unit 110 may store various parameters for setting the range for detecting an object using the transmitted wave transmitted from the transmitting antenna 125 and the reflected wave received from the receiving antenna 131. Also, in one embodiment of the sensor 100, the radar control unit 110 may instruct the signal generation unit 121 to generate a signal, or control the signal generation unit 121 to generate a signal.

[0040] The signal generation unit 121 generates a signal (transmission signal) to be transmitted from the transmitting antenna 125 as a transmission wave, under the control of the radar control unit 110. When generating the transmission signal, the signal generation unit 121 may assign a frequency to the transmission signal, for example, based on control by the radar control unit 110. Specifically, the signal generation unit 121 may assign a frequency to the transmission signal according to parameters set by the radar control unit 110, for example. For example, the signal generation unit 121 generates a signal of a predetermined frequency in a frequency band such as 77-81 GHz by receiving frequency information from the radar control unit 110 or any storage unit. The signal generation unit 121 may include a functional unit such as a voltage-controlled oscillator (VCO).

[0041] The signal generation unit 121 may be configured as hardware having the function, or as a microcontroller, for example, or as a processor such as a CPU or DSP and a program executed by that processor. Each of the functional units described below may also be configured as hardware having the function, or, where possible, as a microcontroller, for example, or as a processor such as a CPU or DSP and a program executed by that processor.

[0042] In a sensor 100 according to one embodiment, the signal generation unit 121 may generate a transmission signal (transmission chirp signal), such as a chirp signal. In particular, the signal generation unit 121 may generate a signal whose frequency changes periodically linearly (linear chirp signal). For example, the signal generation unit 121 may generate a chirp signal whose frequency increases periodically linearly from 77 GHz to 81 GHz as time progresses. Alternatively, for example, the signal generation unit 121 may generate a signal whose frequency periodically repeats a linear increase (up chirp) and decrease (down chirp) from 77 GHz to 81 GHz as time progresses. The signal generated by the signal generation unit 121 may be pre-set in, for example, the radar control unit 110. The signal generated by the signal generation unit 121 may also be pre-stored in, for example, any memory unit. Since chirp signals used in technical fields such as radar are known, a more detailed explanation will be simplified or omitted as appropriate. The signal generated by the signal generation unit 121 is supplied to the synthesizer 122.

[0043] Figure 3 illustrates an example of a chirp signal generated by the signal generation unit 121.

[0044] In Figure 3, the horizontal axis represents elapsed time, and the vertical axis represents frequency. In the example shown in Figure 3, the signal generation unit 121 generates a linear chirp signal whose frequency changes linearly and periodically. In Figure 3, each chirp signal is shown as c1, c2, ..., c8. As shown in Figure 3, the frequency of each chirp signal increases linearly with the passage of time.

[0045] In the example shown in Figure 3, eight chirp signals, such as c1, c2, ..., c8, are included in one subframe. That is, subframe 1 and subframe 2, etc., shown in Figure 3, are each composed of eight chirp signals, such as c1, c2, ..., c8. Also, in the example shown in Figure 3, 16 subframes, such as subframe 1 to subframe 16, are included in one frame. That is, frame 1 and frame 2, etc., shown in Figure 3, are each composed of 16 subframes. Furthermore, as shown in Figure 3, a frame interval of a predetermined length may be included between frames. One frame shown in Figure 3 may be, for example, about 30 to 50 milliseconds long.

[0046] In Figure 3, frames 2 and beyond may have a similar configuration. Similarly, in Figure 3, frames 3 and beyond may have a similar configuration. In the sensor 100 according to one embodiment, the signal generation unit 121 may generate the transmission signal as any number of frames. Also, in Figure 3, some chirp signals are omitted. Thus, the relationship between the time and frequency of the transmission signal generated by the signal generation unit 121 may be stored in, for example, any memory unit.

[0047] Thus, the sensor 100 according to one embodiment may transmit a transmission signal consisting of subframes containing multiple chirp signals. Alternatively, the sensor 100 according to one embodiment may transmit a transmission signal consisting of frames containing a predetermined number of subframes.

[0048] Hereinafter, the sensor 100 will be described as transmitting a transmission signal with a frame structure as shown in Figure 3. However, the frame structure shown in Figure 3 is just one example, and the number of chirp signals included in one subframe is not limited to eight. In one embodiment, the signal generation unit 121 may generate subframes containing any number (e.g., any multiple) chirp signals. Also, the subframe structure shown in Figure 3 is just one example, and the number of subframes included in one frame is not limited to 16. In one embodiment, the signal generation unit 121 may generate a frame containing any number (e.g., any multiple) subframes. The signal generation unit 121 may generate signals of different frequencies. The signal generation unit 121 may generate multiple discrete signals with different bandwidths and frequencies f.

[0049] Returning to Figure 2, the synthesizer 122 increases the frequency of the signal generated by the signal generation unit 121 to a frequency in a predetermined frequency band. The synthesizer 122 may increase the frequency of the signal generated by the signal generation unit 121 to a frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125. The frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be set, for example, by the radar control unit 110. Alternatively, the frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be stored, for example, in any memory unit. The signal whose frequency has been increased by the synthesizer 122 is supplied to the phase control unit 123 and the mixer 133. If there are multiple phase control units 123, the signal whose frequency has been increased by the synthesizer 122 may be supplied to each of the multiple phase control units 123. Furthermore, if there are multiple receiving units 130, the signal whose frequency has been increased by the synthesizer 122 may be supplied to each mixer 133 in the multiple receiving units 130.

[0050] The phase control unit 123 controls the phase of the transmission signal supplied from the synthesizer 122. Specifically, the phase control unit 123 may adjust the phase of the transmission signal by appropriately advancing or delaying the phase of the signal supplied from the synthesizer 122, for example, based on control by the radar control unit 110. In this case, the phase control unit 123 may adjust the phase of each transmission signal based on the path difference of the respective transmission waves transmitted from the multiple transmission antennas 125. By appropriately adjusting the phase of each transmission signal, the transmission waves transmitted from the multiple transmission antennas 125 reinforce each other in a predetermined direction to form a beam (beamforming). In this case, the correlation between the direction of beamforming and the amount of phase to be controlled for the transmission signals transmitted by each of the multiple transmission antennas 125 may be stored in, for example, an arbitrary memory unit. The transmission signal whose phase has been controlled by the phase control unit 123 is supplied to the amplifier 124.

[0051] The amplifier 124 amplifies the power of the transmitted signal supplied from the phase control unit 123, for example, based on control by the radar control unit 110. If the sensor 100 has multiple transmitting antennas 125, multiple amplifiers 124 may each amplify the power of the transmitted signal supplied from the corresponding phase control unit 123, for example, based on control by the radar control unit 110. The technique for amplifying the power of the transmitted signal is already known, so a more detailed explanation is omitted. The amplifier 124 is connected to the transmitting antenna 125.

[0052] The transmitting antenna 125 outputs (transmits) the transmission signal amplified by the amplifier 124 as a transmission wave. If the sensor 100 has multiple transmitting antennas 125, each of the multiple transmitting antennas 125 may output (transmit) the transmission signal amplified by the corresponding amplifier 124 as a transmission wave. The transmitting antenna 125 can be configured in the same way as transmitting antennas used in known radar technology, so a more detailed explanation is omitted.

[0053] In this way, the sensor 100 according to one embodiment includes a transmitting antenna 125 and can transmit a transmission signal (e.g., a transmitting chirp signal) as a transmission wave from the transmitting antenna 125. Here, at least one of the functional parts constituting the sensor 100 may be housed in a single housing. In this case, the single housing may be constructed in a way that prevents it from being easily opened. For example, the transmitting antenna 125, the receiving antenna 131, and the amplifier 124 may be housed in a single housing, and this housing may be constructed in a way that prevents it from being easily opened. Furthermore,

[0054] The sensor 100 shown in Figure 2 is an example that includes one transmitting antenna 125. However, in one embodiment, the sensor 100 may include any number of transmitting antennas 125. On the other hand, in one embodiment, the sensor 100 may include multiple transmitting antennas 125 so that the transmitted waves transmitted from the transmitting antennas 125 form a beam in a predetermined direction. In one embodiment, the sensor 100 may include any number of transmitting antennas 125. In this case, the sensor 100 may also include multiple phase control units 123 and amplifiers 124, corresponding to the multiple transmitting antennas 125. The multiple phase control units 123 may each control the phase of multiple transmitted waves supplied from the synthesizer 122 and transmitted from the multiple transmitting antennas 125. The multiple amplifiers 124 may each amplify the power of multiple transmitted signals transmitted from the multiple transmitting antennas 125. In this case, the sensor 100 may be configured to include multiple transmitting antennas. Thus, if the sensor 100 is equipped with multiple transmitting antennas 125, it may also be configured to include multiple functional units necessary for transmitting a wave from each of the multiple transmitting antennas 125.

[0055] The receiving antenna 131 receives reflected waves. The reflected waves may be those that have been reflected by a predetermined object 200 from the transmitted waves. The receiving antenna 131 may be configured to include multiple antennas. If the sensor 100 according to one embodiment includes multiple receiving antennas 131, the multiple receiving antennas 131 may constitute a receiving antenna array (receiving array antenna). The receiving antenna 131 can be configured in the same way as receiving antennas used in known radar technology, so a more detailed explanation is omitted. The receiving antenna 131 is connected to the LNA 132. The received signal based on the reflected waves received by the receiving antenna 131 is supplied to the LNA 132. Thus, if the sensor 100 is equipped with multiple receiving antennas 131, it may also be configured to include multiple functional units necessary for receiving and processing reflected waves from the multiple receiving antennas 131.

[0056] In one embodiment, the sensor 100 can receive reflected waves from a predetermined object 200, which are transmitted as a transmission signal (transmitted chirp signal), such as a chirp signal, from a plurality of receiving antennas 131. When a transmitted chirp signal is transmitted as a transmission wave, the received signal based on the received reflected wave is also referred to as a received chirp signal. That is, the sensor 100 receives a received signal (for example, a received chirp signal) as a reflected wave from the receiving antennas 131.

[0057] The LNA132 amplifies the received signal based on the reflected wave received by the receiving antenna 131 with low noise. The LNA132 functions as a low-noise amplifier, amplifying the received signal supplied from the receiving antenna 131 with low noise. The received signal amplified by the LNA132 is supplied to the mixer 133.

[0058] Mixer 133 generates a beat signal by mixing (multiplying) the RF frequency received signal supplied from LNA 132 with the transmitted signal supplied from synthesizer 122. The beat signal mixed by mixer 133 is supplied to IF unit 134.

[0059] The IF unit 134 performs frequency conversion on the beat signal supplied from the mixer 133, thereby reducing the frequency of the beat signal to an intermediate frequency (IF (Intermediate Frequency) frequency). The beat signal whose frequency has been reduced by the IF unit 134 is supplied to the AD conversion unit 135.

[0060] The AD conversion unit 135 digitizes the analog beat signal supplied from the IF unit 134. The AD conversion unit 135 may be composed of any analog-to-digital converter (ADC). The beat signal digitized by the AD conversion unit 135 is supplied to the radar control unit 110. If there are multiple receiving units 130, the beat signals digitized by each of the multiple AD conversion units 135 may be supplied to the radar control unit 110.

[0061] The radar control unit 110 may perform FFT processing (hereinafter referred to as "distance FFT processing") on the beat signal digitized by the AD conversion unit 135. For example, the radar control unit 110 may perform FFT processing on the complex signal supplied from the AD conversion unit 135. The beat signal digitized by the AD conversion unit 135 can be represented as a time change in signal strength (power). By performing FFT processing on such a beat signal, the radar control unit 110 can represent it as a signal strength (power) corresponding to each frequency. By performing distance FFT processing in the radar control unit 110, a complex signal corresponding to distance can be obtained based on the beat signal digitized by the AD conversion unit 135.

[0062] The radar control unit 110 may determine that a predetermined object 200 exists at the distance corresponding to the peak if the peak in the result obtained by distance FFT processing is above a predetermined threshold. For example, a method is known in which, such as detection processing using a constant false alarm rate (CFAR), if a peak value above a threshold is detected from the average power or amplitude of the disturbance signal, it is determined that an object that reflects the transmitted wave (reflecting object) exists.

[0063] Thus, according to one embodiment, the sensor 100 can detect an object 200 that reflects a transmitted wave as a target, based on the transmitted signal that is transmitted as a transmitted wave and the received signal that is received as a reflected wave.

[0064] The radar control unit 110 can estimate the distance to a predetermined object based on a single chirp signal (for example, c1 shown in Figure 3). That is, the sensor 100 can measure (estimate) the distance between the sensor 100 and the predetermined object 200 by performing distance FFT processing. Since the technique of measuring (estimating) the distance to a predetermined object by performing FFT processing on a beat signal is known, a more detailed explanation will be simplified or omitted as appropriate.

[0065] Furthermore, the radar control unit 110 may perform an FFT (Fast FFT) operation on the beat signal that has undergone distance FFT processing (hereinafter referred to as "velocity FFT processing" as appropriate). For example, the radar control unit 110 may perform an FFT operation on the complex signal that has undergone distance FFT processing. The radar control unit 110 can estimate the relative velocity with a predetermined object based on a subframe of the chirp signal (for example, subframe 1 shown in Figure 3). By performing velocity FFT processing on multiple chirp signals in the radar control unit 110, a complex signal corresponding to the relative velocity is obtained based on the complex signal corresponding to the distance obtained by the distance FFT processing.

[0066] As described above, applying distance FFT processing to a beat signal can generate multiple vectors. By determining the phase of the peaks in the results of velocity FFT processing on these multiple vectors, the relative velocity with a given object can be estimated. In other words, electronic device 1 can measure (estimate) the relative velocity between sensor 100 and a given object 200 by performing velocity FFT processing. The technique of measuring (estimating) the relative velocity with a given object by performing velocity FFT processing on the results of distance FFT processing is well known, so a more detailed explanation will be simplified or omitted as appropriate.

[0067] In typical FMCW radar technology, the presence or absence of a target can be determined based on the results of extracting the beat frequency from the received signal and performing a Fast Fourier Transform (FFT). However, the results obtained by extracting the beat frequency from the received signal and performing a FFT include noise components such as clutter (unwanted reflections). Therefore, it may be possible to remove the noise components from the processed received signal and perform processing to extract only the target signal.

[0068] Furthermore, the radar control unit 110 may estimate the direction (angle of arrival) from which the reflected wave arrives from a predetermined object 200 based on the determination of whether or not a target exists. The radar control unit 110 may perform the estimation of the angle of arrival for points where it has been determined that a target exists. The sensor 100 can estimate the direction from which the reflected wave arrives by receiving the reflected wave from a plurality of receiving antennas 131. For example, the plurality of receiving antennas 131 are arranged at predetermined intervals. In this case, the transmitted wave transmitted from the transmitting antenna 125 is reflected by the predetermined object 200 and becomes a reflected wave, and the plurality of receiving antennas 131 arranged at predetermined intervals each receive the reflected wave R. The radar control unit 110 can then estimate the direction from which the reflected wave arrives at the receiving antennas 131 based on the phase of the reflected wave received by each of the plurality of receiving antennas 131 and the path difference of each reflected wave. That is, the sensor 100 can measure (estimate) the angle of arrival θ, which indicates the direction from which the reflected wave reflected by the target arrives, based on the results of velocity FFT processing.

[0069] Various techniques have been proposed for estimating the direction from which a reflected wave R arrives, based on the results of velocity FFT processing. For example, known algorithms for estimating the direction of arrival include MUSIC (Multiple Signal Classification) and ESPRIT (Estimation of Signal Parameters via Rotational Invariance Technique). Therefore, more detailed explanations of known techniques will be simplified or omitted as appropriate.

[0070] The radar control unit 110 detects objects within the range from which the transmitted wave was transmitted, based on at least one of distance FFT processing, velocity FFT processing, and angle of arrival estimation. The radar control unit 110 may also perform object detection by, for example, clustering processing based on the supplied distance information, velocity information, and angle information. Known algorithms for clustering data include, for example, DBSCAN (Density-based spatial clustering of applications with noise). In the clustering process, for example, the average power of the points constituting the detected object may be calculated.

[0071] As described above, the sensor 100 can detect objects that reflect transmitted waves in three-dimensional space as point cloud information. That is, in one embodiment, based on the detection result output from the sensor 100, it is possible to determine (detect) whether or not an object that reflects transmitted waves exists at a certain coordinate in three-dimensional space. Also, in one embodiment, the sensor 100 can detect the signal strength and velocity of each point in three-dimensional space. As explained above, the sensor 100 according to one embodiment may detect objects that reflect transmitted waves as point cloud information in three-dimensional space based on the transmitted signal transmitted as a transmitted wave and the received signal received as a reflected wave from which the transmitted wave has been reflected.

[0072] Furthermore, as shown in Figure 1, the electronic device 1 according to one embodiment may include an imaging unit 300. The electronic device 1 and the imaging unit 300 may be connected by wire, wireless, or a combination of wire and wireless.

[0073] The imaging unit 300 may include an image sensor that electronically captures images, such as a digital camera. The imaging unit 300 may include an image sensor that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 300 may capture images of objects located in front of it. Here, objects located in front of the imaging unit 300 may be, for example, cars, people, and / or any objects in the surroundings. The imaging unit 300 may convert the captured image into a signal and transmit it to the electronic device 1. For example, the imaging unit 300 may transmit a signal based on the captured image to the extraction unit 11, storage unit 20, and / or controller 10 of the electronic device 1. The imaging unit 300 is not limited to imaging devices such as digital cameras, but may be any device that can capture images of objects. The imaging unit 300 may be, for example, a LIDAR (Light Detection And Ranging).

[0074] In one embodiment, the imaging unit 300 may capture still images, for example, at predetermined intervals (e.g., 15 frames per second). In another embodiment, the imaging unit 300 may capture, for example, a continuous video.

[0075] Next, the arrangement of the sensor 100 and imaging unit 300 connected to the electronic device 1 according to one embodiment will be described.

[0076] Figures 4(A) and 4(B) show examples of the configuration of a detection device in which the sensor 100 and the imaging unit 300 are arranged.

[0077] Figure 4(A) is a front view showing an example of a detection device 3 according to one embodiment, viewed from the front. Figure 4(B) is a side view showing an example of a detection device 3 according to one embodiment, viewed from the side (left). The coordinate axes shown in Figures 4(A) and 4(B) are aligned with the coordinate axes showing the propagation direction of the transmitted and / or reflected waves of the sensor 100 shown in Figure 2.

[0078] As shown in Figures 4(A) and 4(B), the detection device 3 according to one embodiment may include a sensor 100 and an imaging unit 300. Also, as shown in Figures 4(A) and 4(B), the detection device 3 according to one embodiment may appropriately include at least one of a stand 5 and a grounding unit 7. Furthermore, the detection device 3 according to one embodiment may be placed on any other device or the housing of another device without including at least one of the stand 5 and the grounding unit 7.

[0079] The sensor 100 shown in Figures 4(A) and 4(B) may be the same sensor 100 described in Figures 1 and / or 2. As shown in Figures 4(A) and 4(B), the sensor 100 may include a radio wave input unit 101 that receives reflected waves when a transmitted wave is reflected by an object. As shown in Figure 4(B), the radio wave input unit 101 may be oriented toward the optical axis Ra of the sensor 100. Here, the optical axis of the sensor 100 may be, for example, the direction perpendicular to the plane on which at least one of the transmitting antenna 125 and the receiving antenna 131 of the sensor 100 is installed. Alternatively, if the sensor 100 includes multiple transmitting antennas 125 and receiving antennas 131, the optical axis of the sensor 100 may be the direction perpendicular to the plane on which at least one of the multiple antennas is installed. With this configuration, the sensor 100 can transmit a transmitted wave and / or receive reflected waves around the optical axis Ra. That is, the sensor 100 can detect objects as a point cloud within a range centered on the optical axis Ra.

[0080] One embodiment of the sensor 100 may be directional. That is, the sensor 100 may detect objects using directional radio waves. Here, directionality may be defined as the relationship between the radiation direction and radiation intensity of radio waves as a characteristic of the antenna. Whether or not there is directionality is related to the application of the antenna. A highly directional antenna strongly radiates radio waves in a specific direction. The directionality may be the same for both transmission and reception. The electric field strength of the radio waves radiated by the antenna can be expressed in decibels (dB) as the antenna gain. By being directional, one embodiment of the sensor 100 may have, for example, a main lobe (main beam) in the direction of the optical axis Ra shown in Figure 4(B). That is, one embodiment of the sensor 100 may have, for example, the strongest radiation level in the direction of the optical axis Ra.

[0081] Furthermore, the imaging unit 300 shown in Figures 4(A) and 4(B) may be the same imaging unit 300 described in Figure 1. As shown in Figures 4(A) and 4(B), the imaging unit 300 may include an optical input unit 301 that receives light reflected by an object. As shown in Figure 4(B), the optical input unit 301 may be oriented toward the optical axis La of the imaging unit 300. Alternatively, the optical input unit 301 may be located where the lens is positioned in the imaging unit 300. Here, the optical axis of the imaging unit 300 may be, for example, the direction perpendicular to the plane on which the light-receiving element (or image sensor) used for imaging is installed in the imaging unit 300. With this configuration, the imaging unit 300 can capture an image centered on the optical axis La.

[0082] As shown in Figures 4(A) and 4(B), the stand 5 maintains the sensor 100 and imaging unit 300 of the detection device 3 at a predetermined height from the ground. The stand 5 may maintain the sensor 100 at a height that makes it easy for the sensor 100 to detect a predetermined object. The stand 5 may also maintain the sensor 100 at a height that makes it easy for the imaging unit 300 to image a predetermined object. The stand 5 may be equipped with a mechanism that allows the sensor 100 and imaging unit 300 to be adjusted, for example, in the height direction, of the detection device 3.

[0083] As shown in Figures 4(A) and 4(B), the grounding portion 7 fixes the sensor 100 and imaging unit 300 of the detection device 3 to the ground surface. The grounding portion 7 can be configured in various ways, such as being shaped like a pedestal, in order to stabilize the detection device 3 which includes the sensor 100 and imaging unit 300.

[0084] As shown in Figures 4(A) and 4(B), in the detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged adjacent to each other in their vicinity. In the example shown in Figures 4(A) and 4(B), the sensor 100 and the imaging unit 300 are arranged adjacent to each other in the vertical direction. In the detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged adjacent to each other, for example, in the left-right direction or diagonally.

[0085] Furthermore, as shown in Figure 4(B), in the detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged so that their respective optical axes Ra and La are parallel. That is, in the electronic device 1 according to one embodiment, the point cloud information from the sensor 100 and the image information from the imaging unit 300 may be used with the imaging unit 300's optical axis La being set parallel to the sensor 100's optical axis Ra.

[0086] Furthermore, as shown in Figure 4(B), in the detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged such that the distance between their respective optical axes Ra and La is maintained at a distance G. By arranging them in this way, the point cloud information from the sensor 100 and the image information from the imaging unit 300 will be shifted by a distance G from each other. For example, in the arrangement shown in Figure 4(B), the image information from the imaging unit 300 will be shifted upward by a distance G from the point cloud information from the sensor 100. Also, in the arrangement shown in Figure 4(B), the point cloud information from the sensor 100 will be shifted downward by a distance G from the image information from the imaging unit 300.

[0087] Therefore, in the arrangement shown in Figure 4(B), for example, by correcting the point cloud information from sensor 100 to shift upward by a distance G, the position of the point cloud information from sensor 100 can be made to correspond to the position of the image information from imaging unit 300. Also, in the arrangement shown in Figure 4(B), for example, by correcting the image information from imaging unit 300 to shift downward by a distance G, the position of the image information from imaging unit 300 can be made to correspond to the position of the point cloud information from sensor 100. In this way, the electronic device 1 may correct at least one of the point cloud information from sensor 100 and the image information from imaging unit 300 so that the point cloud information from sensor 100 and the image information from imaging unit 300 correspond to each other in terms of position.

[0088] In other words, in the electronic device 1 according to one embodiment, at least one of the point cloud information obtained by the sensor 100 detecting an object (target) and the image information obtained by the imaging unit 300 imaging the object (target) may be corrected. The electronic device 1 according to one embodiment may use information obtained by correcting and aligning the point cloud information from the sensor 100 and the image information from the imaging unit 300.

[0089] Furthermore, it is conceivable that the detection range (angle) of the point cloud by the sensor 100 and the image acquisition range (angle or field of view) of the image by the imaging unit 300 may not be the same. In such cases, the electronic device 1 may adjust the wider range (angle) of the two so that the point cloud information from the sensor 100 and the image information from the imaging unit 300 correspond to each other in terms of position. In other words, the electronic device 1 may use only the information of the overlapping range between the imaging range of the imaging unit 300 and the detectable range of the sensor 100, and delete or ignore the information of the non-overlapping ranges.

[0090] As described above, in the electronic device 1 according to one embodiment, the image information obtained by the imaging unit 300 imaging an object may be information that corresponds in position to the point cloud information obtained by the sensor 100 detecting the object.

[0091] The following describes object detection using the detection device 3 shown in Figures 4(A) and 4(B). In the detection device 3, the sensor 100 has directionality as described above, and therefore has a main lobe (main beam) in the direction of the optical axis Ra shown in Figure 4(B). That is, the sensor 100 according to one embodiment has the strongest radiation level in the direction of the optical axis Ra. In this case, the sensor 100 can output a point cloud representing the detected object by irradiating a directional transmission wave and receiving the reflected wave. Here, the sensor 100 that performs such detection has the characteristic of being most likely to detect the point cloud of an object in the direction of the optical axis Ra shown in Figure 4(B), i.e., in front of the sensor 100. Therefore, for example, even if two objects with the same reflection characteristics are equidistant from the sensor 100, the number of point clouds output by detection will differ between the object located in front of the sensor 100 and the object located elsewhere in front of the sensor 100.

[0092] Figures 5(A) and 5(B) illustrate how the number of point clouds output by the sensor 100 differs depending on the position relative to the front of the sensor 100, due to the sensor 100's directivity.

[0093] For example, as shown in Figure 5(A), suppose one car is positioned almost directly in front of the detection device 3, and another car is positioned slightly to the right of the front of the detection device 3. Here, both cars may be positioned at approximately equidistant distances from the sensor 100. Also, both cars may have approximately the same reflective properties. Figure 5(A) may be an image of the two cars positioned in this manner, captured by the imaging unit 300 shown in Figures 4(A) and 4(B). In this case, as shown in Figure 5(A), in the image captured by the imaging unit 300, one car is positioned almost in the center of the image, and the other car is positioned slightly to the right of the center of the image.

[0094] Figure 5(B) shows an example of a point cloud output by sensor 100 in the situation shown in Figure 5(A). As shown in Figure 5(B), the car located in the central part is detected as a relatively large number of points (relatively high point cloud density). On the other hand, the car located slightly to the right of the central part is detected as a relatively small number of points (relatively low point cloud density). Thus, the detection accuracy of objects located away from the front of sensor 100 may be lower than that of objects located in front of sensor 100. In other words, for two objects located at approximately equidistant from sensor 100 and having the same reflectivity, the detection results (detected point cloud information) may differ to a considerable extent. If such point cloud information is used as input data for machine learning, such as by AI, it may become difficult to perform appropriate machine learning.

[0095] Therefore, in one embodiment of the electronic device 1, the signal strength of the received signal based on the reflected wave that is reflected by an object from the transmitted wave transmitted from the sensor 100 may be adjusted according to the directivity of at least one of the transmitted wave and the reflected wave. As described above, an object located in the center as seen from the sensor 100 is detected as a relatively large number of point clouds (relatively high point cloud density). That is, an object located in the center as seen from the sensor 100 is detected as a relatively high intensity by the sensor 100. On the other hand, an object located away from the center as seen from the sensor 100 is detected as a relatively small number of point clouds (relatively low point cloud density). That is, an object located away from the center as seen from the sensor 100 is detected as a relatively low intensity by the sensor 100. Therefore, in one embodiment of the electronic device 1, the detection intensity by the sensor 100 may be adjusted so that the further an object is located away from the center as seen from the sensor 100, the higher the detection intensity by the sensor 100. More specifically, the signal strength of the received signal based on the reflected wave reflected by an object may be adjusted so that the further an object is located away from the center as seen from the sensor 100, the higher the signal strength of the received signal. Furthermore, the signal strength of the received signal based on the reflected wave reflected by an object may be adjusted so that the closer an object is to the center of the sensor 100, the lower the signal strength of that object becomes.

[0096] As described above, the electronic device 1 according to one embodiment includes a controller 10 that detects an object based on a transmitted signal that is transmitted as a transmitted wave and a received signal that is received as a reflected wave after the transmitted wave is reflected by an object. Here, the controller 10 may adjust the signal strength of the received signal according to the directivity of at least one of the transmitted wave and the reflected wave. More specifically, the controller 10 may adjust the signal strength of the received signal so that it becomes stronger as the radiation intensity of at least one of the transmitted wave and the reflected wave weakens. The adjustment of the signal strength of the received signal by the electronic device 1 according to one embodiment will be described further later.

[0097] Next, the operation of the electronic device 1 according to one embodiment will be described. As described above, the electronic device 1 according to one embodiment may detect objects by adjusting the signal strength of the received signal so that it becomes stronger as the radiation intensity of at least one of the transmitted wave and the reflected wave becomes weaker. Alternatively, the electronic device 1 according to one embodiment may perform detection of various objects and / or identification of detected objects by adjusting the signal strength of the point cloud data obtained as a result of detecting objects at various locations and then performing machine learning. The following describes machine learning for detecting objects using the electronic device 1 according to one embodiment.

[0098] The operation of performing machine learning using the electronic device 1 according to one embodiment can typically be divided into a "learning phase" and an "inference phase." The learning phase in this disclosure may include, for example, a training phase for generating parameters used in result output processing. In the learning phase, for example, if an object is present at a predetermined position as seen from the sensor 100, machine learning may be performed by providing correct data (training data) for the point cloud data detected from the object, including the signal strength of the received signal adjusted according to the position. In the inference phase, the results of machine learning performed in the learning phase may be used to perform an operation to detect the presence or absence of an object, and / or an operation to identify the detected object. The "learning phase" and "inference phase" of machine learning performed by the electronic device 1 according to one embodiment will be described below.

[0099] Figure 6 is a flowchart illustrating the operation of the learning phase performed by the electronic device 1 according to one embodiment. The flow of operations performed by the electronic device 1 is described below in general terms. The operation shown in Figure 5 may be started when the electronic device 1 detects an object present in its surroundings.

[0100] When the learning phase shown in Figure 6 begins, the radar control unit 110 of the sensor 100 connected to the electronic device 1 controls the sensor 100 to transmit a wave from its transmitting antenna 125 (step S11).

[0101] When the transmission wave is transmitted in step S11, the radar control unit 110 controls the sensor 100 to receive the reflected wave that has been reflected off an object (step S12).

[0102] In step S12, when the reflected wave is received, the radar control unit 110 performs predetermined signal processing on the beat signal based on the transmitted wave and the reflected wave to generate point cloud information (point cloud data) based on object detection (step S13). In step S13, the radar control unit 110 may perform at least one of the above-mentioned signal processing, such as distance FFT processing, velocity FFT processing, detection processing using constant false alarm probability (CFAR), and predetermined clustering processing. In step S13, the radar control unit 110 is not limited to the above-mentioned signal processing and may perform any processing to generate point cloud data based on object detection. For example, in technologies such as millimeter-wave radar, various processing methods for generating point cloud data based on object detection are known, so a more detailed explanation is omitted.

[0103] In step S13, the radar control unit 110 may generate point cloud data as shown in Figure 5(B), for example, as point cloud data corresponding to the objects shown in Figure 4(A) (in this case, two cars). In this way, the sensor 100 outputs point cloud data based on object detection. The point cloud data output from the sensor 100 in this manner may be input to the controller 10 of the electronic device 1.

[0104] The point cloud data generated in step S13 is information indicating the three-dimensional position of an object detected in three-dimensional space. Therefore, the controller 10 of the electronic device 1 converts the three-dimensional point cloud data generated in step S13 into two-dimensional data (step S14). In step S14, the controller 10 may generate point cloud information that can be processed in two dimensions from the point cloud information in three-dimensional space output by the sensor 100.

[0105] Figure 7 illustrates the point cloud information in three-dimensional space output by the sensor 100. Figure 7 shows an example of a situation in which an object (in this case, a person) Tm located at a certain location is detected three-dimensionally by the sensor 100 installed at the origin O.

[0106] As shown in Figure 7, with the position where the sensor 100 is installed as the reference point (origin O), the direction approaching object Tm is defined as the positive X-axis, and the direction moving away from object Tm is defined as the negative X-axis. Also, as shown in Figure 7, with the position where the sensor 100 is installed as the reference point (origin O), the right side of the sensor 100 is defined as the positive Y-axis, and the left side of the sensor 100 is defined as the negative Y-axis. Furthermore, as shown in Figure 7, with the position where the sensor 100 is installed as the reference point (origin O), the upper side of the sensor 100 is defined as the positive Z-axis, and the lower side of the sensor 100 is defined as the negative Z-axis. That is, in Figure 7, lx represents the distance in the depth direction, ly represents the distance in the horizontal direction, and lz represents the distance in the vertical (height) direction. In the situation shown in Figure 7, the output of the sensor 100 may be four channels of data (signal strength and velocity) representing the position (X, Y, Z) in three-dimensional space at a given moment. In step S14 shown in Figure 6, the controller 10 converts the information detected in this three-dimensional space into two-dimensional information.

[0107] Figure 8 illustrates an example of generating point cloud information that can be processed in two dimensions from point cloud information in three-dimensional space output by sensor 100. Figure 8 shows an example in which an object (in this case, a person) Tm located at a certain location is detected in three dimensions (spatially) by sensor 100 installed at the origin O, and then converted to two dimensions (planarly) by electronic device 1.

[0108] As shown in Figure 8, with the origin O as the reference point, the area to the right of the origin O is defined as the positive X-axis direction, and the area below the origin O is defined as the positive Y-axis direction. That is, in Figure 8, px represents the horizontal coordinate and py represents the vertical coordinate. In the situation shown in Figure 8, the output of sensor 100 is converted at a given moment from the aforementioned 4-channel data to 3-channel data of position (X,Y) data elements (signal strength and velocity) in a 2D plane. In this way, in step S14 shown in Figure 6, the electronic device 1 converts information detected in 3D space into information in a 2D plane.

[0109] When converting the information detected in the three-dimensional space as described above into the information on the two-dimensional plane, the coordinates of the two-dimensional plane may be calculated based on, for example, the following equations (1) and (2).

[0110]

Number

[0111]

Number

[0112] In the above equations (1) and (2), lx i , ly i , lz i represent the output based on the detection result by the sensor 100, that is, the information of the point cloud in the three-dimensional space. In particular, lx i represents the distance in the x direction of the i-th information detected by the sensor 100. Also, ly i represents the distance in the y direction of the i-th information detected by the sensor 100. Also, lz i represents the distance in the z direction of the i-th information detected by the sensor 100.

[0113] Also, in the above equations (1) and (2), px i , py i represent the coordinates of the point cloud converted into the information on the two-dimensional plane by the controller 10 of the electronic device 1. In particular, px i represents the x coordinate of the i-th information detected by the sensor 100. Also, py i represents the y coordinate of the i-th information detected by the sensor 100.

[0114] Furthermore, in equation (1) above, M represents the number of pixels in the horizontal direction when assuming a two-dimensional plane image, and αx represents the horizontal field of view when assuming a two-dimensional plane image. Also, in equation (2) above, N represents the number of pixels in the vertical direction when assuming a two-dimensional plane image, and αy represents the vertical field of view when assuming a two-dimensional plane image.

[0115] In equations (1) and (2) above, px i py i The first decimal place may be rounded to an integer so that it functions as a coordinate value. Also, in equations (1) and (2) above, px i py i The size should fit within the size of the image after it has been converted to a 2D plane, for example, 0 ≤ px i ≤M, or 0 ≤ py i Data that does not satisfy ≤N may be discarded.

[0116] Thus, the electronic device 1 according to one embodiment may generate two-dimensionally processable point cloud information from the output of the sensor 100. In particular, the electronic device 1 according to one embodiment may convert the point cloud information detected by the sensor 100 into two-dimensionally processable point cloud information based on at least one of a predetermined number of pixels and a predetermined field of view in a two-dimensional image. When the electronic device 1 according to one embodiment generates two-dimensionally processable point cloud information from the output of the sensor 100, it may use a conversion formula other than the above-described formulas (1) and (2).

[0117] According to one embodiment of the electronic device 1, the point cloud information in step S13 is reduced in dimensionality in step S14, thereby significantly reducing the computational load on, for example, the controller 10. Therefore, according to one embodiment of the electronic device 1, the processing load on, for example, the controller 10 can be reduced. Each point constituting the two-dimensional point cloud data generated in step S14 may include three channels of information: reflection intensity, distance, and velocity at that point.

[0118] In step S14, once the point cloud data is converted to two dimensions, the controller 10 generates data for machine learning based on the two-dimensional point cloud data (step S15). As described above, the density of the point cloud output from the sensor 100 as a result of object detection by the sensor 100 may vary depending on the orientation of the object relative to the sensor 100. Therefore, in step S15, the controller 10 may adjust the signal strength of the received signal based on the reflected wave, which is the transmitted wave reflected by the object, according to the directivity of at least one of the transmitted wave and the reflected wave. Here, the sensor 100 is assumed to have a directivity, and therefore has a main lobe (main beam) in the direction of the center in front of the sensor 100. In this case, the controller 10 may adjust the sensor 100 so that the detection intensity of an object is higher the further it is from the center relative to the sensor 100.

[0119] In one embodiment, the electronic device 1 may store, for example, a value defined for adjusting the signal strength of the received signal according to the directivity of the sensor 100 in a memory unit 20 or the like. In this case, the electronic device 1 may store, for example, the value for adjusting the signal strength of the received signal according to the directivity of the sensor 100 in a map.

[0120] For example, as shown in Figure 9, the electronic device 1 may map a two-dimensional array of values ​​for adjusting the signal strength of the received signal within the detection range of the sensor 100. In the example shown in Figure 9, the signal strength of the received signal is highest at the center of the front of the detection range of the sensor 100, so the adjustment value for the received signal may be set to 1 (e.g., 1x). Also in the example shown in Figure 9, the signal strength of the received signal decreases as the distance from the center of the front of the detection range of the sensor 100 increases, so the adjustment value for the received signal may be set to 2 (e.g., 2x). In this way, the signal strength of the received signal decreases as the distance from the center of the front of the detection range of the sensor 100 increases. Therefore, the further away from the center of the front of the detection range of the sensor 100, the larger the adjustment value for the received signal may be (e.g., 3x, 4x, etc.). Here, the distance from the center of the front of the detection range of the sensor 100 may be, for example, the Euclidean distance or the Manhattan distance. The map shown in Figure 9 may correspond to, for example, the two-dimensional (planar) region shown in Figure 8. The array maps of this disclosure are not limited to the example shown in Figure 9, and may include, for example, a map in which the adjustment values ​​change by concentric circles, or a map in which triangles, other polygons, or combinations thereof are arranged as shapes in which the adjustment values ​​are placed.

[0121] Thus, in one embodiment, the controller 10 may adjust the signal strength of the received signal as the distance increases from the position where the radiation intensity of at least one of the transmitted wave and the reflected wave is strongest. Furthermore, in this disclosure, in addition to positive integers, decimal values, fractional values, negative values, or 0 may be used as adjustment values ​​for the received signal in an appropriate combination. Also, the adjustment values ​​of the map in this disclosure may be a distribution that reflects one or more side lobes or nulls based on the intensity distribution of the transmitted or received wave.

[0122] Furthermore, as shown in Figure 10(A) or Figure 10(B), for example, the electronic device 1 may map the values ​​for adjusting the signal strength of the received signal within the detection range of the sensor 100 based on a two-dimensional Gaussian distribution. Here, the two-dimensional Gaussian distribution may be the sum of constants proportional to the gain of the antenna. In both Figure 10(A) and Figure 10(B), a plane parallel to the XY plane is shown as the detection range of the sensor 100, and the adjustment values ​​of the received signal are shown by axes perpendicular to the XY plane.

[0123] The maps shown in Figures 10(A) and 10(B), similar to the map in Figure 9, show that the adjustment value for the received signal is set lowest in the central part of the front of the detection range of the sensor 100. Furthermore, the maps shown in Figures 10(A) and 10(B), similar to the map in Figure 9, show that the adjustment value for the received signal increases as you move away from the central part of the front of the detection range of the sensor 100. Figure 10(A) may be used to map the values ​​for adjusting the signal strength of the received signal when using a low-gain antenna with relatively shallow directivity. On the other hand, Figure 10(B) may be used to map the values ​​for adjusting the signal strength of the received signal when using a high-gain antenna with relatively shallow directivity.

[0124] A two-dimensional Gaussian distribution, as shown in Figure 10(A) or Figure 10(B), can be calculated, for example, as shown in equation (3) below. Equation (3) is a diagram showing the Gaussian distribution of the variable n.

[0125]

number

[0126] In equation (3) above, Σ=(σ ij ) represents a positive definite symmetric matrix called the variance-covariance matrix. Also, │Σ│ represents the determinant of Σ.

[0127] Based on equation (3) above, if we reflect the adjustment value G of the received signal based on the antenna directivity for the two variables (x,y), we get the following equation (4).

[0128]

number

[0129] In the above formula (4), μ x =μ y It is acceptable to set = 0. Also, σ x and σ y G may be a constant. The coordinates of the center position of sensor 100 may be (x,y)=(0,0). G may be a variable that reflects the antenna gain. Here, G may reflect, for example, the receiving antenna gain, or it may reflect both the transmitting antenna gain and the receiving antenna gain.

[0130] Thus, in one embodiment, the controller 10 may adjust the signal strength of the received signal based on a two-dimensional Gaussian distribution obtained by integrating a constant proportional to the gain of at least one of the transmitted wave and the reflected wave.

[0131] In step S14 shown in Figure 6, the controller 10 may generate three channels of information for each point constituting the point cloud data: reflection intensity, distance, and velocity. In this case, in step S15 shown in Figure 6, the controller 10 may generate four channels of data by adding one channel of information for a value (map) that adjusts the signal intensity of the received signal at each point to the three channels of information.

[0132] Once machine learning data is generated in step S15, the controller 10 may perform machine learning by inputting the machine learning data into the AI ​​(e.g., a neural network) (step S16). The machine learning performed in step S16 may use 4-channel data as training data, which is obtained by adding 1 channel of information (map) that adjusts the signal strength of the received signal to the 3 channels of information (reflection intensity, distance, and velocity) mentioned above. Alternatively, the machine learning performed in step S16 may use data indicating the position of the object that generated the 4-channel data as training data (ground truth data). For example, if machine learning is performed in step S16 using 4-channel point cloud data as training data as shown in Figure 5(B), the position (coordinates) of the object shown in Figure 5(A) may be used as training data (ground truth data) in step S16.

[0133] Furthermore, the reflection intensity information included in the 3-channel information generated in step S14 may be pre-adjusted based on a value (map) that adjusts the signal intensity of the received signal at each point. In this case, in step S15 shown in Figure 6, the controller 10 may generate 3-channel information of reflection intensity (adjusted), distance, and velocity at each point constituting the point cloud data. Then, in step S16, the controller 10 may perform machine learning using the 3-channel data of the point cloud as training data and the object's position (coordinates) as training data (ground truth data).

[0134] Thus, in one embodiment, the controller 10 may perform machine learning. This machine learning may be performed based on, for example, the following information. (1) Point cloud information obtained by detecting the first object based on the transmission signal transmitted from the sensor 100 and the received signal which is received as a reflected wave when the transmitted wave transmitted from the sensor 100 is reflected by the first object. (2) The signal strength of the received signal adjusted according to the directivity of at least one of the transmitted wave and the reflected wave at each position in the obtained point cloud. (3) Correct information on the position of the first object

[0135] Figure 11 is a flowchart illustrating the operation of the learning phase performed by the electronic device 1 according to one embodiment. The flow of operations performed by the electronic device 1 is described below in general terms. The operation shown in Figure 11 may be started when the electronic device 1 detects an object present in its surroundings.

[0136] The operations from step S21 to step S24 shown in Figure 11 can be performed in the same way as the operations from step S11 to step S14 described in Figure 6. Therefore, a more detailed explanation is omitted.

[0137] In step S24, once the point cloud data has been converted to two dimensions, the controller 10 inputs the two-dimensional point cloud data into the machine learning-trained AI (e.g., a neural network) shown in Figure 6, and performs the process of recognizing a predetermined object (step S25). In step S26, if it is determined that there is a point cloud of an object to be detected, the controller 10 outputs the position of the detected object (step S27). On the other hand, in step S26, if it is determined that there is no point cloud of an object to be detected, the controller 10 terminates the operation shown in Figure 11 without outputting the position of the detected object.

[0138] Thus, in one embodiment, the controller 10 may perform object detection based on the results of machine learning. This object detection may be performed based on the following information, for example. (1) Point cloud information obtained by detecting the second object based on the transmission signal transmitted from sensor 100 and the received signal which is received as a reflected wave when the transmitted wave transmitted from sensor 100 is reflected by the second object. (2) The signal strength of the received signal adjusted according to the directivity of at least one of the transmitted wave and the reflected wave at each position in the obtained point cloud. (3) Results of machine learning performed in step S26 of Figure 6 In one embodiment, the controller 10 may infer the position of the second object based on the information described above.

[0139] As described above, according to the electronic device 1 of one embodiment, even when detecting an object using directional radio waves with a sensor such as a millimeter-wave radar, the change in detection intensity depending on the direction of the object relative to the sensor can be reduced. Therefore, according to the electronic device 1 of one embodiment, the influence on the detection accuracy of the object depending on the direction relative to the sensor such as a millimeter-wave radar can be reduced. Consequently, according to the electronic device 1 of one embodiment, the accuracy of object detection by a directional sensor can be improved.

[0140] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art will find it easy to make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided. While embodiments relating to this disclosure have been described primarily in terms of apparatus, embodiments relating to this disclosure can also be realized as methods including steps performed by each component of the apparatus. Embodiments relating to this disclosure can also be realized as methods, programs, or storage media recording programs executed by a processor in the apparatus. These should also be understood to be included within the scope of this disclosure.

[0141] The embodiments described above are not limited to being implemented as electronic device 1. For example, the embodiments described above may be implemented as electronic device 1 included in electronic device 1. Furthermore, the embodiments described above may be implemented as a monitoring method by a device such as electronic device 1. Moreover, the embodiments described above may be implemented as a program executed by a device such as electronic device 1 or an information processing device (e.g., a computer), or as a storage medium or recording medium on which such a program is recorded. [Explanation of symbols]

[0142] 1 Electronic equipment 3. Detection equipment 5. Stand section 7 Grounding part 10 Controllers 20 Memory section 30 Communications Department 40 Display section 50 Hochi Department 100 sensors 101 Radio wave input section 110 Radar Control Unit 120 Transmitter 121 Signal Generation Unit 122 Synthesizers 123 Phase Control Unit 124 Amplifier 125 Transmitting Antenna 130 Receiver 131 Receiving antenna 132 LNA 133 Mixer 134 IF section 135 AD Conversion Unit 300 Imaging Unit 301 Optical input section

Claims

1. An electronic device comprising a control unit that detects an object based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave when the transmission wave is reflected by an object, The control unit adjusts the signal strength of the received signal according to the directivity of at least one of the transmitted wave and the reflected wave. The control unit, Based on the transmitted signal and the received signal which is received as a reflected wave after the transmitted wave has been reflected by the first object, the point cloud information obtained by detecting the first object, At each position in the point cloud, the signal strength of the received signal is adjusted according to the directivity of at least one of the transmitted wave and the reflected wave, The correct position information of the first object, An electronic device that performs machine learning based on this.

2. The electronic device according to claim 1, wherein the control unit adjusts the signal strength of the received signal to increase as the radiation intensity of at least one of the transmitted wave and the reflected wave decreases.

3. The electronic device according to claim 1, wherein the control unit adjusts the signal strength of the received signal to increase as the distance increases from the position where the radiation intensity of at least one of the transmitted wave and the reflected wave is strongest.

4. The electronic device according to claim 1, wherein the control unit adjusts the signal intensity of the received signal based on a two-dimensional Gaussian distribution obtained by integrating a constant proportional to the gain of at least one of the transmitted wave and the reflected wave.

5. The electronic device according to any one of claims 1 to 4, which detects the object based on a transmitted signal transmitted by a transmitting antenna array as the transmitted wave, and / or a received signal received by a receiving antenna array as the reflected wave.

6. The control unit, Based on the transmitted signal and the received signal, which is received as a reflected wave after the transmitted wave has been reflected by the second object, the point cloud information obtained by detecting the second object, At each position in the point cloud, the signal strength of the received signal is adjusted according to the directivity of at least one of the transmitted wave and the reflected wave, The results of the aforementioned machine learning, The electronic device according to claim 1, which infers the position of the second object based on the above.

7. A step of detecting an object based on a transmitted signal that is transmitted as a transmitted wave, and a received signal that is received as a reflected wave when the transmitted wave is reflected by an object, The steps include adjusting the signal strength of the received signal according to the directivity of at least one of the transmitted wave and the reflected wave, Based on the transmitted signal and the received signal which is received as a reflected wave after the transmitted wave has been reflected by the first object, the point cloud information obtained by detecting the first object, At each position in the point cloud, the signal strength of the received signal is adjusted according to the directivity of at least one of the transmitted wave and the reflected wave, The correct position information of the first object, The steps to perform machine learning based on this, A method for controlling electronic devices, including...

8. On the computer, A step of detecting an object based on a transmitted signal that is transmitted as a transmitted wave, and a received signal that is received as a reflected wave when the transmitted wave is reflected by an object, The steps include adjusting the signal strength of the received signal according to the directivity of at least one of the transmitted wave and the reflected wave, Based on the transmitted signal and the received signal which is received as a reflected wave after the transmitted wave has been reflected by the first object, the point cloud information obtained by detecting the first object, At each position in the point cloud, the signal strength of the received signal is adjusted according to the directivity of at least one of the transmitted wave and the reflected wave, The correct position information of the first object, The steps to perform machine learning based on this, A program that executes something.

Images