Electronic device, method for controlling electronic device, and program

Abstract

To provide an electronic apparatus that can easily calibrate a sensor for detecting an object and a camera for picking up an image of the object, a method for controlling an electronic apparatus, and a program.SOLUTION: A point group is obtained by detecting an object reflecting a transmission wave based on a transmission signal transmitted as the transmission wave and a reception signal received as a reflected wave being the reflected transmission wave. Based on a first coordinate point corresponding to a predetermined position of the object and a second coordinate point corresponding to the predetermined position in a picked-up image of the object in the point group, an electronic apparatus corrects the difference between the position of the object in the point group and the position of the object in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to electronic equipment, methods for controlling electronic equipment, and programs. [Background technology]

[0002] For example, in fields such as the automotive industry, technologies for measuring the distance between a vehicle and a designated object are considered important. In particular, in recent years, various radar (RADAR (Radio Detecting and Ranging)) technologies have been researched, which measure the distance to an object by transmitting radio waves such as millimeter waves and receiving reflected waves reflected from obstacles and other objects. The importance of such distance measurement technologies is expected to increase even further in the future with the development of technologies that assist drivers and technologies related to autonomous driving that automate part or all of the driving. Such technologies for measuring the distance to an object are not limited to fields such as transportation, but are expected to be used in various fields. For example, in nursing homes or medical settings, if the location of a person requiring care or nursing can be detected, it can be useful for tracking or monitoring the actions of that person.

[0003] Recently, research has been progressing on utilizing point cloud information obtained by detecting objects using sensors such as millimeter-wave radar, integrated with image (or video) information captured by digital cameras. For example, research is being conducted on a technology that detects objects with good accuracy by applying AI (Artificial Intelligence) to information that integrates point cloud information of an object detected by a millimeter-wave radar sensor with image information of the same object captured by a camera. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2009-168472 [Overview of the project] [Problems that the invention aims to solve]

[0005] As described above, when integrating point cloud information of an object detected by a millimeter-wave radar sensor with image information of the same object captured by a camera, if the optical axis of the sensor and the optical axis of the camera are not properly aligned, the integrated information may not be able to be used appropriately. Therefore, when using such technology, it is desirable that the misalignment between the optical axis of the sensor and the optical axis of the camera be easily calibrated. For example, as a technique for calibrating a laser scanner, Patent Document 1 proposes a technique that allows for the calibration of a laser scanner without the use of specialized equipment.

[0006] The purpose of this disclosure is to provide an electronic device, a control method, and a program that can easily calibrate a sensor for detecting an object and a camera for imaging said object. [Means for solving the problem]

[0007] An electronic device according to one embodiment is A point cloud is obtained by detecting an object that reflects the transmitted wave, based on the transmitted signal that is transmitted as a transmitted wave and the received signal that is received as a reflected wave of the transmitted wave. Based on the information, the region with the largest number of points among the regions divided horizontally or vertically in the point cloud information is determined to be the center position of the point cloud. From the information of the image captured of the object, the central position of the object in the image is determined, Determine whether the distance between the center position of the point cloud and the center position of the object in the image is greater than or equal to a predetermined value. The determination Based on this, the discrepancy between the position of the object in the point cloud and the position of the object in the image is corrected. Equipped with a control unit .

[0008] A control method for electronic equipment according to one embodiment is: A point cloud is obtained by detecting an object that reflects the transmitted wave, based on the transmitted signal that is transmitted as a transmitted wave and the received signal that is received as a reflected wave of the transmitted wave. Based on the information, the region with the largest number of points among the regions divided horizontally or vertically in the point cloud information is determined to be the center position of the point cloud. From the information of the image captured of the object, the central position of the object in the image is determined, Determine whether the distance between the center position of the point cloud and the center position of the object in the image is greater than or equal to a predetermined value. The determination Based on the above, it includes the step of correcting the deviation between the position of the object in the point cloud and the position of the object in the image. 。

[0009] A program according to an embodiment is to a computer A point cloud obtained by detecting an object that reflects the transmitted wave based on a transmitted signal transmitted as a transmitted wave and a received signal received as a reflected wave of the transmitted wave Based on the information, the region with the largest number of points among the regions divided horizontally or vertically in the point cloud information is determined to be the center position of the point cloud. From the information of the image captured of the object, the central position of the object in the image is determined, Determine whether the distance between the center position of the point cloud and the center position of the object in the image is greater than or equal to a predetermined value. The determination Based on the above, the step of correcting the deviation between the position of the object in the point cloud and the position of the object in the image is executed.

Effect of the Invention

[0010] According to an embodiment, it is possible to provide an electronic device, a control method of the electronic device, and a program that can easily calibrate a sensor for detecting an object and a camera for imaging the object.

Brief Description of the Drawings

[0011] [Figure 1] It is a block diagram showing a functional configuration of an electronic device according to an embodiment. [Figure 2] It is a block diagram showing a functional configuration of a sensor according to an embodiment. [Figure 3] It is a diagram showing an example of a transmitted signal transmitted by a sensor according to an embodiment. [Figure 4] It is a diagram showing an example of a configuration of a detection device according to an embodiment. [Figure 5] It is a diagram showing an example of object detection by a sensor according to an embodiment. [Figure 6]This diagram shows the equipment configuration used when performing calibration according to one embodiment. [Figure 7] This figure shows an example of operation by an electronic device according to one embodiment. [Figure 8] This figure shows an example of operation by an electronic device according to one embodiment. [Figure 9] This is a flowchart illustrating the operation of an electronic device according to one embodiment. [Figure 10] This figure shows an example of processing by an electronic device according to one embodiment. [Figure 11] This figure shows an example of processing by an electronic device according to one embodiment. [Figure 12] This figure shows an example of operation by an electronic device according to one embodiment. [Figure 13] This figure shows an example of operation by an electronic device according to one embodiment. [Figure 14] This figure shows an example of operation by an electronic device according to one embodiment. [Figure 15] This is a flowchart illustrating the operation of an electronic device according to one embodiment. [Modes for carrying out the invention]

[0012] In this disclosure, “electronic device” may mean an electric-powered device. “System” may mean a device that includes an electric-powered device. “User” may mean a person (typically a human) who uses the system and / or electronic device according to one embodiment. A user may include a person who benefits from detecting various objects by using the system and / or electronic device according to one embodiment.

[0013] An electronic device according to one embodiment described below can generate point cloud information that can be processed in two dimensions from point cloud information in a three-dimensional space detected by a sensor based on technology such as millimeter-wave radar. Furthermore, the electronic device according to one embodiment can determine whether or not there is a misalignment between the optical axis of the sensor and the optical axis of the camera, based on the point cloud information acquired from the aforementioned sensor and the image (video) information captured by the camera. Moreover, the electronic device according to one embodiment can also determine the degree of misalignment between the optical axis of the sensor and the optical axis of the camera. Therefore, the electronic device according to one embodiment can correct the misalignment between the optical axis of the sensor and the optical axis of the camera. The electronic device according to one embodiment may be made capable of detecting the presence and position of an object based on the point cloud information that can be processed in two dimensions by employing technology such as image recognition. The electronic device according to one embodiment will be described in detail below with reference to the drawings. The electronic device, method, and program of this disclosure can be used, for example, to detect people, objects, or animals present in a predetermined space such as a room, bed, bathroom, toilet, automobile, bus, train, corridor, or street.

[0014] Figure 1 is a functional block diagram schematically showing the configuration of an electronic device 1 according to one embodiment. As shown in Figure 1, the electronic device 1 according to one embodiment includes a controller 10. In one embodiment, the electronic device 1 may appropriately include at least one of, for example, a storage unit 20, a communication unit 30, a display unit 40, and a notification unit 50. The controller 10, storage unit 20, communication unit 30, display unit 40, and notification unit 50 described above may be arranged or built into any part of the electronic device 1. Furthermore, at least one of the controller 10, storage unit 20, communication unit 30, display unit 40, and notification unit 50 described above may be arranged outside the electronic device 1 and connected to each other by a wired, wireless, or a combination thereof network. In the electronic device 1 according to one embodiment, at least some of the functional units shown in Figure 1 may be omitted, and other functional units other than those shown in Figure 1 may be appropriately included.

[0015] The electronic device 1 according to one embodiment may be various types of devices. For example, the electronic device according to one embodiment may be any device, such as a specially designed terminal, a general-purpose smartphone, tablet, phablet, notebook PC, computer, or server. Furthermore, the electronic device according to one embodiment may have the function of communicating with other electronic devices, such as a mobile phone or smartphone. Here, the "other electronic devices" mentioned above may be electronic devices such as a mobile phone or smartphone, or any device such as a base station, server, dedicated terminal, or computer. The "other electronic devices" may be, for example, the sensor 100 and / or imaging unit 300 described later. Furthermore, the "other electronic devices" in this disclosure may also be devices or devices that are powered by electricity. When the electronic device according to one embodiment communicates with other electronic devices, it may communicate by wire and / or wirelessly.

[0016] As shown in Figure 1, the electronic device 1 according to one embodiment may be connected to the sensor 100 by wire and / or wirelessly. Through such a connection, the electronic device 1 according to one embodiment can acquire information about the results detected by the sensor 100. Furthermore, the electronic device 1 according to one embodiment may be connected to the imaging unit 300 by wire and / or wirelessly. Through such a connection, the electronic device 1 according to one embodiment can acquire information about the image captured by the imaging unit 300. The sensor 100 and the imaging unit 300 will be described further later.

[0017] The controller 10 controls and / or manages the entire electronic device 1, including each functional part that constitutes the electronic device 1. The controller 10 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing capabilities for performing various functions. The controller 10 may be implemented as a single processor, as several processors, or as separate processors. The processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be implemented as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented based on various other known technologies.

[0018] In one embodiment, the controller 10 may be configured as, for example, a CPU or DSP and a program executed by the CPU or DSP. The program executed by the controller 10, and the results of the processing performed by the controller 10, may be stored in, for example, a storage unit 20. The controller 10 may also include memory necessary for the operation of the controller 10 as appropriate.

[0019] In one embodiment of the electronic device 1, the controller 10 can perform various processes on information output as a result of detection by, for example, the sensor 100. For this reason, in the electronic device 1, the controller 10 may be connected to the sensor 100 by wire and / or wirelessly. Also, in one embodiment of the electronic device 1, the controller 10 can perform various processes on information (images) output as a result of imaging by, for example, the imaging unit 300. For this reason, in the electronic device 1, the controller 10 may be connected to the imaging unit 300 by wire and / or wirelessly. The operation of the controller 10 in one embodiment of the electronic device 1 will be described further later.

[0020] The storage unit 20 may function as a memory for storing various types of information. For example, the storage unit 20 may store programs executed in the controller 10 and the results of processes executed in the controller 10. The storage unit 20 may also store or accumulate detection results from the sensor 100 and / or images captured by the imaging unit 300. Furthermore, the storage unit 20 may function as the work memory of the controller 10. The storage unit 20 can be made of, for example, a semiconductor memory, but is not limited to this, and can be any storage device. For example, the storage unit 20 may be a storage medium such as a memory card inserted into the electronic device 1 according to one embodiment. Furthermore, the storage unit 20 may be made up of, for example, a hard disk drive (HDD) and / or a solid state drive (SSD). Furthermore, the storage unit 20 may be the internal memory of the CPU used as the controller 10 described later, or it may be connected to the controller 10 as a separate unit.

[0021] The memory unit 20 may store, for example, machine learning data. Here, machine learning data may be data generated by machine learning. Furthermore, machine learning may be based on AI (Artificial Intelligence) technology that enables the execution of specific tasks through training. More specifically, machine learning may be a technology in which an information processing device, such as a computer, learns a large amount of data and automatically builds an algorithm or model to perform tasks such as classification and / or prediction. In this specification, machine learning may be included as a part of AI.

[0022] In this specification, machine learning may include supervised learning, which learns the features or rules of input data based on correct answer data. It may also include unsupervised learning, which learns the features or rules of input data without correct answer data. Furthermore, machine learning may include reinforcement learning, which learns the features or rules of input data by providing rewards or punishments. In this specification, machine learning may be an arbitrary combination of supervised learning, unsupervised learning, and reinforcement learning. The concept of machine learning data in this embodiment may include an algorithm that outputs a predetermined inference (estimation) result using an algorithm learned on input data. This embodiment can use, for example, linear regression to predict the relationship between dependent and independent variables, a neural network (NN) that mathematically models the neurons of the human brain, the least squares method which calculates by squaring the error, a decision tree which structures problem solving in a tree structure, and regularization which transforms data in a predetermined way, as well as other appropriate algorithms. This embodiment may utilize deep learning, a type of neural network. Deep learning is a type of neural network, and a neural network with a deep network hierarchy is called deep learning. The computer learning phase in this disclosure may include, for example, a training phase in which parameters used for outputting results are generated.

[0023] The communication unit 30 has the function of an interface for communication by wire or wireless. The communication method performed by the communication unit 30 in one embodiment may be a wireless communication standard. For example, wireless communication standards include cellular phone communication standards such as 2G, 3G, 4G, and 5G. For example, cellular phone communication standards include LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (Registered Trademark) (Global System for Mobile communications), and PHS (Personal Handy-phone System). For example, wireless communication standards include WiMAX (Worldwide Interoperability for Microwave Access), IEEE802.11, WiFi, Bluetooth (Registered Trademark), IrDA (Infrared Data Association), and NFC (Near Field Communication). The communication unit 30 may include a modem whose communication method has been standardized by, for example, the ITU-T (International Telecommunication Union Telecommunication Standardization Sector). The communication unit 30 can support one or more of the above-mentioned communication standards.

[0024] The communication unit 30 may include, for example, an antenna for transmitting and receiving radio waves and a suitable RF unit. The communication unit 30 may also communicate wirelessly with, for example, the communication unit of another electronic device via the antenna. The communication unit 30 may also be configured as an interface such as a connector for wired connection to the outside. Since the communication unit 30 can be configured using known technologies for wireless communication, a more detailed explanation of the hardware will be omitted.

[0025] The various types of information received by the communication unit 30 may be supplied, for example, to the storage unit 20 and / or the controller 10. The various types of information received by the communication unit 30 may be stored, for example, in the memory built into the controller 10. The communication unit 30 may also transmit, for example, the processing results by the controller 10 and / or the information stored in the storage unit 20 to an external source.

[0026] The display unit 40 may be any display device, such as a liquid crystal display (LCD), an organic electro-luminescence panel (OLED), or an inorganic electro-luminescence panel (IEL). The display unit 40 may display various types of information, such as characters, graphics, or symbols. The display unit 40 may also display various GUI objects and icon images to prompt the user to operate the electronic device 1. Various data necessary for displaying information in the display unit 40 may be supplied from, for example, the controller 10 or the storage unit 20. Furthermore, if the display unit 40 includes, for example, an LCD, it may be configured to include a backlight as appropriate. In one embodiment, the display unit 40 may display information based on the results of detection by, for example, the sensor 100. In another embodiment, the display unit 40 may display information based on the results of imaging by, for example, the imaging unit 300.

[0027] The notification unit 50 may notify a predetermined warning to alert the user of the electronic device 1, etc., based on a predetermined signal output from the controller 10. The notification unit 50 may be any functional unit that stimulates at least one of the user's hearing, sight, or touch as a predetermined warning, such as sound, voice, light, text, images, and vibration. Specifically, the notification unit 50 may be at least one of the following: an audio output unit such as a buzzer or speaker, a light-emitting unit such as an LED, a display unit such as an LCD, and a tactile presentation unit such as a vibrator. In this way, the notification unit 50 may notify a predetermined warning based on a predetermined signal output from the controller 10. In one embodiment, the notification unit 50 may notify a predetermined alarm as information that acts on at least one of the hearing, sight, and touch.

[0028] The electronic device 1 shown in Figure 1 incorporates a notification unit 50. However, in one embodiment, the notification unit 50 may be provided outside the electronic device 1. In this case, the notification unit 50 and the electronic device 1 may be connected by wire, wireless, or a combination of wire and wireless.

[0029] As shown in Figure 1, at least a portion of each functional part constituting the electronic device 1 according to one embodiment may be configured by specific means in which software and hardware resources cooperate.

[0030] The sensor 100 shown in Figure 1 is configured to detect objects (targets), such as automobiles or human bodies, as point cloud information in three-dimensional space. A sensor 100 according to one embodiment will be described in more detail below.

[0031] Figure 2 is a functional block diagram schematically showing the configuration of a sensor 100 according to one embodiment. The sensor 100 shown in Figure 2 is, as an example, based on millimeter-wave radar (RADAR (Radio Detecting and Ranging)) technology (millimeter-wave radar sensor). However, the sensor 100 according to one embodiment is not limited to a millimeter-wave radar sensor. For example, the sensor 100 according to one embodiment may be a quasi-millimeter-wave radar sensor. Furthermore, the sensor 100 according to one embodiment is not limited to a millimeter-wave radar sensor or a quasi-millimeter-wave radar sensor, but may be various types of radar sensors that transmit and receive radio waves. In addition, the sensor 100 according to one embodiment may be, for example, a microwave sensor, an ultrasonic sensor, or a sensor based on technology such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0032] When measuring distance and other parameters using millimeter-wave radar, frequency-modulated continuous wave radar (hereinafter referred to as FMCW radar) is sometimes used. FMCW radar generates a transmitted signal by sweeping the frequency of the transmitted radio waves. Therefore, for example, in a millimeter-wave FMCW radar using radio waves in the 79 GHz frequency band, the radio wave frequencies used will have a frequency bandwidth of 4 GHz, such as 77 GHz to 81 GHz. Radar in the 79 GHz frequency band has the advantage of a wider usable frequency bandwidth than other millimeter-wave / quasi-millimeter-wave radars, such as those in the 24 GHz, 60 GHz, and 76 GHz frequency bands. The following describes an example of employing such an FMCW radar. The FMCW radar system used in this disclosure may include a Fast-Chirp Modulation (FCM) system that transmits chirp signals at a shorter period than usual. The signal generated by sensor 100 is not limited to an FMCW signal. The signal generated by sensor 100 may be a signal of various types other than the FMCW method. The transmitted signal sequence stored as the transmitted signal may differ depending on these various methods. For example, in the case of the FMCW radar signal described above, signals whose frequency increases and decreases with each time sample may be used. Since known techniques can be appropriately applied to the various methods described above, more detailed explanations will be omitted as appropriate.

[0033] As shown in Figure 2, the sensor 100 according to one embodiment may include a radar control unit 110, a transmitting unit 120, and a receiving unit 130. The radar control unit 110, transmitting unit 120, and receiving unit 130 described above may be placed or built into any location in the sensor 100. Furthermore, at least one of the radar control unit 110, transmitting unit 120, and receiving unit 130 described above may be placed outside the sensor 100. The sensor 100 according to one embodiment may omit at least some of the functional units shown in Figure 2, or may appropriately include other functional units other than those shown in Figure 2.

[0034] The radar control unit 110 controls and / or manages the entire sensor 100, including each functional part that constitutes the sensor 100. The radar control unit 110 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing capabilities for performing various functions. The radar control unit 110 may be implemented as a single processor, as several processors, or as separate processors. The processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be implemented as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented based on various other known technologies.

[0035] In one embodiment, the radar control unit 110 may be configured as, for example, a CPU or DSP and a program executed by the CPU or DSP. The program executed in the radar control unit 110, and the results of the processing performed in the radar control unit 110, may be stored in, for example, any storage unit built into the radar control unit 110. The radar control unit 110 may also include memory necessary for the operation of the radar control unit 110 as appropriate.

[0036] In the sensor 100 according to one embodiment, the radar control unit 110 may perform various processes as appropriate, such as distance FFT (Fast Fourier Transform) processing, velocity FFT processing, angle of arrival estimation processing, and clustering processing. Since each of these processes performed by the radar control unit 110 is known as general radar technology, a more detailed explanation will be omitted.

[0037] As shown in Figure 2, the transmitting unit 120 may include a signal generation unit 121, a synthesizer 122, a phase control unit 123, an amplifier 124, and a transmitting antenna 125. The sensor 100 according to one embodiment may include a plurality of transmitting antennas 125. In this case, the sensor 100 may also include a plurality of phase control units 123 and amplifiers 124 corresponding to each of the plurality of transmitting antennas 125. When the sensor 100 according to one embodiment includes a plurality of transmitting antennas 125, the plurality of transmitting antennas 125 may constitute a transmitting antenna array (transmitting array antenna).

[0038] As shown in Figure 2, the receiving unit 130 may include a receiving antenna 131, an LNA 132, a mixer 133, an IF unit 134, and an AD conversion unit 135. In one embodiment, the sensor 100 may include multiple receiving units 130, each corresponding to one of the multiple transmitting antennas 125.

[0039] In one embodiment of the sensor 100, the radar control unit 110 can control at least one of the transmitting unit 120 and the receiving unit 130. In this case, the radar control unit 110 may control at least one of the transmitting unit 120 and the receiving unit 130 based on various information stored in any storage unit. For example, any storage unit built into the radar control unit 110 may store various parameters for setting the range for detecting an object using the transmitted wave transmitted from the transmitting antenna 125 and the reflected wave received from the receiving antenna 131. Also, in one embodiment of the sensor 100, the radar control unit 110 may instruct the signal generation unit 121 to generate a signal, or control the signal generation unit 121 to generate a signal.

[0040] The signal generation unit 121 generates a signal (transmission signal) to be transmitted from the transmitting antenna 125 as a transmission wave, under the control of the radar control unit 110. When generating the transmission signal, the signal generation unit 121 may assign a frequency to the transmission signal, for example, based on control by the radar control unit 110. Specifically, the signal generation unit 121 may assign a frequency to the transmission signal according to parameters set by the radar control unit 110, for example. For example, the signal generation unit 121 generates a signal of a predetermined frequency in a frequency band such as 77-81 GHz by receiving frequency information from the radar control unit 110 or any storage unit. The signal generation unit 121 may include a functional unit such as a voltage-controlled oscillator (VCO).

[0041] The signal generation unit 121 may be configured as hardware having the function, or as a microcontroller, for example, or as a processor such as a CPU or DSP and a program executed by that processor. Each of the functional units described below may also be configured as hardware having the function, or, where possible, as a microcontroller, for example, or as a processor such as a CPU or DSP and a program executed by that processor.

[0042] In a sensor 100 according to one embodiment, the signal generation unit 121 may generate a transmission signal (transmission chirp signal), such as a chirp signal. In particular, the signal generation unit 121 may generate a signal whose frequency changes periodically linearly (linear chirp signal). For example, the signal generation unit 121 may generate a chirp signal whose frequency increases periodically linearly from 77 GHz to 81 GHz as time progresses. Alternatively, for example, the signal generation unit 121 may generate a signal whose frequency periodically repeats a linear increase (up chirp) and decrease (down chirp) from 77 GHz to 81 GHz as time progresses. The signal generated by the signal generation unit 121 may be pre-set in, for example, the radar control unit 110. The signal generated by the signal generation unit 121 may also be pre-stored in, for example, any memory unit. Since chirp signals used in technical fields such as radar are known, a more detailed explanation will be simplified or omitted as appropriate. The signal generated by the signal generation unit 121 is supplied to the synthesizer 122.

[0043] Figure 3 illustrates an example of a chirp signal generated by the signal generation unit 121.

[0044] In Figure 3, the horizontal axis represents elapsed time, and the vertical axis represents frequency. In the example shown in Figure 3, the signal generation unit 121 generates a linear chirp signal whose frequency changes linearly and periodically. In Figure 3, each chirp signal is shown as c1, c2, ..., c8. As shown in Figure 3, the frequency of each chirp signal increases linearly with the passage of time.

[0045] In the example shown in Figure 3, eight chirp signals, such as c1, c2, ..., c8, are included in one subframe. That is, subframe 1 and subframe 2, etc., shown in Figure 3, are each composed of eight chirp signals, such as c1, c2, ..., c8. Also, in the example shown in Figure 3, 16 subframes, such as subframe 1 to subframe 16, are included in one frame. That is, frame 1 and frame 2, etc., shown in Figure 3, are each composed of 16 subframes. Furthermore, as shown in Figure 3, a frame interval of a predetermined length may be included between frames. One frame shown in Figure 3 may be, for example, about 30 to 50 milliseconds long.

[0046] In Figure 3, frames 2 and beyond may have a similar configuration. Similarly, in Figure 3, frames 3 and beyond may have a similar configuration. In the sensor 100 according to one embodiment, the signal generation unit 121 may generate the transmission signal as any number of frames. Also, in Figure 3, some chirp signals are omitted. Thus, the relationship between the time and frequency of the transmission signal generated by the signal generation unit 121 may be stored in, for example, any memory unit.

[0047] Thus, the sensor 100 according to one embodiment may transmit a transmission signal consisting of subframes containing multiple chirp signals. Alternatively, the sensor 100 according to one embodiment may transmit a transmission signal consisting of frames containing a predetermined number of subframes.

[0048] Hereinafter, the sensor 100 will be described as transmitting a transmission signal with a frame structure as shown in Figure 3. However, the frame structure shown in Figure 3 is just one example, and the number of chirp signals included in one subframe is not limited to eight. In one embodiment, the signal generation unit 121 may generate subframes containing any number (e.g., any multiple) chirp signals. Also, the subframe structure shown in Figure 3 is just one example, and the number of subframes included in one frame is not limited to 16. In one embodiment, the signal generation unit 121 may generate a frame containing any number (e.g., any multiple) subframes. The signal generation unit 121 may generate signals of different frequencies. The signal generation unit 121 may generate multiple discrete signals with different bandwidths and frequencies f.

[0049] Returning to Figure 2, the synthesizer 122 increases the frequency of the signal generated by the signal generation unit 121 to a frequency in a predetermined frequency band. The synthesizer 122 may increase the frequency of the signal generated by the signal generation unit 121 to a frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125. The frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be set, for example, by the radar control unit 110. Alternatively, the frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be stored, for example, in any memory unit. The signal whose frequency has been increased by the synthesizer 122 is supplied to the phase control unit 123 and the mixer 133. If there are multiple phase control units 123, the signal whose frequency has been increased by the synthesizer 122 may be supplied to each of the multiple phase control units 123. Furthermore, if there are multiple receiving units 130, the signal whose frequency has been increased by the synthesizer 122 may be supplied to each mixer 133 in the multiple receiving units 130.

[0050] The phase control unit 123 controls the phase of the transmission signal supplied from the synthesizer 122. Specifically, the phase control unit 123 may adjust the phase of the transmission signal by appropriately advancing or delaying the phase of the signal supplied from the synthesizer 122, for example, based on control by the radar control unit 110. In this case, the phase control unit 123 may adjust the phase of each transmission signal based on the path difference of the respective transmission waves transmitted from the multiple transmission antennas 125. By appropriately adjusting the phase of each transmission signal, the transmission waves transmitted from the multiple transmission antennas 125 reinforce each other in a predetermined direction to form a beam (beamforming). In this case, the correlation between the direction of beamforming and the amount of phase to be controlled for the transmission signals transmitted by each of the multiple transmission antennas 125 may be stored in, for example, an arbitrary memory unit. The transmission signal whose phase has been controlled by the phase control unit 123 is supplied to the amplifier 124.

[0051] The amplifier 124 amplifies the power of the transmitted signal supplied from the phase control unit 123, for example, based on control by the radar control unit 110. If the sensor 100 has multiple transmitting antennas 125, multiple amplifiers 124 may each amplify the power of the transmitted signal supplied from the corresponding phase control unit 123, for example, based on control by the radar control unit 110. The technique for amplifying the power of the transmitted signal is already known, so a more detailed explanation is omitted. The amplifier 124 is connected to the transmitting antenna 125.

[0052] The transmitting antenna 125 outputs (transmits) the transmission signal amplified by the amplifier 124 as a transmission wave. If the sensor 100 has multiple transmitting antennas 125, each of the multiple transmitting antennas 125 may output (transmit) the transmission signal amplified by the corresponding amplifier 124 as a transmission wave. The transmitting antenna 125 can be configured in the same way as transmitting antennas used in known radar technology, so a more detailed explanation is omitted.

[0053] In this way, the sensor 100 according to one embodiment includes a transmitting antenna 125 and can transmit a transmission signal (e.g., a transmitting chirp signal) as a transmission wave from the transmitting antenna 125. Here, at least one of the functional parts constituting the sensor 100 may be housed in a single housing. In this case, the single housing may be constructed in a way that prevents it from being easily opened. For example, the transmitting antenna 125, the receiving antenna 131, and the amplifier 124 may be housed in a single housing, and this housing may be constructed in a way that prevents it from being easily opened. Furthermore,

[0054] The sensor 100 shown in Figure 2 is an example that includes one transmitting antenna 125. However, in one embodiment, the sensor 100 may include any number of transmitting antennas 125. On the other hand, in one embodiment, the sensor 100 may include multiple transmitting antennas 125 so that the transmitted waves transmitted from the transmitting antennas 125 form a beam in a predetermined direction. In one embodiment, the sensor 100 may include any number of transmitting antennas 125. In this case, the sensor 100 may also include multiple phase control units 123 and amplifiers 124, corresponding to the multiple transmitting antennas 125. The multiple phase control units 123 may each control the phase of multiple transmitted waves supplied from the synthesizer 122 and transmitted from the multiple transmitting antennas 125. The multiple amplifiers 124 may each amplify the power of multiple transmitted signals transmitted from the multiple transmitting antennas 125. In this case, the sensor 100 may be configured to include multiple transmitting antennas. Thus, if the sensor 100 is equipped with multiple transmitting antennas 125, it may also be configured to include multiple functional units necessary for transmitting a wave from each of the multiple transmitting antennas 125.

[0055] The receiving antenna 131 receives reflected waves. The reflected waves may be those that have been reflected by a predetermined object 200 from the transmitted waves. The receiving antenna 131 may be configured to include multiple antennas. If the sensor 100 according to one embodiment includes multiple receiving antennas 131, the multiple receiving antennas 131 may constitute a receiving antenna array (receiving array antenna). The receiving antenna 131 can be configured in the same way as receiving antennas used in known radar technology, so a more detailed explanation is omitted. The receiving antenna 131 is connected to the LNA 132. The received signal based on the reflected waves received by the receiving antenna 131 is supplied to the LNA 132. Thus, if the sensor 100 is equipped with multiple receiving antennas 131, it may also be configured to include multiple functional units necessary for receiving and processing reflected waves from the multiple receiving antennas 131.

[0056] In one embodiment, the sensor 100 can receive reflected waves from a predetermined object 200, which are transmitted as a transmission signal (transmitted chirp signal), such as a chirp signal, from a plurality of receiving antennas 131. When a transmitted chirp signal is transmitted as a transmission wave, the received signal based on the received reflected wave is also referred to as a received chirp signal. That is, the sensor 100 receives a received signal (for example, a received chirp signal) as a reflected wave from the receiving antennas 131.

[0057] The LNA132 amplifies the received signal based on the reflected wave received by the receiving antenna 131 with low noise. The LNA132 functions as a low-noise amplifier, amplifying the received signal supplied from the receiving antenna 131 with low noise. The received signal amplified by the LNA132 is supplied to the mixer 133.

[0058] Mixer 133 generates a beat signal by mixing (multiplying) the RF frequency received signal supplied from LNA 132 with the transmitted signal supplied from synthesizer 122. The beat signal mixed by mixer 133 is supplied to IF unit 134.

[0059] The IF unit 134 performs frequency conversion on the beat signal supplied from the mixer 133, thereby reducing the frequency of the beat signal to an intermediate frequency (IF (Intermediate Frequency) frequency). The beat signal whose frequency has been reduced by the IF unit 134 is supplied to the AD conversion unit 135.

[0060] The AD conversion unit 135 digitizes the analog beat signal supplied from the IF unit 134. The AD conversion unit 135 may be composed of any analog-to-digital converter (ADC). The beat signal digitized by the AD conversion unit 135 is supplied to the radar control unit 110. If there are multiple receiving units 130, the beat signals digitized by each of the multiple AD conversion units 135 may be supplied to the radar control unit 110.

[0061] The radar control unit 110 may perform FFT processing (hereinafter referred to as "distance FFT processing") on the beat signal digitized by the AD conversion unit 135. For example, the radar control unit 110 may perform FFT processing on the complex signal supplied from the AD conversion unit 135. The beat signal digitized by the AD conversion unit 135 can be represented as a time change in signal strength (power). By performing FFT processing on such a beat signal, the radar control unit 110 can represent it as a signal strength (power) corresponding to each frequency. By performing distance FFT processing in the radar control unit 110, a complex signal corresponding to distance can be obtained based on the beat signal digitized by the AD conversion unit 135.

[0062] The radar control unit 110 may determine that a predetermined object 200 exists at the distance corresponding to the peak if the peak in the result obtained by distance FFT processing is above a predetermined threshold. For example, a method is known in which, such as detection processing using a constant false alarm rate (CFAR), if a peak value above a threshold is detected from the average power or amplitude of the disturbance signal, it is determined that an object that reflects the transmitted wave (reflecting object) exists.

[0063] Thus, according to one embodiment, the sensor 100 can detect an object 200 that reflects a transmitted wave as a target, based on the transmitted signal that is transmitted as a transmitted wave and the received signal that is received as a reflected wave.

[0064] The radar control unit 110 can estimate the distance to a predetermined object based on a single chirp signal (for example, c1 shown in Figure 3). That is, the sensor 100 can measure (estimate) the distance between the sensor 100 and the predetermined object 200 by performing distance FFT processing. Since the technique of measuring (estimating) the distance to a predetermined object by performing FFT processing on a beat signal is known, a more detailed explanation will be simplified or omitted as appropriate.

[0065] Furthermore, the radar control unit 110 may perform an FFT (Fast FFT) operation on the beat signal that has undergone distance FFT processing (hereinafter referred to as "velocity FFT processing" as appropriate). For example, the radar control unit 110 may perform an FFT operation on the complex signal that has undergone distance FFT processing. The radar control unit 110 can estimate the relative velocity with a predetermined object based on a subframe of the chirp signal (for example, subframe 1 shown in Figure 3). By performing velocity FFT processing on multiple chirp signals in the radar control unit 110, a complex signal corresponding to the relative velocity is obtained based on the complex signal corresponding to the distance obtained by the distance FFT processing.

[0066] As described above, applying distance FFT processing to a beat signal can generate multiple vectors. By determining the phase of the peaks in the results of velocity FFT processing on these multiple vectors, the relative velocity with a given object can be estimated. In other words, electronic device 1 can measure (estimate) the relative velocity between sensor 100 and a given object 200 by performing velocity FFT processing. The technique of measuring (estimating) the relative velocity with a given object by performing velocity FFT processing on the results of distance FFT processing is well known, so a more detailed explanation will be simplified or omitted as appropriate.

[0067] In typical FMCW radar technology, the presence or absence of a target can be determined based on the results of extracting the beat frequency from the received signal and performing a Fast Fourier Transform (FFT). However, the results obtained by extracting the beat frequency from the received signal and performing a FFT include noise components such as clutter (unwanted reflections). Therefore, it may be possible to remove the noise components from the processed received signal and perform processing to extract only the target signal.

[0068] Furthermore, the radar control unit 110 may estimate the direction (angle of arrival) from which the reflected wave arrives from a predetermined object 200 based on the determination of whether or not a target exists. The radar control unit 110 may perform the estimation of the angle of arrival for points where it has been determined that a target exists. The sensor 100 can estimate the direction from which the reflected wave arrives by receiving the reflected wave from a plurality of receiving antennas 131. For example, the plurality of receiving antennas 131 are arranged at predetermined intervals. In this case, the transmitted wave transmitted from the transmitting antenna 125 is reflected by the predetermined object 200 and becomes a reflected wave, and the plurality of receiving antennas 131 arranged at predetermined intervals each receive the reflected wave R. The radar control unit 110 can then estimate the direction from which the reflected wave arrives at the receiving antennas 131 based on the phase of the reflected wave received by each of the plurality of receiving antennas 131 and the path difference of each reflected wave. That is, the sensor 100 can measure (estimate) the angle of arrival θ, which indicates the direction from which the reflected wave reflected by the target arrives, based on the results of velocity FFT processing.

[0069] Various techniques have been proposed for estimating the direction from which a reflected wave R arrives, based on the results of velocity FFT processing. For example, known algorithms for estimating the direction of arrival include MUSIC (Multiple Signal Classification) and ESPRIT (Estimation of Signal Parameters via Rotational Invariance Technique). Therefore, more detailed explanations of known techniques will be simplified or omitted as appropriate.

[0070] The radar control unit 110 detects objects within the range from which the transmitted wave was transmitted, based on at least one of distance FFT processing, velocity FFT processing, and angle of arrival estimation. The radar control unit 110 may also perform object detection by, for example, clustering processing based on the supplied distance information, velocity information, and angle information. Known algorithms for clustering data include, for example, DBSCAN (Density-based spatial clustering of applications with noise). In the clustering process, for example, the average power of the points constituting the detected object may be calculated.

[0071] As described above, the sensor 100 can detect objects that reflect transmitted waves in three-dimensional space as point cloud information. That is, in one embodiment, based on the detection result output from the sensor 100, it is possible to determine (detect) whether or not an object that reflects transmitted waves exists at a certain coordinate in three-dimensional space. In another embodiment, the sensor 100 can detect the signal strength and velocity of each point in three-dimensional space. As explained above, the sensor 100 according to one embodiment may detect objects that reflect transmitted waves as point cloud information in three-dimensional space based on the transmitted signal transmitted as a transmitted wave and the received signal received as a reflected wave from which the transmitted wave has been reflected. In this disclosure, the sensor 100 may detect objects as point cloud information in two-dimensional space.

[0072] Furthermore, as shown in Figure 1, the electronic device 1 according to one embodiment may include an imaging unit 300. The electronic device 1 and the imaging unit 300 may be connected by wire, wireless, or a combination of wire and wireless.

[0073] The imaging unit 300 may include an image sensor that electronically captures images, such as a digital camera. The imaging unit 300 may include an image sensor that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 300 may capture images of objects located in front of it. Here, objects located in front of the imaging unit 300 may be, for example, cars, people, and / or any objects in the surroundings. The imaging unit 300 may convert the captured image into a signal and transmit it to the electronic device 1. For example, the imaging unit 300 may transmit a signal based on the captured image to the extraction unit 11, storage unit 20, and / or controller 10 of the electronic device 1. The imaging unit 300 is not limited to imaging devices such as digital cameras, but may be any device that can capture images of objects. The imaging unit 300 may be, for example, a LIDAR (Light Detection And Ranging).

[0074] In one embodiment, the imaging unit 300 may capture still images, for example, at predetermined intervals (e.g., 15 frames per second). In another embodiment, the imaging unit 300 may capture, for example, a continuous video.

[0075] Next, the arrangement of the sensor 100 and imaging unit 300 connected to the electronic device 1 according to one embodiment will be described.

[0076] Figures 4(A) and 4(B) show examples of the configuration of a detection device in which the sensor 100 and the imaging unit 300 are arranged.

[0077] Figure 4(A) is a front view showing an example of a detection device 3 according to one embodiment, viewed from the front. Figure 4(B) is a side view showing an example of a detection device 3 according to one embodiment, viewed from the side (left). The coordinate axes shown in Figures 4(A) and 4(B) are aligned with the coordinate axes showing the propagation direction of the transmitted and / or reflected waves of the sensor 100 shown in Figure 2.

[0078] As shown in Figures 4(A) and 4(B), the detection device 3 according to one embodiment may include a sensor 100 and an imaging unit 300. Also, as shown in Figures 4(A) and 4(B), the detection device 3 according to one embodiment may appropriately include at least one of a stand 5 and a grounding unit 7. Furthermore, the detection device 3 according to one embodiment may be placed on any other device or the housing of another device without including at least one of the stand 5 and the grounding unit 7.

[0079] The sensor 100 shown in Figures 4(A) and 4(B) may be the same sensor 100 described in Figures 1 and / or 2. As shown in Figures 4(A) and 4(B), the sensor 100 may include a radio wave input unit 101 that receives reflected waves when a transmitted wave is reflected by an object. As shown in Figure 4(B), the radio wave input unit 101 may be oriented toward the optical axis Ra of the sensor 100. Here, the optical axis of the sensor 100 may be, for example, the direction perpendicular to the plane on which at least one of the transmitting antenna 125 and the receiving antenna 131 of the sensor 100 is installed. Alternatively, if the sensor 100 includes multiple transmitting antennas 125 and receiving antennas 131, the optical axis of the sensor 100 may be the direction perpendicular to the plane on which at least one of the multiple antennas is installed. With this configuration, the sensor 100 can transmit a transmitted wave and / or receive reflected waves around the optical axis Ra. That is, the sensor 100 can detect objects as a point cloud within a range centered on the optical axis Ra.

[0080] One embodiment of the sensor 100 may, for example, have directionality. That is, the sensor 100 may detect objects using directional radio waves. Here, directionality may refer to the relationship between the radiation direction and radiation intensity of radio waves as a characteristic of the antenna. Whether or not there is directionality is related to the application of the antenna. A highly directive antenna strongly radiates radio waves in a specific direction. The directionality may be the same for both transmission and reception. The electric field strength of the radio waves radiated by the antenna can be expressed in decibels (dB) as the antenna gain. By having directionality, one embodiment of the sensor 100 may, for example, have a main lobe (main beam) in the direction of the optical axis Ra shown in Figure 4(B). That is, one embodiment of the sensor 100 may, for example, have the strongest radiation level in the direction of the optical axis Ra.

[0081] Furthermore, the imaging unit 300 shown in Figures 4(A) and 4(B) may be the same imaging unit 300 described in Figure 1. As shown in Figures 4(A) and 4(B), the imaging unit 300 may include an optical input unit 301 that receives light reflected by an object. As shown in Figure 4(B), the optical input unit 301 may be oriented toward the optical axis La of the imaging unit 300. Alternatively, the optical input unit 301 may be located where the lens is positioned in the imaging unit 300. Here, the optical axis of the imaging unit 300 may be, for example, the direction perpendicular to the plane on which the light-receiving element (or image sensor) used for imaging is installed in the imaging unit 300. With this configuration, the imaging unit 300 can capture an image centered on the optical axis La.

[0082] As shown in Figures 4(A) and 4(B), the stand 5 maintains the sensor 100 and imaging unit 300 of the detection device 3 at a predetermined height from the ground. The stand 5 may maintain the sensor 100 at a height that makes it easy for the sensor 100 to detect a predetermined object. The stand 5 may also maintain the sensor 100 at a height that makes it easy for the imaging unit 300 to image a predetermined object. The stand 5 may be equipped with a mechanism that allows the sensor 100 and imaging unit 300 to be adjusted, for example, in the height direction, of the detection device 3.

[0083] As shown in Figures 4(A) and 4(B), the grounding portion 7 fixes the sensor 100 and imaging unit 300 of the detection device 3 to the ground surface. The grounding portion 7 can be configured in various ways, such as being shaped like a pedestal, in order to stabilize the detection device 3 which includes the sensor 100 and imaging unit 300.

[0084] As shown in Figures 4(A) and 4(B), in the detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged adjacent to each other in their vicinity. In the example shown in Figures 4(A) and 4(B), the sensor 100 and the imaging unit 300 are arranged adjacent to each other in the vertical direction. In the detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged adjacent to each other, for example, in the left-right direction or diagonally.

[0085] Furthermore, as shown in Figure 4(B), in the detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged so that their respective optical axes Ra and La are parallel. That is, in the electronic device 1 according to one embodiment, the point cloud information from the sensor 100 and the image information from the imaging unit 300 may be used with the imaging unit 300's optical axis La being set parallel to the sensor 100's optical axis Ra.

[0086] Furthermore, as shown in Figure 4(B), in the detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged such that the distance between their respective optical axes Ra and La is maintained at a distance G. By arranging them in this way, the point cloud information from the sensor 100 and the image information from the imaging unit 300 will be shifted by a distance G from each other. For example, in the arrangement shown in Figure 4(B), the image information from the imaging unit 300 will be shifted upward by a distance G from the point cloud information from the sensor 100. Also, in the arrangement shown in Figure 4(B), the point cloud information from the sensor 100 will be shifted downward by a distance G from the image information from the imaging unit 300.

[0087] Therefore, in the arrangement shown in Figure 4(B), for example, by correcting the point cloud information from sensor 100 to shift upward by a distance G, the position of the point cloud information from sensor 100 can be made to correspond to the position of the image information from imaging unit 300. Also, in the arrangement shown in Figure 4(B), for example, by correcting the image information from imaging unit 300 to shift downward by a distance G, the position of the image information from imaging unit 300 can be made to correspond to the position of the point cloud information from sensor 100. In this way, the electronic device 1 may correct at least one of the point cloud information from sensor 100 and the image information from imaging unit 300 so that the point cloud information from sensor 100 and the image information from imaging unit 300 correspond to each other in terms of position.

[0088] In other words, in the electronic device 1 according to one embodiment, at least one of the point cloud information obtained by the sensor 100 detecting an object (target) and the image information obtained by the imaging unit 300 imaging the object (target) may be corrected or calibrated. The electronic device 1 according to one embodiment may use information obtained by correcting or calibrating the point cloud information from the sensor 100 and the image information from the imaging unit 300 to match. The correction or calibration of the point cloud information from the sensor 100 and the image information from the imaging unit 300 will be described further later.

[0089] Furthermore, it is conceivable that the detection range (angle) of the point cloud by the sensor 100 and the image acquisition range (angle or field of view) of the image by the imaging unit 300 may not be the same. In such cases, the electronic device 1 may adjust the wider range (angle) of the two so that the point cloud information from the sensor 100 and the image information from the imaging unit 300 correspond to each other in terms of position. In other words, the electronic device 1 may use only the information of the overlapping range between the imaging range of the imaging unit 300 and the detectable range of the sensor 100, and delete or ignore the information of the non-overlapping ranges.

[0090] As described above, in the electronic device 1 according to one embodiment, the image information obtained by the imaging unit 300 imaging an object may be information that corresponds in position to the point cloud information obtained by the sensor 100 detecting the object.

[0091] The following describes object detection using the detection device 3 shown in Figures 4(A) and 4(B). In the detection device 3, the sensor 100 has directionality as described above, and therefore has a main lobe (main beam) in the direction of the optical axis Ra shown in Figure 4(B). That is, the sensor 100 according to one embodiment has the strongest radiation level in the direction of the optical axis Ra. In this case, the sensor 100 can output a point cloud representing the detected object by irradiating a directional transmission wave and receiving the reflected wave. Here, the sensor 100 that performs such detection has the characteristic of being most likely to detect the point cloud of an object in the direction of the optical axis Ra shown in Figure 4(B), i.e., in front of the sensor 100. Therefore, for example, even if two objects with the same reflection characteristics are equidistant from the sensor 100, the number of point clouds output by detection will differ between the object located in front of the sensor 100 and the object located elsewhere in front of the sensor 100.

[0092] Figures 5(A) and 5(B) illustrate the presence or absence of a misalignment between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300. Here, as an example, we assume a case where a car located a short distance away from the detection device 3 in the positive X-axis direction is detected by the detection device 3 as shown in Figures 4(A) and 4(B).

[0093] For example, consider the case where the optical axis Ra of sensor 100 is parallel to the optical axis La of imaging unit 300. In this case, as shown in Figure 5(A), a point cloud PT based on the detection of object 200 by sensor 100 is output at a position corresponding to the area of ​​object 200 (in this case, a car) included in the image IM captured by imaging unit 300. In such a case, since the optical axis Ra of sensor 100 and the optical axis La of imaging unit 300 are not misaligned, there is no need to calibrate these optical axes. It can be assumed that coordinates are set in the image IM shown in Figures 5(A) and 5(B). In addition, in the image IM shown in Figures 5(A) and 5(B), the first coordinates of the points corresponding to predetermined positions in the point cloud PT based on the detection of object 200 by sensor 100 are converted to coordinates on the image IM and projected onto the image IM. The control unit (e.g., controller 10) of this disclosure may determine whether the distance between a second coordinate point corresponding to a predetermined position of the object 200 among the points on the image in which the object 200 was captured, and a first coordinate point of the point cloud PT based on the detection of the object 200 by the sensor 100, and the corresponding point on the image in which the object 200 was captured, is greater than or equal to a predetermined value. The image IM may have coordinates set on two mutually orthogonal axes, for example.

[0094] On the other hand, consider a case where, for example, the direction of the optical axis Ra of the sensor 100 is not parallel to the direction of the optical axis La of the imaging unit 300. In this case, as shown in Figure 5(B), a point cloud PT based on the detection of object 200 by the sensor 100 is output at a position shifted from the area of ​​object 200 (in this case, a car) included in the image IM captured by the imaging unit 300. In such a case, since the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300 are misaligned, it is desirable to calibrate these optical axes. More specifically, the direction of at least one of the optical axes Ra and optical axis La may be changed by changing at least one of the angles at which the sensor 100 is positioned on the detection device 3 and the angles at which the imaging unit 300 is positioned on the detection device 3.

[0095] Without such calibration, it may become impossible to properly utilize the information obtained by integrating the point cloud information output from the sensor 100 and the image information captured by the imaging unit 300. For example, even if AI technology is applied to this integrated information, it may become difficult to detect objects detected by the sensor 100 with good accuracy. Therefore, the electronic device 1 according to one embodiment determines whether or not such calibration is necessary, and performs calibration if necessary.

[0096] In the examples shown in Figures 5(A) and 5(B), the presence or absence of optical axis misalignment and the necessity of calibration were explained only in the left-right direction, i.e., the horizontal direction (Y-axis direction). That is, in the examples shown in Figures 5(A) and 5(B), it is assumed that there is no optical axis misalignment in the vertical direction (Z-axis direction) because calibration in the up-down direction, i.e., the vertical direction (Z-axis direction) has already been performed as appropriate. As shown in Figure 4(B), if the distance between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300 is maintained at a distance G, then if vertical direction (Z-axis direction) calibration is not performed, the position of the point cloud PT shown in Figure 5(B) will be output with a further vertical shift corresponding to the distance G. Even such vertical direction (Z-axis direction) misalignment can be calibrated according to the same principle as horizontal direction (Y-axis direction) misalignment according to the electronic device 1 of one embodiment.

[0097] Therefore, to simplify the explanation below, we will assume that in the electronic device 1 according to one embodiment, the optical axis in the vertical direction (Z-axis direction) is not misaligned, for example, by performing appropriate calibration in the vertical direction (Z-axis direction). In other words, the detection and calibration of misalignment of the optical axis only in the horizontal direction (Y-axis direction) will be described below in the electronic device 1 according to one embodiment.

[0098] Figure 6 shows the arrangement of detection equipment 3 and other components when performing calibration with the electronic device 1 according to one embodiment. As shown in Figure 6, when performing calibration with the electronic device 1 according to one embodiment, a flat plate BD is placed as a calibration board at a position slightly away from the detection equipment 3 shown in Figures 4(A) and 4(B) in the positive X-axis direction. In one embodiment, the flat plate BD may be capable of being imaged by the imaging unit 300 and detected by the sensor 100. For example, the flat plate BD may have a planar portion that includes a component perpendicular to the X-axis direction as shown in Figure 6, and this planar portion may reflect the transmitted wave transmitted from the sensor 100 to the sensor 100 and be imaged by the imaging unit 300.

[0099] The size of the flat plate BD is not particularly limited, but it may be sized such that, for example, the entire flat portion of the flat plate BD is captured by the imaging unit 300. The size of the flat plate BD may also be sized such that, for example, the entire flat portion of the flat plate BD is detected by the sensor 100. Furthermore, the size of the flat plate BD may be sized such that, when the entire flat portion of the flat plate BD is captured by the imaging unit 300, the area occupied by the flat plate BD in the captured image does not become excessively small. Additionally, the distance between the detection device 3 and the flat plate BD (the distance in the X-axis direction shown in Figure 6) may be adjusted so that the size of the flat portion of the flat plate BD captured by the imaging unit 300 and / or the size of the flat portion of the flat plate BD detected by the sensor 100 are appropriate.

[0100] The material of the flat plate BD is not particularly limited, but it may be a material that can be well detected by the sensor 100, such as a millimeter-wave radar. For example, insulating materials may not be used as the material of the flat plate BD. Also, the surface of the flat plate BD that reflects the transmitted waves transmitted by the radar 100 may be finished in such a way that the intensity of the reflected waves is not weakened more than necessary. For example, the surface of the flat plate BD that reflects the transmitted waves transmitted by the radar 100 may be a surface with few irregularities or no irregularities at all. Also, the shape of the flat plate BD is not particularly limited, but it may have a planar portion of a shape based on a rectangle, for example. In the following description, the shape of the flat plate BD will be rectangular as an example.

[0101] Next, the operating principle of the electronic device 1 according to one embodiment will be described.

[0102] Figures 7(A) and 7(B), similar to Figures 5(A) and 5(B), show whether or not there is a misalignment between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300. Here, we assume a case where the detection device 3 detects a flat plate BD located slightly away from the detection device 3 in the positive X-axis direction, as shown in the configuration in Figure 6.

[0103] For example, suppose the optical axis Ra of sensor 100 is parallel to the optical axis La of imaging unit 300. In this case, as shown in Figure 7(A), a point cloud PT based on the detection of the flat plate BD by sensor 100 is output at a position corresponding to the area near the center of the flat plate BD included in the image IM captured by imaging unit 300. In such a case, since the optical axis Ra of sensor 100 and the optical axis La of imaging unit 300 are not misaligned, it is not necessary to calibrate these optical axes.

[0104] On the other hand, consider a case where, for example, the direction of the optical axis Ra of the sensor 100 is not parallel to the direction of the optical axis La of the imaging unit 300. In this case, as shown in Figure 7(B), the point cloud PT based on the detection of the flat plate BD by the sensor 100 is output at a position shifted from the area near the center of the flat plate BD included in the image IM captured by the imaging unit 300. In such a case, since the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300 are misaligned, it is desirable to perform calibration of these optical axes.

[0105] Therefore, in order to perform optical axis calibration, the electronic device 1 according to one embodiment determines the extent of the misalignment between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300. The controller 10 of the electronic device 1 determines the region with the most points in the point cloud PT based on the detection of the flat plate BD by the sensor 100 as shown in Figure 7, among the regions divided at predetermined intervals in the direction to be calibrated (in this case, the horizontal direction).

[0106] Figure 8 illustrates the process for determining the degree of misalignment between the optical axis Ra of sensor 100 and the optical axis La of imaging unit 300. In the point cloud PT based on the detection of a flat plate BD by sensor 100 as shown in Figure 7, the region with the most points among the regions divided horizontally at minute intervals is the region Cm shown in Figure 8. Controller 10 may determine this region Cm as the horizontal center position of the point cloud PT based on the detection of the flat plate BD by sensor 100. Hereinafter, the center position determined in this way will also be referred to as the "center position of the point cloud corresponding to the flat plate BD" or the "center position of the point cloud". Controller 10 may also determine the center position of the point cloud corresponding to the flat plate BD as the position in which the optical axis Ra of sensor 100 points. The interval of the regions divided in the direction of calibration as described above may be adjusted appropriately so that the center position of the point cloud corresponding to the flat plate BD can be properly determined.

[0107] Furthermore, the controller 10 determines the center position of the plane BD in the image IM captured by the imaging unit 300 in the direction of calibration (in this case, the horizontal direction). Hereinafter, the center position determined in this way will also be referred to as the "center position of the image on the plane BD" or the "center position of the image". The controller 10 may then determine the center position of the image on the plane BD to be the position in which the optical axis La of the imaging unit 300 is facing.

[0108] In one embodiment, the controller 10 may determine that the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300 are not misaligned (in this case, horizontally) if the center position of the point cloud corresponding to the flat plate BD coincides with the center position of the image on the flat plate BD. In this case, the controller 10 may determine that it is not necessary to perform calibration between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300. On the other hand, the controller 10 may determine that the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300 are misaligned (in this case, horizontally) if the center position of the point cloud corresponding to the flat plate BD does not coincide with the center position of the image on the flat plate BD. In this case, the controller 10 may determine that it is necessary to perform calibration between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300.

[0109] Furthermore, in one embodiment, the controller 10 may determine that the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300 are not misaligned (in this case, horizontally) if the difference between the center position of the point cloud corresponding to the flat plate BD and the center position of the image on the flat plate BD is within a predetermined range. In this case, the controller 10 may determine that it is not necessary to perform calibration between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300. On the other hand, if the difference between the center position of the point cloud corresponding to the flat plate BD and the center position of the image on the flat plate BD exceeds a predetermined range, the controller 10 may determine that the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300 are misaligned (in this case, horizontally). In this case, the controller 10 may determine that it is necessary to perform calibration between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300.

[0110] Furthermore, the degree of deviation (amount of deviation) between the center position of the point cloud corresponding to the flat plate BD and the center position of the image on the flat plate BD may be used as a correction value for calibration between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300. In one embodiment, the controller 10 may correct the coordinates of the point cloud output from the sensor 100 based on the amount of deviation described above. Such software-based correction allows calibration to be performed without changing the physical arrangement of the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300.

[0111] In the examples shown in Figures 7(B) and 8, the position of the center of the image IM (the entire image IM) captured by the imaging unit 300 coincides with the position of the center of the flat plate BD in the image IM captured by the imaging unit 300. However, the position of the center of the image IM captured by the imaging unit 300 and the position of the center of the flat plate BD in the image IM captured by the imaging unit 300 do not necessarily have to coincide. For example, when an object having a planar portion of a certain size, such as a flat plate BD, is detected by the sensor 100, the point cloud obtained by the detection becomes denser the closer it is to the center of the planar portion. Therefore, even if the position of the center of the image IM captured by the imaging unit 300 and the position of the center of the flat plate BD in the image IM captured by the imaging unit 300 do not coincide, the controller 10 can still determine the position of the center of the image of the flat plate BD.

[0112] Figure 9 is a flowchart illustrating the operation of the electronic device 1 according to one embodiment. The operation shown in Figure 9 may be initiated using, for example, a detection device 3 with the arrangement configuration shown in Figure 6. Alternatively, the operation shown in Figure 9 may be initiated when the electronic device 1 determines whether or not it is necessary to perform calibration between the optical axis Ra of the sensor 100 and the optical axis La of the imaging unit 300.

[0113] At the start of the operation shown in Figure 9, the imaging unit 300 is assumed to be capturing an image of the flat plate BD. Also, at the start of the operation shown in Figure 9, the sensor 100 is assumed to be outputting point cloud information obtained by detecting an object (flat plate BD) that reflects the transmitted wave, based on the transmitted signal transmitted as the transmitted wave and the received signal received as the reflected wave of the transmitted wave.

[0114] Here, the radar control unit 110 may perform predetermined signal processing on the beat signal based on the transmitted wave and the reflected wave to generate point cloud information (point cloud data) based on the detection of an object. In this case, the radar control unit 110 may perform at least one of the above-mentioned signal processing, such as distance FFT processing, velocity FFT processing, detection processing using constant false alarm probability (CFAR), and predetermined clustering processing. Furthermore, the radar control unit 110 is not limited to the above-mentioned signal processing and may perform any processing to generate point cloud data based on object detection. For example, various processing methods for generating point cloud data based on object detection are known in technologies such as millimeter-wave radar, so a more detailed explanation will be omitted.

[0115] The radar control unit 110 may generate point cloud data such as the point cloud PT shown in Figure 7(B) as point cloud data corresponding to the object (flat plate BD) shown in Figure 7(B). In this way, the sensor 100 outputs point cloud data based on object detection. The point cloud data output from the sensor 100 in this manner may be input to the controller 10 of the electronic device 1.

[0116] When the operation shown in Figure 9 begins, the controller 10 acquires the image captured by the imaging unit 300 (step S11). The image acquired in step S11 is assumed to be an image of the flat plate BD.

[0117] Next, the controller 10 acquires point cloud information (point cloud data) output from the sensor 100 (step S12). The point cloud data acquired in step S12 corresponds to the position of the flat plate BD detected by the sensor 100.

[0118] The point cloud data acquired in step S12 is information indicating the three-dimensional position of an object detected in three-dimensional space. Therefore, the controller 10 of the electronic device 1 converts the three-dimensional point cloud data acquired in step S12 into two-dimensional data (step S13). In step S13, the controller 10 may generate point cloud information that can be processed in two dimensions from the point cloud information in three-dimensional space output by the sensor 100.

[0119] Figure 10 illustrates the point cloud information in three-dimensional space output by the sensor 100. Figure 10 shows an example of a situation in which an object (in this case, a person) Tm located at a certain location is detected three-dimensionally by the sensor 100 installed at the origin O.

[0120] As shown in Figure 10, with the position where the sensor 100 is installed as the reference point (origin O), the direction approaching object Tm is defined as the positive X-axis, and the direction moving away from object Tm is defined as the negative X-axis. Also, as shown in Figure 10, with the position where the sensor 100 is installed as the reference point (origin O), the right side of the sensor 100 is defined as the positive Y-axis, and the left side of the sensor 100 is defined as the negative Y-axis. Furthermore, as shown in Figure 10, with the position where the sensor 100 is installed as the reference point (origin O), the upper side of the sensor 100 is defined as the positive Z-axis, and the lower side of the sensor 100 is defined as the negative Z-axis. That is, in Figure 10, lx represents the distance in the depth direction, ly represents the distance in the horizontal direction, and lz represents the distance in the vertical (height) direction. In the situation shown in Figure 10, the output of the sensor 100 may be 4-channel data of position (X, Y, Z) in 3D space (signal strength and velocity) at a given moment. In step S13 shown in Figure 9, the controller 10 converts the information detected in this three-dimensional space into two-dimensional information.

[0121] Figure 11 illustrates an example of generating point cloud information that can be processed in two dimensions from point cloud information in three-dimensional space output by sensor 100. Figure 11 shows an example in which an object (in this case, a person) Tm located at a certain location is detected in three dimensions (spatially) by sensor 100 installed at the origin O, and then converted to two dimensions (planarly) by electronic device 1.

[0122] As shown in Figure 11, with the origin O as the reference point, the area to the right of the origin O is defined as the positive X-axis direction, and the area below the origin O is defined as the positive Y-axis direction. That is, in Figure 11, px represents the horizontal coordinate and py represents the vertical coordinate. In the situation shown in Figure 11, the output of sensor 100 is converted at a given moment from the aforementioned 4-channel data to 3-channel data of position (X,Y) data elements (signal strength and velocity) in a 2D plane. In this way, in step S13 shown in Figure 9, the electronic device 1 converts information detected in 3D space into information in a 2D plane.

[0123] As described above, when converting information detected in three-dimensional space into information in a two-dimensional plane, the coordinates of the two-dimensional plane may be calculated based on, for example, the following equations (1) and (2).

[0124]

number

[0125]

number

[0126] In equations (1) and (2) above, lx i ,ly i ,lz i This shows the output based on the detection results by sensor 100, i.e., point cloud information in 3D space. In particular, lx irepresents the distance in the x - direction of the i - th information detected by the sensor 100. Also, ly i represents the distance in the y - direction of the i - th information detected by the sensor 100. Also, lz i represents the distance in the z - direction of the i - th information detected by the sensor 100.

[0127] Also, in the above formulas (1) and (2), px i , py i represent the coordinates of the point cloud converted into two - dimensional plane information by the controller 10 of the electronic device 1. In particular, px i represents the x - coordinate of the i - th information detected by the sensor 100. Also, py i represents the y - coordinate of the i - th information detected by the sensor 100.

[0128] Furthermore, in the above formula (1), M represents the number of pixels in the horizontal direction when assuming an image on a two - dimensional plane, and αx represents the horizontal viewing angle when assuming an image on a two - dimensional plane. Also, in the above formula (2), N represents the number of pixels in the vertical direction when assuming an image on a two - dimensional plane, and αy represents the vertical viewing angle when assuming an image on a two - dimensional plane.

[0129] In the above formulas (1) and (2), px i , py i may be rounded to the first decimal place and converted to an integer so as to function as a coordinate value. Also, in the above formulas (1) and (2), px i , py i should be within the size of the image after conversion to a two - dimensional plane. For example, data that does not satisfy 0≦px i ≦M or 0≦py i ≦N may be discarded.

[0130] Thus, the electronic device 1 according to one embodiment may generate two-dimensionally processable point cloud information from the output of the sensor 100. In particular, the electronic device 1 according to one embodiment may convert the point cloud information detected by the sensor 100 into two-dimensionally processable point cloud information based on at least one of a predetermined number of pixels and a predetermined field of view in a two-dimensional image. When the electronic device 1 according to one embodiment generates two-dimensionally processable point cloud information from the output of the sensor 100, it may use a conversion formula other than the above-described formulas (1) and (2).

[0131] According to one embodiment of the electronic device 1, the point cloud information in step S12 is reduced in dimensionality in step S13, thereby significantly reducing the computational load on, for example, the controller 10. Therefore, according to one embodiment of the electronic device 1, the processing load on, for example, the controller 10 can be reduced. Each point constituting the two-dimensional point cloud data generated in step S13 may include three channels of information: reflection intensity, distance, and velocity at that point.

[0132] After step S13, the controller 10 determines the region with the most points in the point cloud data based on the detection of the flat plate BD, among the regions divided at predetermined intervals in the direction of calibration (in this case, the horizontal direction) (step S14). Here, the region with the most points may be a "column" as shown as region Cm in Figure 8. That is, in step S14, the controller 10 determines the center position of the point cloud corresponding to the flat plate BD. Next, the controller 10 determines whether the column with the most points, i.e., the center position of the point cloud corresponding to the flat plate BD, coincides with the center position of the image of the flat plate BD (step S15). In step S15, instead of determining whether the center position of the point cloud corresponding to the flat plate BD coincides with the center position of the image of the flat plate BD as described above, the controller 10 may determine whether the positions of both are within a predetermined range.

[0133] In step S15, if the center position of the point cloud corresponding to the flat plate BD coincides with (or is within a predetermined range of) the center position of the image on the flat plate BD, the controller 10 may determine that calibration is not necessary. In this case, the controller 10 may terminate the operation shown in Figure 9. In this case, the controller 10 may also notify the user that the center position of the point cloud corresponding to the flat plate BD coincides with the center position of the image on the flat plate BD, and / or that calibration is not necessary.

[0134] On the other hand, in step S15, if the center position of the point cloud corresponding to the flat plate BD does not coincide with (or exceeds a predetermined range of) the center position of the image on the flat plate BD, the controller 10 may determine that calibration (or correction) is necessary (step S16). In step S16, the controller 10 may correct the coordinates of the point cloud detected by the sensor 100 based on the amount by which the center position of the point cloud corresponding to the flat plate BD is deviated from the center position of the image on the flat plate BD. In such a case, the controller 10 may notify the user that the center position of the point cloud corresponding to the flat plate BD did not coincide with (or was deviated from) the center position of the image on the flat plate BD, and / or that the coordinates of the point cloud output from the sensor 100 have been corrected. The controller 10 may also correct the coordinates of the center position of the image on the flat plate BD based on the amount by which it is deviated from the center position of the image on the flat plate BD.

[0135] As described above, the controller 10 may determine the center position of the point cloud from the point cloud information obtained by detecting an object (e.g., a flat plate BD) that reflects the transmitted wave, based on the transmitted signal transmitted as a transmitted wave and the received signal received as a reflected wave of the transmitted wave. The controller 10 may also determine the center position of the object (flat plate BD) in the image from the information of the image in which the object (flat plate BD) is captured. Furthermore, the controller 10 may determine whether the distance between the center position of the point cloud and the center position of the object (flat plate BD) in the image is greater than or equal to a predetermined value.

[0136] In one embodiment, the controller 10 may determine the region with the largest number of points among the regions divided horizontally or vertically in the point cloud information as the center position of the point cloud. The controller 10 may also convert the point cloud information detected in three-dimensional space into two-dimensional information. Furthermore, the controller 10 may correct the center position (coordinates) of the point cloud and / or the center position (coordinates) of the object (plane BD) in the image based on whether the distance between the center position of the point cloud and the center position of the object (plane BD) in the image is greater than or equal to a predetermined value.

[0137] In one embodiment, the controller 10 may determine the center position of the point cloud from the point cloud information obtained by detecting a plate-shaped member having a planar portion, such as a flat plate BD, as an object that reflects the transmitted wave. In this case, the controller 10 may determine the center position of the plate-shaped member in the image from the information of the image in which the plate-shaped member is captured. The controller 10 may also determine whether the distance between the center position of the point cloud and the center position of the plate-shaped member in the image is greater than or equal to a predetermined value.

[0138] Thus, according to the electronic device 1 of one embodiment, the misalignment between the optical axis of the sensor and the optical axis of the camera can be easily calibrated. Therefore, according to the electronic device 1 of one embodiment, the sensor that detects an object and the camera that captures an image of the object can be easily calibrated. According to the electronic device 1 of one embodiment, for example, by applying AI technology to information that integrates point cloud information of an object detected by a millimeter-wave radar sensor and image information of the object captured by the camera, the object can be detected with good accuracy. In this case, the controller 10 may learn the position of the object using machine learning based on the point cloud information and the image information of the object.

[0139] (Other embodiments) In the embodiment described above, as shown in Figure 8, the center position of the point cloud corresponding to the flat plate BD was determined based on the region with the most points (region Cm) among the regions divided horizontally at minute intervals. However, in one embodiment, the center position of the point cloud corresponding to the flat plate BD may be determined based on other information.

[0140] For example, the information actually output from sensor 100, as shown in Figure 12, includes not only the original point cloud (indicated by ●) based on the detection of the flat plate BD, but also points detected as noise (indicated by ○). If there are many such points detected as noise, there is a concern that the region with the most points in the point cloud (region Cm) among the regions divided horizontally at minute intervals as described above may not be correctly determined.

[0141] Therefore, when integrating the point cloud information output from the sensor 100 as shown in Figure 12 for multiple frames of the transmitted wave, the frequency of point cloud appearance in those multiple frames may be considered. Figure 13 is a diagram showing an example of a heat map representing the frequency of point cloud appearance when the point cloud information output from the sensor 100 is integrated for multiple frames of the transmitted wave. In one embodiment, the controller 10 may apply a heat map as shown in Figure 13 when integrating the point cloud information output from the sensor 100 as shown in Figure 12 for multiple frames of the transmitted wave.

[0142] Figure 14 shows an example of the result when a heat map representing the frequency of point cloud occurrences is applied when integrating point cloud information output from sensor 100 across multiple frames of the transmitted wave. As shown in Figure 14, the effect of noise can be reduced by applying a heatmap that represents the frequency of occurrence in the point cloud.

[0143] As shown in Figure 14, the controller 10 may calculate the integral value of the reflection intensity (power) of each point constituting the point cloud PT based on the detection of the flat plate BD. In this case, the controller 10 may determine the region in the point cloud PT based on the detection of the flat plate BD by the sensor 100 that has the maximum integral value of the reflection intensity (power) among the regions divided at predetermined intervals in the direction of calibration (in this case, the horizontal direction). As shown in Figure 14, in the point cloud PT based on the detection of the flat plate BD by the sensor 100 that has the maximum integral value of the reflection intensity (power) among the regions divided horizontally at minute intervals is the region Em shown in Figure 14. The controller 10 may determine this region Em as the horizontal center position of the point cloud PT based on the detection of the flat plate BD by the sensor 100 (the center position of the point cloud corresponding to the flat plate BD). In this case as well, the interval of the region divided in the direction of calibration may be adjusted appropriately so that the center position of the point cloud corresponding to the flat plate BD can be properly determined.

[0144] Figure 15 is a flowchart illustrating the operation of the electronic device 1 according to another embodiment described above. In Figure 15, the differences from Figure 9 will be explained in detail, and explanations that are the same as or similar to those in Figure 9 will be simplified or omitted as appropriate.

[0145] Once the operation shown in Figure 15 begins, steps S11 through S13 may be performed in the same manner as the operations shown in Figure 9.

[0146] After step S13, the controller 10 integrates the point cloud data converted to two dimensions in step S13 on a frame-by-frame basis (step S21). After step S21, the controller 10 determines whether the number of frames integrated in step S21 has reached a predetermined number (N frames) (step S22). Here, the number of frames to be integrated may be set as appropriate.

[0147] If the number of integrated frames has not reached a predetermined number in step S22, the controller 10 repeats the operations from step S11 to step S13 and step S21. If the number of integrated frames has reached a predetermined number in step S22, the controller 10 applies a heatmap representing the frequency of point cloud occurrences, as illustrated in Figure 14, to the point cloud data of the N integrated frames (step S23).

[0148] After step S23, the controller 10 determines the region in the point cloud data based on the detection of the flat plate BD where the integrated value of the reflection intensity (power) is maximum, among regions divided at predetermined intervals in the direction of calibration (in this case, the horizontal direction) (step S24). Here, the region where the integrated value of the reflection intensity (power) is maximum may be a "column" as shown as region Em in Figure 14. That is, in step S24, the controller 10 determines the center position of the point cloud corresponding to the flat plate BD. Next, the controller 10 determines whether the region where the integrated value of the reflection intensity (power) is maximum, i.e., the center position of the point cloud corresponding to the flat plate BD, coincides with the center position of the image of the flat plate BD (step S25). In step S25, instead of determining whether the center position of the point cloud corresponding to the flat plate BD coincides with the center position of the image of the flat plate BD as described above, the controller 10 may determine whether the positions of both are within a predetermined range. Subsequent operations may be the same as those in Figure 9.

[0149] As described above, the controller 10 may determine the center position of the point cloud as the region in the point cloud information that is divided horizontally or vertically, and in which the integrated power value of the point cloud contained in that region is maximum. In this case, the controller 10 may perform the above determination based on the result of overlapping a predetermined number of frames of the transmitted wave. Alternatively, the controller 10 may perform the above determination based on the frequency at which the point cloud is detected based on the result of overlapping a predetermined number of frames of the transmitted wave.

[0150] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art will find it easy to make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided. While embodiments relating to this disclosure have been described primarily in terms of apparatus, embodiments relating to this disclosure can also be realized as methods including steps performed by each component of the apparatus. Embodiments relating to this disclosure can also be realized as methods, programs, or storage media recording programs executed by a processor in the apparatus. These should also be understood to be included within the scope of this disclosure.

[0151] The embodiments described above are not limited to being implemented as electronic device 1. For example, the embodiments described above may be implemented as electronic device 1 included in electronic device 1. Furthermore, the embodiments described above may be implemented as a monitoring method by a device such as electronic device 1. Moreover, the embodiments described above may be implemented as a program executed by a device such as electronic device 1 or an information processing device (e.g., a computer), or as a storage medium or recording medium on which such a program is recorded. [Explanation of symbols]

[0152] 1 Electronic equipment 3. Detection equipment 5. Stand section 7 Grounding part 10 Controllers 20 Memory section 30 Communications Department 40 Display section 50 Hochi Department 100 sensors 101 Radio wave input section 110 Radar Control Unit 120 Transmitter 121 Signal Generation Unit 122 Synthesizers 123 Phase Control Unit 124 Amplifier 125 Transmitting Antenna 130 Receiver 131 Receiving antenna 132 LNA 133 Mixer 134 IF section 135 AD Conversion Unit 300 Imaging Unit 301 Optical input section

Claims

1. Based on the transmitted signal that is transmitted as a transmitted wave and the received signal that is received as a reflected wave of the transmitted wave, the point cloud information obtained by detecting an object that reflects the transmitted wave is used to determine the center position of the point cloud, which is the region with the largest number of points among the regions divided horizontally or vertically in the point cloud information. From the information of the image captured of the object, the central position of the object in the image is determined, Determine whether the distance between the center position of the point cloud and the center position of the object in the image is greater than or equal to a predetermined value. An electronic device comprising a control unit that corrects the discrepancy between the position of the object in the point cloud and the position of the object in the image based on the determination.

2. The electronic device according to claim 1, wherein the control unit determines the region in the point cloud information that is divided horizontally or vertically, where the integral value of the power indicated by the point cloud included in that region is the maximum, as the center position of the point cloud.

3. The electronic device according to claim 2, wherein the control unit determines, based on the result of superimposing a predetermined number of frames of the transmitted wave, the region in the point cloud information that is divided horizontally or vertically and in such a region has the maximum integrated power value of the point cloud contained within that region, and identifies that region as the center position of the point cloud.

4. The electronic device according to claim 3, wherein the control unit determines, based on the frequency with which the point cloud is detected, the region in the point cloud information that is divided horizontally or vertically, where the integral value of the power indicated by the point cloud included in that region is the maximum, as the result of superimposing a predetermined number of frames of the transmitted wave, and determines that region is the center position of the point cloud.

5. The electronic device according to claim 1, wherein the control unit converts the information of the point cloud detected in three-dimensional space into two-dimensional information.

6. The electronic device according to claim 1, wherein the control unit corrects the coordinates of the center position of the point cloud and / or the center position of the object in the image based on a determination of whether the distance between the center position of the point cloud and the center position of the object in the image is greater than or equal to a predetermined value.

7. The electronic device according to claim 1, wherein the control unit performs machine learning to determine the position of the object based on the point cloud information and the information of the image in which the object was captured.

8. The control unit, The center position of the point cloud is determined from the point cloud information obtained by detecting a plate-shaped member having a flat portion as an object that reflects the transmitted wave. From the information of the image in which the plate-shaped member is captured, the central position of the plate-shaped member in the image is determined. The electronic device according to claim 1, which determines whether the distance between the center position of the point cloud and the center position of the plate-shaped member in the image is greater than or equal to a predetermined value.

9. Based on the transmitted signal that is transmitted as a transmitted wave and the received signal that is received as a reflected wave of the transmitted wave, the point cloud information obtained by detecting an object that reflects the transmitted wave is used to determine the center position of the point cloud, which is the region with the largest number of points among the regions divided horizontally or vertically in the point cloud information. From the information of the image captured of the object, the central position of the object in the image is determined, Determine whether the distance between the center position of the point cloud and the center position of the object in the image is greater than or equal to a predetermined value. A method for controlling electronic equipment, comprising the step of correcting the discrepancy between the position of the object in the point cloud and the position of the object in the image based on the determination.

10. On the computer, Based on the transmitted signal that is transmitted as a transmitted wave and the received signal that is received as a reflected wave of the transmitted wave, the point cloud information obtained by detecting an object that reflects the transmitted wave is used to determine the center position of the point cloud, which is the region with the largest number of points among the regions divided horizontally or vertically in the point cloud information. From the information of the image captured of the object, the central position of the object in the image is determined, Determine whether the distance between the center position of the point cloud and the center position of the object in the image is greater than or equal to a predetermined value. A program that, based on the determination, performs a step of correcting the discrepancy between the position of the object in the point cloud and the position of the object in the image.

Images