Electronic device, method for controlling electronic device, and program

Abstract

This electronic device for estimating the position of a sensor on which an antenna is mounted calculates, on the basis of the position of an object and the reception intensity based on a reflected wave that are obtained from a transmitted wave transmitted from the antenna and a reflected wave obtained when the transmitted wave is reflected by the object, the average and variance of the products of the position of the object and the reception intensity in a voxel, which is a prescribed region, calculates an evaluation function on the basis of the position of the object obtained from the transmitted wave transmitted from the antenna and the reflected wave obtained when the transmitted wave is reflected by the object, and a function indicating the density of a point cloud in the voxel, and estimates the position of the sensor on the basis of the evaluation function.

Description

Electronic device, electronic device control method, and program CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2024-153478, filed on September 5, 2024, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to an electronic device, a control method for an electronic device, and a program.

[0003] For example, in fields such as the automotive industry, technology for measuring the distance between a vehicle and a predetermined object is gaining importance. In particular, in recent years, various RADAR (Radio Detecting and Ranging) technologies have been researched, which measure the distance between an object and the vehicle by transmitting radio waves such as millimeter waves and receiving the waves reflected by the object, such as an obstacle. The importance of such technology for measuring distance is expected to increase in the future with the development of technologies for assisting drivers and technologies related to autonomous driving, which automates driving partially or completely. Hereinafter, radar sensors using millimeter waves will also be referred to as "millimeter wave sensors."

[0004] Furthermore, there is known a technology, such as SLAM (Simultaneous Localization and Mapping), that simultaneously estimates the self-position of a moving object and creates an environmental map. Research is also underway into a technology that utilizes the output of, for example, a millimeter-wave sensor, LiDAR (Light Detection and Ranging), and / or a camera that captures images, in order to accurately estimate the self-position of a moving object. For example, Patent Document 1 proposes adjusting the detection results of a millimeter-wave sensor in accordance with the three-dimensional shape of each region of a three-dimensional map generated from images captured by an imaging device.

[0005] Japanese Patent Application Laid-Open No. 2021-86298

[0006] An electronic device according to one embodiment is an electronic device that estimates the position of a sensor equipped with an antenna, and calculates an average A(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within a voxel that is a predetermined region, based on a position X(N-1) of the object and a reception intensity P(N-1) based on the reflected wave, the average A(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within the voxel, the average A(N-1), the variance V(N-1), the position X(N) of the object obtained from the transmission wave transmitted from the antenna and the reflection wave of the transmission wave reflected by the object, in the N-1th frame that is a predetermined time interval, where N is an integer of 1 or greater, and a function V(N-1) indicating the density of a point cloud within the voxel. and calculating an evaluation function ρ(N) based on the evaluation function ρ(N), and estimating the position of the sensor based on the evaluation function ρ(N).

[0007] A control method according to one embodiment is a control method for an electronic device that estimates the position of a sensor equipped with an antenna, comprising: calculating, based on a position X(N-1) of an object and a reception strength P(N-1) based on a reflected wave, which are obtained from a transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object, in an N-1th frame that is a predetermined time interval, where N is an integer of 1 or more; an average A(N-1) of the product of the object's position X(N-1) and the reception strength P(N-1) within a voxel that is a predetermined region; and a variance V(N-1) of the product of the object's position X(N-1) and the reception strength P(N-1) within the voxel; and calculating the average A(N-1), the variance V(N-1), and the position X(N) of the object obtained from the transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object, in an Nth frame that is the next frame after the N-1th frame. An evaluation function ρ(N) is calculated based on a function V(N−1) indicating the density of the point cloud in the voxel, and the position of the sensor is estimated based on the evaluation function ρ(N).

[0008] a program according to one embodiment causes an electronic device that estimates the position of a sensor equipped with an antenna to calculate, based on a position X(N-1) of an object and a reception strength P(N-1) based on a reflected wave, the position X(N-1) of the object being obtained from a transmission wave transmitted from the antenna and a reflected wave of the transmission wave reflected by the object in an N-1th frame, which is a predetermined time interval, where N is an integer of 1 or more, and the average A(N-1) of the product of the position X(N-1) of the object within a voxel that is a predetermined region and the reception strength P(N-1) of the object within the voxel, and a variance V(N-1) of the product of the position X(N-1) of the object within the voxel and the reception strength P(N-1); the average A(N-1), the variance V(N-1), the position X(N) of the object being obtained from a transmission wave transmitted from the antenna and a reflected wave of the transmission wave reflected by the object in an Nth frame that is the next frame after the N-1th frame, and a function V(N-1) that indicates the density of a point cloud within the voxel; and calculating an evaluation function ρ(N) based on the evaluation function ρ(N), and estimating the position of the sensor based on the evaluation function ρ(N).

[0009] 1 is a diagram showing an example of an ambient environment detected by a LiDAR sensor or a millimeter wave sensor. FIG. 2 is a diagram showing an example of the results of detecting the environment of FIG. 1 by a LiDAR sensor. FIG. 3 is a diagram showing an example of detecting the environment of FIG. 1 by a millimeter wave sensor. FIG. 4 is a diagram explaining an example of approximating a probability density function based on the results detected by the LiDAR sensor. FIG. 5 is a diagram explaining an example of modeling the results detected by the millimeter wave sensor as a probability density. FIG. 6 is a block diagram showing the functional configuration of an electronic device according to an embodiment. FIG. 7 is a diagram explaining signal processing by an electronic device according to an embodiment. FIG. 8 is a flowchart explaining the operation of an electronic device according to an embodiment. FIG. 9 is a flowchart explaining the operation of an electronic device according to an embodiment. FIG. 10 is a block diagram showing the functional configuration of an electronic device according to another embodiment. FIG. 11 is a flowchart explaining the operation of an electronic device according to another embodiment. FIG. 12 is a block diagram showing the functional configuration of an electronic device according to another embodiment.

[0010] If the position of a device or the like can be detected with high accuracy and efficiency by transmitting and receiving radio waves, such as millimeter waves, it is expected to be useful in a wide variety of fields. The present disclosure relates to providing an electronic device, a control method for an electronic device, and a program that can detect a position with high accuracy and efficiency by transmitting and receiving radio waves. According to one embodiment, it is possible to provide an electronic device, a control method for an electronic device, and a program that can detect a position with high accuracy and efficiency by transmitting and receiving radio waves. Hereinafter, one embodiment will be described with reference to the drawings.

[0011] In the present disclosure, an "electronic device" may refer to a device driven by electricity. Furthermore, in the present disclosure, a "system" may include a device driven by electricity. Furthermore, in the present disclosure, a "user" may refer to a person (typically a human) who uses a system and / or an electronic device according to an embodiment. The user may include a person who enjoys the benefit of detecting the location of the electronic device or other device by using a system and / or an electronic device according to an embodiment.

[0012] An electronic device according to an embodiment is mounted on a vehicle (mobile body) such as an automobile and is capable of detecting a predetermined object (target) present around the mobile body as a target. To this end, the electronic device according to an embodiment can transmit a transmission wave (radio wave) to the surroundings of the mobile body from a transmitting antenna (transmitting antenna array) installed on the mobile body. The electronic device according to an embodiment can also receive a reflected wave of the transmission wave from a receiving antenna (receiving antenna array) installed on the mobile body. At least one of the transmitting antenna and the receiving antenna may be included in a radar sensor, such as a millimeter-wave sensor, installed on the mobile body.

[0013] Hereinafter, as a typical example, a configuration in which an electronic device according to an embodiment is mounted on a mobile object such as an automobile will be described. However, the electronic device according to an embodiment is not limited to being mounted on a mobile object. The electronic device according to an embodiment may be mounted on various mobile objects, such as self-driving cars, buses, taxis, trucks, motorcycles, bicycles, ships, airplanes, helicopters, agricultural equipment such as tractors, snowplows, sweepers, police cars, ambulances, drones, and rockets. Furthermore, the electronic device according to an embodiment is not necessarily mounted on a mobile object that moves under its own power. For example, the mobile object on which the electronic device according to an embodiment is mounted may be a trailer towed by a tractor. Furthermore, the mobile object (e.g., an automobile) according to the present disclosure is not limited by its overall length, overall width, overall height, engine displacement, passenger capacity, or payload. For example, the automobile according to the present disclosure may include automobiles with an engine displacement of more than 660 cc and automobiles with an engine displacement of 660 cc or less, such as so-called light vehicles. Furthermore, the automobile according to the present disclosure may also include automobiles that use electricity for part or all of their energy and that use a motor. Furthermore, the electronic device according to an embodiment does not necessarily have to be mounted on a mobile object. For example, an electronic device according to an embodiment may be attached to or built into another device that is fixed to the ground. Also, the electronic device according to an embodiment may be fixed to the ground, for example.

[0014] An electronic device according to an embodiment can measure the distance between a sensor and a predetermined object in a situation where at least one of the sensor and the object may be moving. Also, an electronic device according to an embodiment can measure the distance between the sensor and the object even when both the sensor and the object are stationary.

[0015] An electronic device according to an embodiment described below can efficiently detect its own position with high accuracy by transmitting and receiving radio waves using a millimeter wave sensor, etc. The electronic device according to an embodiment can efficiently estimate the self-position of a moving object with high accuracy using a technology such as SLAM.

[0016] (Background to the Idea of ​​the Electronic Device According to an Embodiment) In the following, when describing the electronic device according to an embodiment, detection of the surrounding environment by a millimeter wave sensor and detection of the surrounding environment by a sensor other than a millimeter wave sensor will be described. Here, a LiDAR sensor (LiDAR sensor) will be used as an example of the sensor other than a millimeter wave sensor.

[0017] Fig. 1 shows the interior of a typical room, which is prepared as the surrounding environment to be detected by the sensors. The results of detecting this surrounding environment using a LiDAR sensor and a millimeter-wave sensor are described below.

[0018] Figure 2 is a diagram showing a plot of a point cloud obtained as a result of detecting the surrounding environment shown in Figure 1 using a general LiDAR sensor. The point cloud plot shown in Figure 2 shows an overhead view of the surrounding environment shown in Figure 1. Figure 2 shows that the walls of the surrounding environment shown in Figure 1 are detected with good accuracy using a general LiDAR sensor.

[0019] Fig. 3 is a diagram showing a plot of the probability density distribution obtained as a result of detecting the surrounding environment shown in Fig. 1 using a general millimeter wave sensor. Similar to Fig. 2, the plot shown in Fig. 3 shows an overview of the surrounding environment shown in Fig. 1. Fig. 3 shows that when a general millimeter wave sensor is used, noise is detected from other places in the surrounding environment besides the walls shown in Fig. 1.

[0020] Furthermore, comparing the results shown in Figure 2 with those shown in Figure 3 reveals differences between the two in the following respects. First, the detection results obtained by a millimeter-wave sensor have more noise than those obtained by a LiDAR sensor. On the other hand, it is known that the detection results obtained by a LiDAR sensor are susceptible to light reflection and other issues. Furthermore, the detection results obtained by a millimeter-wave sensor are composed of a smaller number of point clouds than those obtained by a LiDAR sensor, i.e., have lower resolution. Generally, the detection results obtained by a millimeter-wave sensor have lower resolution than those obtained by a sensor that captures images, such as a camera. Conversely, detection by a LiDAR sensor results in a larger number of point clouds per frame. Therefore, in order to use a method such as NDT (Normal Distribution Transform) with a LiDAR sensor, some kind of processing to reduce the number of point clouds may be necessary. NDT is a method of normalizing the point cloud of an object by dividing the environmental map (search space) into voxels (in a grid), calculating a normal distribution, and matching it with the point cloud data obtained from the LiDAR sensor.

[0021] Here, we will further discuss NDT, which is known as a technique used for matching point cloud data. NDT efficiently represents the three-dimensional structure of the environment and is useful for estimating location and / or creating maps. In NDT, point cloud data obtained by a sensor is modeled as a normal distribution.

[0022] The basic steps of NDT can be performed as follows: First, the environment is divided into voxels. Next, the distribution of point cloud data contained in each voxel is calculated. Specifically, the mean and variance are calculated from the positions of the point clouds, and these are modeled as normal distributions. This normal distribution obtained for each voxel is then used to statistically represent the point cloud data for the entire environment. Next, the current point cloud data (data obtained after the modeled data) is compared with this modeled data, and the position and orientation are optimized. Specifically, the position and orientation are adjusted so that the point cloud data best fits the modeled normal distribution.

[0023] The detection results of a millimeter-wave sensor are not output as a point cloud, but rather as a probability density function representing the surrounding terrain. A typical millimeter-wave sensor calculates the median value of the output obtained in the form of a probability density function and outputs the center of that value as the measurement point. The probability distribution information that is deleted when converting the output from a millimeter-wave sensor into a point cloud format includes information about the surrounding terrain (surrounding environment). In other words, expressing the output from a millimeter-wave sensor in the form of a point cloud results in the probability distribution extending in the direction of obstacles in the surrounding environment. Through demonstration experiments and other testing, the applicant has confirmed that by utilizing the probability distribution information that is deleted when converting the output from a millimeter-wave sensor into a point cloud format, it is possible to effectively compensate for the lower resolution and accuracy compared to LiDAR sensors.

[0024] FIG. 4 is a conceptual diagram illustrating an example of applying NDT to the output of a LiDAR sensor.

[0025] To estimate a self-location based on the output of a LiDAR sensor, a method is known in which NDT is applied to a point cloud detected by the LiDAR sensor to approximate a probability density function. For example, assume that a point cloud is detected by the LiDAR sensor, as shown on the left side of FIG. 4 . By applying NDT to the detected point cloud, the point cloud can be matched. Specifically, as shown on the right side of FIG. 4 , the point cloud is first divided into voxels (in a grid pattern). Then, as shown on the right side of FIG. 4 , matching is performed by assigning a normal distribution to the point cloud within each voxel.

[0026] As shown in Figure 4, when NDT is applied to the output of a LiDAR sensor, the number of points in one frame is large, as described above, and it may be necessary to reduce the number of points by some method. Furthermore, in such a method, NDT is applied to each point in each frame of the output of the LiDAR sensor. For this reason, the method of applying NDT to the output of a LiDAR sensor tends to require a large amount of calculation.

[0027] FIG. 5 is a diagram showing data obtained by performing signal processing such as FFT (Fast Fourier Transform) on the output of a millimeter wave sensor, modeled as a probability density.

[0028] As shown in Figure 5, data obtained by performing signal processing such as FFT on the output of a millimeter-wave sensor can be considered as data equivalent to NDT data without matching each point cloud. Therefore, data obtained by performing signal processing such as FFT on the output of a millimeter-wave sensor can be efficiently matched between frames using a small number of point clouds. A method for estimating the self-position of a moving object based on the output of a millimeter-wave sensor allows for a probability density function to be directly obtained from the millimeter-wave sensor, even if the number of obtained point clouds is relatively small. Therefore, this method reduces the computational cost of the conversion process. Furthermore, this method reduces the data size, making it possible to perform calculations even when the required memory is relatively small.

[0029] Here, a frame in the present disclosure will be described. In the present disclosure, a frame refers to distance, speed, and / or angle data acquired by a millimeter wave sensor in one measurement cycle. The higher the frame rate (the number of frames acquired per second), the more real-time and / or accurate the data. One frame may consist of distance data, speed data, and / or angle data. Distance data is obtained by measuring the distance from the sensor to an object. This data can be calculated based on the time it takes for radio waves to reflect off the object and return. Speed ​​data is obtained by measuring the relative speed of an object using the Doppler effect. This data can be used to determine whether an object is approaching or receding from the sensor. Furthermore, angle data is obtained by measuring the direction of the object. This data is obtained by calculating the angle based on the phase difference of the arriving radio waves. Here, the frame collection process may include transmitting radio waves, receiving reflected waves, data analysis, and data aggregation. By transmitting and receiving radio waves and receiving reflected waves for the number of antennas, and then analyzing and aggregating the data, distance, speed, and / or angle data can be obtained.

[0030] A method for estimating self-location by applying NDT to the output of a millimeter-wave sensor using an electronic device according to an embodiment will be described below. This method can improve the accuracy of self-location estimation even when a small number of points are detected by combining information on error distribution, which has not been used in the past. An electronic device according to an embodiment efficiently utilizes information on a smaller number of points than when using a LiDAR sensor, i.e., utilizes RAW data from the millimeter-wave sensor. Here, RAW data may refer to information read by an image sensor such as a millimeter-wave sensor, recorded as is. According to an electronic device according to an embodiment, signal processing (e.g., CFAR processing and / or direction-of-arrival estimation processing) that is typically performed when acquiring a point cloud may not be performed on the output of the millimeter-wave sensor. For example, an electronic device according to an embodiment may use the RAW data output from the millimeter-wave sensor as is. Furthermore, an electronic device according to an embodiment may perform only CFAR processing on the millimeter-wave sensor, without performing direction-of-arrival estimation processing. According to an electronic device according to an embodiment, SLAM using a millimeter-wave sensor may be realized.

[0031] (Configuration of Electronic Device According to One Embodiment) Next, a functional configuration of the electronic device according to one embodiment will be described.

[0032] 6 is a block diagram showing the functional configuration of an electronic device according to an embodiment. As shown in FIG. 6, the electronic device according to an embodiment may include a physical layer 1, a signal processing layer 10, and an application layer 20.

[0033] 1 may be hardware constituting a millimeter wave sensor, for example, and may include a transmitting antenna 2, a receiving antenna 3, an RF unit 4, a relay unit 5, a DSP (Digital Signal Processor) 6, and an FPGA (Field Programmable Gate Array) 7.

[0034] The transmitting antenna 2 may be, for example, an antenna array (array antenna) as an antenna for transmitting radio waves. The receiving antenna 3 may be, for example, an antenna array (array antenna) as an antenna for receiving radio waves reflected by an object or the like from the transmitting antenna 2. The RF unit 4 may be a circuit providing various functions necessary for the transmission of radio waves by the transmitting antenna 2 and / or the reception of radio waves by the receiving antenna 3. The relay unit 5 may be a circuit providing a signal relay function, such as converting digital signals to analog signals and vice versa. The DSP 6 may be a processor to which setting values ​​set by the antenna setting unit 12 of the signal processing layer 10 are supplied. The DSP 6 has a function of performing various settings when transmitting radio waves from the transmitting antenna 2 of the physical layer 1 based on the setting values ​​set by the antenna setting unit 12 of the signal processing layer 10. The FPGA 7 may be a device having a function of supplying a signal based on the radio waves received from the receiving antenna 3 as RAW data to the FFT processing unit 14 of the signal processing layer 10. Each functional unit constituting the physical layer 1 may be the same as or based on a similar concept to each functional unit constituting a known millimeter wave sensor, and therefore a detailed description of each functional unit constituting the physical layer 1 will be omitted.

[0035] The signal processing layer 10 shown in Fig. 2 may be configured, for example, by any circuit having the function of performing various processes on signals transmitted and received by the millimeter wave sensor, a central processing unit (CPU), a DSP, etc. The signal processing layer 10 may also be configured by specific means in which software and hardware resources work together. As shown in Fig. 2, the signal processing layer 10 may include, for example, an antenna setting unit 12, an FFT (Fast Fourier Transform) processing unit 14, a CFAR (Constant False Alarm Rate) processing unit 16, and a direction-of-arrival estimation unit 18.

[0036] The antenna setting unit 12 sets setting values ​​for various settings when transmitting radio waves from the transmitting antenna 2 of the physical layer 1. The setting values ​​set by the antenna setting unit 12 may be supplied to the DSP 6 of the physical layer 1. The FFT processing unit 14 has a function of performing FFT processing on RAW data supplied from the physical layer 1. The data FFT-processed by the FFT processing unit 14 may be supplied to the CFAR processing unit 16. The CFAR processing unit 16 has a function of performing CFAR processing on the data FFT-processed by the FFT processing unit 14. The data CFAR-processed by the CFAR processing unit 16 may be supplied to the direction-of-arrival estimating unit 18. The direction-of-arrival estimating unit 18 has a function of estimating the direction of arrival of radio waves (received waves) based on the data CFAR-processed by the CFAR processing unit 16.

[0037] As a result of signal processing by each functional unit included in the signal processing layer 10, the signal processing layer 10 outputs information on the object's position coordinates, signal strength (received radio wave strength), and / or object speed. The information output from the signal processing layer 10 may be supplied to the application layer 20. Each functional unit constituting the signal processing layer 10 may be based on the same or similar concept as each functional unit constituting a known millimeter wave sensor. Therefore, a detailed description of each functional unit constituting the signal processing layer 10 will be omitted.

[0038] The application layer 20 shown in FIG. 2 may be application software executed in, for example, a signal processing unit or an arbitrary control unit constituting the millimeter wave sensor. Alternatively, the application layer 20 may be configured by specific means in which software and hardware resources cooperate with each other. The application layer 20 may include, for example, a front end 22 and a back end 24. The front end 22 may include an NDT (Normal Distribution Transform) processing unit 222 and a Doppler processing unit 224. The back end 24 may include an odometry processing unit 242, a loop closing processing unit 244, and a pose graph optimization processing unit 246.

[0039] In the front end 22 of the application layer 20, the NDT processing unit 222 has a function of performing NDT processing based on information supplied from the signal processing layer 10. In addition, the Doppler processing unit 224 has a function of performing Doppler processing based on information supplied from the signal processing layer 10. The information processed in the front end 22 of the application layer 20 may be supplied to the back end 24. In the present disclosure, various types of SLAM algorithms may be used.

[0040] In the back end 24 of the application layer 20, the odometry processing unit 242 performs odometry processing based on information supplied from the front end. The information processed by the odometry processing unit 242 may be supplied to the pose graph optimization processing unit 246. Furthermore, the loop closure processing unit 244 performs loop closure processing based on information supplied from the front end. The information processed by the loop closure processing unit 244 may be supplied to the pose graph optimization processing unit 246. The pose graph optimization processing unit 246 performs pose graph optimization processing based on the information processed by the odometry processing unit 242 and the information processed by the loop closure processing unit 244. As a result of processing by each functional unit included in the application layer 20, estimated position information of the object is output from the application layer 20.

[0041] The electronic device according to the embodiment may not include at least some of the functional units shown in FIG. 6, or may include functional units other than the functional units shown in FIG.

[0042] Furthermore, an electronic device according to an embodiment may include the physical layer 1, the signal processing layer 10, and the application layer 20 as an integrated unit, or may include at least some of them as separate units. For example, an electronic device according to an embodiment may include all of the physical layer 1, the signal processing layer 10, and the application layer 20 as an integrated unit. For example, an electronic device according to an embodiment may include the physical layer 1 and the signal processing layer 10 in the housing of the millimeter-wave sensor, and the application layer 20 in another external device, such as a server. For example, an electronic device according to an embodiment may include the physical layer 1 in the housing of the millimeter-wave sensor, and the signal processing layer 10 and the application layer 20 in another external device, such as a server.

[0043] (Operation of Electronic Device According to One Embodiment) Fig. 7 is a diagram conceptually explaining the operation of the electronic device according to one embodiment. The operation of the electronic device according to one embodiment will be described below.

[0044] Fig. 7 schematically illustrates only the physical layer 1 of the electronic device illustrated in Fig. 6 . That is, the physical layer 1 illustrated in Fig. 7 may represent hardware of a millimeter-wave sensor including a transmitting antenna and / or a receiving antenna. The lightly shaded area A2 in Fig. 7 represents the detection range of the millimeter-wave sensor illustrated as the physical layer 1. The darkly shaded area A1 in Fig. 7 represents the detection range of one element of the antenna of the millimeter-wave sensor illustrated as the physical layer 1. Fig. 7 illustrates an overhead view of the detection range of the millimeter-wave sensor illustrated as the physical layer 1.

[0045] As described above, an electronic device according to an embodiment estimates its own position by applying NDT to the output of a millimeter-wave sensor. To improve the accuracy of the shape and matching of the NDT, which can significantly affect the accuracy of SLAM, the electronic device according to an embodiment operates as follows: [1] Fitting is performed using not only the coordinates of the point cloud obtained as the output of the millimeter-wave sensor but also its reception intensity. [2] When matching the point cloud obtained as the output of the millimeter-wave sensor with the NDT, the density of the point cloud according to distance is taken into consideration (i.e., voxels closer to the own position where the point cloud is dense are emphasized).

[0046] In order to realize the above-described operation, an electronic device according to one embodiment may employ an algorithm including the following steps 1 to 3.

[0047] Step 1: Divide the point cloud obtained as the output of the millimeter wave sensor into voxels of a certain size.

[0048] Voxels of a certain size may be, for example, a lattice pattern as shown in Fig. 7. In Fig. 7, the reception intensity of the point cloud that constitutes the voxel is shown according to the gray depth on the grayscale. That is, darker parts of the voxel indicate higher reception intensity of the point cloud, and lighter parts of the voxel indicate lower reception intensity of the point cloud.

[0049] Step 2: Apply the received intensity information when calculating the mean and variance from the point cloud within each voxel

[0050] The average q of the voxels shown in FIG. 7 can be expressed as the following equation (1). In the above formula (1), n ​​represents the number of points contained in a voxel. ｋ represents the points contained in the voxel. ｋ represents the received intensity of the points contained in the voxel.

[0051] 7 can be expressed as the following equation (2): In equation (2), T represents the transpose of a matrix.

[0052] Step 3: Matching is performed by reflecting the ratio of the point cloud output from the millimeter wave sensor to the voxels according to the distance from the antenna in the physical layer 1.

[0053] The evaluation function used when performing the above-mentioned matching, that is, the matching evaluation function (normal distribution) ρ, can be expressed as in the following equation (3): In equation (3), T represents the transpose of a matrix. In the above formula (3), V ｊ represents the ratio of the detection range of one antenna element to the voxel. The letters in the following formula (4) represent the currently scanned point cloud.

[0054] 8 is a flowchart illustrating the operation of the electronic device according to the embodiment. First, an overview of the operation of the electronic device according to the embodiment will be described below with reference to FIG.

[0055] 8 starts, the application layer 20 of the electronic device according to an embodiment acquires points (measurement points) detected by the physical layer 1 and the signal processing layer 10 that constitute the millimeter wave sensor (step S1). In step S1, the application layer 20 may acquire measurement points that have been subjected to direction-of-arrival estimation, as shown in Fig. 6. Alternatively, in step S1, the application layer 20 may acquire measurement points that have been subjected to CFAR processing, as described below, or may acquire RAW data that has been subjected to FFT processing.

[0056] When the application layer 20 acquires the measurement points, the NDT processing unit 222 calculates the NDT based on the acquired measurement points (step S2). In step S2, the NDT processing unit 222 may calculate the mean q shown in the above formula (1) and the variance C shown in the above formula (2). In addition, in step S2, the NDT processing unit 222 may apply information on the reception strength when calculating the mean q and the variance C. That is, when calculating the mean q and the variance C, the NDT processing unit 222 uses x ｋ and p ｋ In short, the NDT processing unit 222 calculates the mean q and the variance C based on not only the reception intensity of the point cloud but also the position information.

[0057] After the NDT is calculated in step S2, the odometry processing unit 242 calculates the evaluation function ρ as shown in the above formula (3) (step S3). In step S3, the odometry processing unit 242 calculates the evaluation function ρ using the density distribution V in the voxel and the radio wave reception strength p. That is, in step S3, the odometry processing unit 242 calculates the evaluation function ρ by placing emphasis on relatively nearby voxels.

[0058] In the above processing, the NDT processing unit 222 may perform the processing up to equation (1) and equation (2). Furthermore, equation (3) indicates odometry processing (calculation of the trajectory of the self-position). Therefore, the odometry processing unit 242 may perform the processing of equation (3).

[0059] After the evaluation function ρ is calculated in step S3, the odometry processing unit 242 calculates (estimates) the trajectory of the millimeter-wave sensor's self-position using the evaluation function ρ (step S4). In step S4, the odometry processing unit 242 may calculate (estimate) the trajectory of the millimeter-wave sensor's self-position, for example, by calculating R and t that maximize the evaluation function ρ. Alternatively, in step S4, the odometry processing unit 242 may calculate (estimate) the trajectory of the millimeter-wave sensor's self-position by calculating R and t that minimize the evaluation function ρ, depending on the algorithm used. The processing in step S4 may be performed based on conventionally known knowledge. Here, R and t represent estimated values ​​of the posture of a robot, mobile object, or the like on which the millimeter-wave sensor is installed. Specifically, R represents rotation, and t represents translational movement. Using R and t, the current point cloud can be converted to the point cloud of the previous frame.

[0060] Fig. 9 is a flowchart specifically illustrating the operation of the electronic device according to the embodiment shown in Fig. 6. Hereinafter, an outline of the operation of the electronic device according to the embodiment shown in Fig. 6 will be specifically described with reference to Fig. 9.

[0061] 9 starts, the arrival direction estimation unit 18 estimates the arrival direction of the received wave (reflected wave) (step S11). The processing of step S11 may correspond to the processing of step S1 shown in Fig. 8. In the operation shown in Fig. 9, the application layer 20 can obtain a point cloud output after performing the arrival direction estimation processing.

[0062] After the arrival direction of the received wave (reflected wave) is estimated in step S11, the NDT processing unit 222 divides the obtained point cloud into voxels of a certain size (step S12). In step S12, the NDT processing unit 222 may divide the obtained point cloud into voxels of a certain size, as shown in FIG.

[0063] After the point cloud is divided into voxels of a certain size in step S12, the NDT processing unit 222 applies a normal distribution to the distribution in each voxel (step S13). In step S13, the NDT processing unit 222 calculates the mean (the above formula (1)) and variance (the above formula (2)) in each voxel as information on the normal distribution.

[0064] After fitting a normal distribution from the distribution in each voxel in step S13, the NDT processing unit 222 reflects the reception intensity of the point cloud (measurement points) (step S14). In step S14, the NDT processing unit 222 reflects the reception intensity of the measurement points, like the average in the above formula (1) and the variance in the above formula (2). The processing from step S12 to step S14 may correspond to the processing of step S2 shown in FIG. 8.

[0065] After reflecting the reception intensity of the measurement point in step S14, the pose graph optimization processor 246 reflects the influence of the voxel (step S15). In step S15, the pose graph optimization processor 246 calculates V ｊ The influence of the voxel may be reflected by using the following formula: The process of step S15 may correspond to the process of step S3 shown in FIG.

[0066] After step S15, the application layer 20 can estimate the self-position of the millimeter wave sensor through optimization processing.

[0067] According to an embodiment of the present invention, the electronic device can efficiently detect its position with high accuracy by transmitting and receiving radio waves. Therefore, the electronic device according to the embodiment of the present invention is expected to be useful in a wide variety of situations, such as for estimating its own position in SLAM and the like.

[0068] Another embodiment of the electronic device according to the embodiment described above will now be described.

[0069] 10 is a block diagram showing the functional configuration of an electronic device according to another embodiment. Only the differences from the electronic device according to the embodiment shown in FIG. 6 will be described below.

[0070] 10 , in an electronic device according to another embodiment, a signal that has undergone FFT processing by the FFT processing unit 14 of the signal processing layer 10 may be supplied to the application layer 20. That is, in the electronic device according to another embodiment shown in Fig. 10 , a signal that has undergone FFT processing by the FFT processing unit 14 of the signal processing layer 10 may be supplied to the application layer 20 without undergoing CFAR processing and direction of arrival estimation. In this case, in step S1 shown in Fig. 8 , the application layer 20 may acquire RAW data that has undergone FFT processing.

[0071] Fig. 11 is a flowchart specifically illustrating the operation of the electronic device according to the other embodiment shown in Fig. 10. Hereinafter, an outline of the operation of the electronic device according to the other embodiment shown in Fig. 10 will be specifically described with reference to Fig. 11.

[0072] When the operation shown in Fig. 11 starts, the FFT processing unit 14 performs FFT processing on a signal based on the received wave (reflected wave) (step S21). In the operation shown in Fig. 11, the application layer 20 performs FFT processing on the RAW data to obtain a heat map. Here, the heat map obtained in step S21 may be, for example, one like that shown in Fig. 5. In Fig. 5, for example, the vertical axis may indicate distance and the horizontal axis may indicate angle. Although Fig. 5 shows a two-dimensional heat map for simplicity, a three-dimensional heat map may also be obtained.

[0073] After the heat map is obtained by the FFT processing in step S21, the NDT processing unit 222 divides the heat map into cells (step S22). In a modified example of another embodiment, the processing of step S22 may be skipped.

[0074] After the heat map is divided into cells in step S22, the NDT processing unit 222 applies a distribution such as a normal distribution to the heat map output using a method such as a Gaussian Mixture Model (GMM) (step S23). In another modification of the embodiment, the processing of step S23 may be skipped.

[0075] After fitting the normal distribution in step S23, the NDT processing unit 222 divides the obtained normal distribution into voxels of a certain size (step S24). In step S24, the NDT processing unit 222 may divide the obtained point cloud into voxels of a certain size, as shown in FIG.

[0076] Once the normal distribution has been divided into voxels of a certain size in step S24, the NDT processing unit 222 applies a normal distribution to the distribution in each voxel (step S25). In step S25, the NDT processing unit 222 calculates the mean (the above formula (1)) and variance (the above formula (2)) for each voxel as information on the normal distribution. In a modified example of another embodiment, when performing the processing of step S25, ｋ and V ｊ The evaluation function ρ of the above formula (3) may be calculated using the average and variance of only the position x, without using the above formula (3).

[0077] After fitting a normal distribution from the distribution in each voxel in step S25, the NDT processing unit 222 reflects the reception intensity of the normal distribution (step S26). In step S26, the NDT processing unit 222 may reflect the reception intensity of the normal distribution, like the average in the above formula (1) and the variance in the above formula (2).

[0078] After reflecting the reception intensity of the normal distribution in step S26, the pose graph optimization processor 246 reflects the influence of the voxel (step S27). In step S27, the pose graph optimization processor 246 reflects the influence of the voxel by adjusting V ｊ The influence of the voxel may be reflected by using

[0079] In this way, the electronic device according to the other embodiment shown in FIG. 10 can also detect its position efficiently and with good accuracy by transmitting and receiving radio waves through the operation shown in FIG.

[0080] Another embodiment 2 Hereinafter, still another embodiment of the electronic device according to the above-described embodiment will be described.

[0081] 12 is a block diagram showing the functional configuration of an electronic device according to another embodiment. Only the differences from the electronic device according to the embodiment shown in FIG. 6 will be described below.

[0082] 12 , in an electronic device according to another embodiment, a signal subjected to FFT processing by the FFT processing unit 14 of the signal processing layer 10 may be subjected to CFAR processing by the CFAR processing unit 16 and then supplied to the application layer 20. That is, in the electronic device according to another embodiment shown in FIG. 12 , a signal subjected to FFT processing by the FFT processing unit 14 of the signal processing layer 10 may be supplied to the application layer 20 without undergoing direction-of-arrival estimation. In this case, in step S1 shown in FIG. 8 , the application layer 20 may acquire measurement points at which CFAR processing has been performed.

[0083] In this way, in the electronic device according to the other embodiment shown in FIG. 12, the position can be detected efficiently and with good accuracy by transmitting and receiving radio waves through operations similar to those shown in FIG. 11.

[0084] While the present disclosure has been described based on various drawings and examples, it should be noted that those skilled in the art can easily make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present disclosure. For example, the functions included in each functional unit can be rearranged so as not to cause logical inconsistencies. Multiple functional units can be combined into one or divided. The above-described embodiments of the present disclosure are not limited to faithful implementation of each of the described embodiments, but can be implemented by combining each feature or omitting some features as appropriate. In other words, those skilled in the art can make various modifications and alterations to the contents of the present disclosure based on the present disclosure. Therefore, these modifications and alterations are within the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. can be added to other embodiments so as not to cause logical inconsistencies, or can be replaced with each functional unit, each means, each step, etc. of other embodiments. Furthermore, in each embodiment, multiple functional units, each means, each step, etc. can be combined into one or divided into two or more. Furthermore, each of the above-described embodiments of the present disclosure is not limited to being implemented faithfully according to each of the described embodiments, but can also be implemented by combining each feature or omitting some of them as appropriate.

[0085] The above-described embodiments are not limited to implementation as electronic devices. For example, the above-described embodiments may be implemented as a control method for a device such as an electronic device. Furthermore, the above-described embodiments may be implemented as a program executed by a device such as an electronic device, or as a storage medium or recording medium on which a program is recorded.

[0086] The above describes electronic devices and the like according to one embodiment, but the electronic devices, control methods, and / or programs and the like according to one embodiment may be implemented, for example, as follows, as long as there are no physical or logical contradictions. [Supplementary Note 1] An electronic device that estimates the position of a sensor equipped with an antenna, comprising: an electronic device that calculates, based on a position X(N-1) of an object and a reception strength P(N-1) based on a reflected wave, obtained from a transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object, in an N-1th frame that is a predetermined time interval, where N is an integer of 1 or more; an average A(N-1) of the product of the position X(N-1) of the object and the reception strength P(N-1) within a voxel that is a predetermined region; and a variance V(N-1) of the product of the position X(N-1) of the object within the voxel and the reception strength P(N-1), wherein the average A(N-1), the variance V(N-1), the position X(N) of the object obtained from a transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object, in an Nth frame that is the next frame after the N-1th frame; and a function V(N-1) that indicates the density of a point cloud within the voxel. and estimating the position of the sensor based on the evaluation function ρ(N). [Supplementary Note 2] The electronic device calculates an evaluation function ρ(N) based on the evaluation function ρ(N), and estimates the position of the sensor based on the evaluation function ρ(N). ｋ The reception intensity of the point group included in the voxel is expressed as p ｋ The ratio of the detection range of one antenna element to the voxel is V ｊ The currently scanned point cloud is expressed as the following equation (5): The average A(N-1) is expressed as q in the following equation (6): The variance V(N-1) is expressed as C in the following equation (7) (T represents the transpose of a matrix): The electronic device according to claim 1, wherein the evaluation function ρ(N) is given by the following equation (8): [Supplementary Note 3] When there are n points in the voxel, ρ ｊThe electronic device according to Supplementary Note 1 or 2, wherein the position of the sensor is estimated based on a value obtained by multiplying (N) by n points and using the maximum or minimum value of the value. [Supplementary Note 4] The electronic device according to any of Supplements 1 to 3, wherein the function V is a function indicating a ratio of the number of objects detected in the voxel depending on distance. [Supplementary Note 5] The electronic device according to any of Supplements 1 to 4, wherein the function V is a function indicating a ratio of an area or volume of the detection range of the antenna to an area or volume of the voxel. [Supplementary Note 6] The electronic device according to any of Supplements 1 to 5, wherein the function V is a function of an angle at which two objects can be separated. [Supplementary Note 7] The electronic device according to any of Supplements 1 to 6, wherein the position X(N-1) of the object within the voxel that is the predetermined region is calculated based on a result of performing CFAR (Constant False Alarm Rate) processing on a result of FFT (Fast Fourier Transform) processing on values ​​based on the transmitted wave and the reflected wave, and a result of estimating the direction of arrival of the reflected wave. [Supplementary Note 8] The electronic device according to any one of Supplementary Notes 1 to 6, wherein the position X(N-1) of the object within a voxel that is the predetermined region is calculated based on a result of performing CFAR (Constant False Alarm Rate) processing on a result of FFT (Fast Fourier Transform) processing on values ​​based on the transmitted wave and the reflected wave. [Supplementary Note 9] The electronic device according to any one of Supplementary Notes 1 to 6, wherein the position X(N-1) of the object within a voxel that is the predetermined region is calculated based on a result of FFT (Fast Fourier Transform) processing on values ​​based on the transmitted wave and the reflected wave. [Supplementary Note 10] The electronic device according to Supplementary Note 9, wherein a normal distribution is applied to the result of the FFT processing by a Gaussian Mixture Model (GMM) technique to calculate a value after applying the normal distribution.[Supplementary Note 11] A control method for an electronic device that estimates the position of a sensor equipped with an antenna, comprising the steps of: calculating an average A(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within a voxel that is a predetermined region, and a variance V(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within the voxel, based on a position X(N-1) of the object and a reception intensity P(N-1) based on the reflected wave, the position X(N-1) being obtained from a transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object in an N-1th frame that is a predetermined time interval, where N is an integer of 1 or more; A control method comprising: a step of calculating an evaluation function ρ(N) based on a function V(N-1) indicating the density of a point cloud within the voxel; and a step of estimating a position of the sensor based on the evaluation function ρ(N). [Supplementary Note 12] An electronic device that estimates the position of a sensor equipped with an antenna includes the steps of: calculating an average A(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within a voxel that is a predetermined region, based on a position X(N-1) of the object and a reception intensity P(N-1) based on the reflected wave, the position X(N-1) being obtained from a transmission wave transmitted from the antenna and a reflected wave of the transmission wave reflected by the object in an N-1th frame that is a predetermined time interval, where N is an integer of 1 or more; the average A(N-1); the variance V(N-1); the position X(N) of the object obtained from a transmission wave transmitted from the antenna and a reflected wave of the transmission wave reflected by the object in an Nth frame that is the next frame after the N-1th frame; and a function V(N-1) that indicates the density of a point cloud within the voxel. and estimating a position of the sensor based on the evaluation function ρ(N).

[0087] REFERENCE SIGNS LIST 1 Physical layer 2 Transmitting antenna (transmitting antenna array) 3 Receiving antenna (receiving antenna array) 4 RF unit 5 Relay unit 6 DSP (Digital Signal Processor) 7 FPGA (Field Programmable Gate Array) 10 Signal processing layer 12 Antenna setting unit 14 FFT (Fast Fourier Transform) processing unit 16 CFAR (Constant False Alarm Rate) processing unit 18 Direction of arrival estimation unit 20 Application layer 22 Front end 222 NDT (Normal Distribution Transform) processing unit 224 Doppler processing unit 24 Back end 242 Odometry processing unit 244 Loop closing processing unit 246 Pose graph optimization processing unit

Claims

1. An electronic device that estimates the position of a sensor equipped with an antenna, wherein, in an N-1th frame, which is a predetermined time interval (where N is an integer greater than or equal to 1), based on the position X(N-1) of the object and the reception strength P(N-1) based on the reflected wave, the position X(N-1) of the object is obtained from a transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object, the N-1th frame, where N is an integer greater than or equal to 1, the electronic device calculates an average A(N-1) of the product of the position X(N-1) of the object within a voxel, which is a predetermined region, and the reception strength P(N-1) of the product of the position X(N-1) of the object within the voxel and the reception strength P(N-1), and a variance V(N-1) of the product of the position X(N-1) of the object within the voxel, the average A(N-1), the variance V(N-1), the position X(N) of the object obtained from the transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object, in the Nth frame following the N-1th frame, and a function V(N-1) that indicates the density of the point cloud within the voxel. and estimating a position of the sensor based on the evaluation function ρ(N).

2. Let the number of points contained in the voxel be n, and let the number of points contained in the voxel be x ｋ The reception intensity of the point group included in the voxel is expressed as p ｋ The ratio of the detection range of one antenna element to the voxel is V ｊ The currently scanned point cloud is expressed as the following equation (1): The average A(N-1) is expressed as q in the following formula (2): The variance V(N-1) is expressed as C in the following equation (3) (where T represents the transpose of a matrix): The electronic device according to claim 1 , wherein the evaluation function ρ(N) is given by the following equation (4):

3. If there are n points in the voxel, then ρ ｊ 3. The electronic device according to claim 1, wherein the position of the sensor is estimated based on a value obtained by multiplying (N) by n points, using a maximum or minimum value of the value.

4. The electronic device according to any one of claims 1 to 3, wherein the function V is a function that indicates the ratio of the number of detected objects in the voxel depending on the distance.

5. The electronic device according to any one of claims 1 to 4, wherein the function V is a function indicating the ratio of the area or volume of the detection range of the antenna to the area or volume of the voxel.

6. The electronic device according to any one of claims 1 to 5, wherein the function V is a function of the angle at which two objects can be separated.

7. The electronic device according to any one of claims 1 to 6, wherein the position X(N-1) of the object within the voxel that is the predetermined region is calculated based on the result of CFAR (Constant False Alarm Rate) processing performed on the result of FFT (Fast Fourier Transform) processing of values ​​based on the transmitted wave and the reflected wave, and the result of estimating the direction of arrival of the reflected wave.

8. The electronic device according to any one of claims 1 to 7, wherein the position X(N-1) of the object within the voxel that is the predetermined region is calculated based on the result of performing CFAR (Constant False Alarm Rate) processing on the result of FFT (Fast Fourier Transform) processing on values ​​based on the transmitted wave and the reflected wave.

9. The electronic device according to any one of claims 1 to 8, wherein the position X(N-1) of the object within the voxel that is the predetermined region is calculated based on the result of FFT (Fast Fourier Transform) processing of values ​​based on the transmitted wave and the reflected wave.

10. The electronic device according to claim 9, wherein a normal distribution is applied to the result of the FFT processing using a Gaussian Mixture Model (GMM) method to calculate a value after applying the normal distribution.

11. A control method for electronic equipment that estimates the position of a sensor equipped with an antenna, comprising: calculating an average A(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within a voxel, which is a predetermined region, and a variance V(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within the voxel, based on the object's position X(N-1) and the reception intensity P(N-1) obtained from a transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object in the N-1th frame, where N is an integer greater than or equal to 1, which is a predetermined time interval; and and estimating a position of the sensor based on the evaluation function ρ(N).

12. An electronic device that estimates the position of a sensor equipped with an antenna, calculates an average A(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within a voxel that is a predetermined region, based on the object's position X(N-1) and the reception intensity P(N-1) obtained from a transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object in an N-1th frame that is a predetermined time interval, where N is an integer greater than or equal to 1, and a variance V(N-1) of the product of the object's position X(N-1) and the reception intensity P(N-1) within the voxel, the average A(N-1), the variance V(N-1), the object's position X(N) obtained from a transmission wave transmitted from the antenna and a wave reflected from the transmission wave reflected by the object in the Nth frame that follows the N-1th frame, and a function V(N-1) that indicates the density of the point cloud within the voxel. and estimating a position of the sensor based on the evaluation function ρ(N).

Images