Information processing method, information processing device, and program

FMCW LiDAR-based obstacle detection and prediction allow for immediate and accurate trajectory planning, addressing the delay issues in detecting fast-moving obstacles and ensuring safe movement of autonomous objects by varying prediction accuracy with the number of measurement points.

WO2024241892A1PCT designated stage expired Publication Date: 2024-11-28SONY GROUP CORP
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017203
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-05-09
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing obstacle detection methods, such as those using images acquired at different timings, face delays in detecting fast-moving obstacles, which can lead to delayed trajectory planning for autonomous moving objects, potentially resulting in collisions.

Method used

The use of FMCW LiDAR to acquire and process information for predicting the movement of surrounding objects, allowing for immediate obstacle detection and trajectory planning with varying prediction accuracy based on the number of distance measurement points, enabling swift avoidance actions.

Benefits of technology

Enables immediate detection and avoidance of obstacles, reducing the time required for trajectory planning and enhancing safety by predicting the movement of objects with accuracy dependent on the number of LiDAR measurement points, thus preventing collisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017203_28112024_PF_FP_ABST
    Figure JP2024017203_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present technology relates to an information processing method, an information processing device, and a program that make it possible to immediately detect an animal. Information for predicting movement of an object located in the surroundings is acquired through Frequency Modulated Continuous Wave Light Detection and Ranging (FMCW LiDAR), the object is detected using the information, and movement of the object is predicted with prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the object. If the number of ranging points of the FMCW LiDAR on the object is one, a direction in which a plane composed of candidate vectors that can potentially be a movement vector representing the direction in which the object is moving is located is taken as the result of the prediction of movement of the object. As such, the present technology is applicable to, for example, autonomously moving drones.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing method, information processing device, and program

[0001] The present technology relates to an information processing method, an information processing device, and a program, and, for example, to an information processing method, an information processing device, and a program that, when planning a trajectory that is a movement path of a moving object, can instantly detect obstacles and appropriately plan an avoidable trajectory.

[0002] When an autonomous moving object plans a trajectory, which is a path to a destination, the trajectory is planned so as to avoid surrounding obstacles. Patent Document 1 proposes detecting obstacles using images acquired at different times.

[0003] Japanese Patent Publication No. 2023-021910

[0004] In Patent Document 1, since obstacles are detected using images acquired at different times, detection cannot be performed until multiple images are acquired, which may result in a delay in detecting an obstacle that is moving at high speed. It is desirable to be able to detect obstacles immediately and plan an optimal trajectory.

[0005] The present technology has been developed in light of such circumstances, and enables obstacles to be detected immediately.

[0006] An information processing method according to one aspect of the present technology is an information processing method that acquires information for predicting the movement of surrounding objects from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging), detects the objects using the information, and predicts the movement of the objects with a prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the objects.

[0007] An information processing device according to one aspect of the present technology is an information processing device that includes: a detection unit that acquires information for predicting the movement of an object located around a moving body from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging) equipped on the moving body and detects the object using the information; and a prediction unit that predicts the movement of the object with a prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the object detected by the detection unit.

[0008] A program according to one aspect of the present technology is a program for causing a computer to execute processing including steps of acquiring information for predicting the movement of surrounding objects from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging), detecting the object using the information, and predicting the movement of the object with a prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the object.

[0009] In an information processing method, an information processing device, and a program according to one aspect of the present technology, information for predicting the movement of surrounding objects is obtained from an FMCW LiDAR, the object is detected using the information, and the movement of the object is predicted with a prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the object.

[0010] The information processing device may be an independent device or an internal block constituting a single device.

[0011] The program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0012] 10 is a diagram for explaining avoidance behavior. A diagram showing a configuration of an embodiment of a moving body to which the present technology is applied. A flowchart for explaining operation of a moving body. A diagram for explaining operation of an FMCW LiDAR. A diagram for explaining operation of an FMCW LiDAR. A diagram for explaining a movement component. A diagram for explaining movement component. A diagram for explaining detection of an instance. A diagram for explaining estimation of a movement direction. A diagram for explaining estimation of a movement direction. A diagram for explaining estimation of a movement direction. A diagram for explaining estimation of a movement direction. A diagram for explaining estimation of a movement direction. A diagram for explaining estimation of a movement direction. A diagram for explaining estimation of a movement direction when there is one ranging point. A diagram for explaining trajectory planning when there is one ranging point. A diagram for explaining estimation of a movement direction when there are two ranging points. A diagram for explaining trajectory planning when there are two ranging points. A diagram for explaining trajectory planning when there are two ranging points. A diagram for explaining estimation of a movement direction when there are three ranging points. A diagram for explaining trajectory planning when there are three ranging points. A diagram showing another configuration of a moving body to which the present technology is applied. A diagram for explaining detection of an instance. A flowchart for explaining operation of a moving body. 10A and 10B are diagrams for explaining a tracking operation and a configuration of a PC, respectively;

[0013] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described.

[0014] The present disclosure provides a method for planning an autonomously moving vehicle's trajectory, which is a path that avoids obstacles, in accordance with the surrounding conditions. While the following description will be given using a drone as an example of the vehicle, the present disclosure can also be applied to vehicles other than drones.

[0015] As shown in Figure 1, consider a situation in which moving bodies 1-1 and 1-2 are flying in opposite directions. If the moving bodies 1-1 and 1-2 approach each other and there is a possibility of a collision if they continue moving forward, the moving body 1-1 will change course to avoid the moving body 1-2.

[0016] In particular, when the moving body 1-1 and the moving body 1-2 are flying in opposition to each other, the relative speed between them becomes high, so it is desirable that such avoidance action be taken immediately. In the moving body 1 to which the present technology is applied, such avoidance action can be taken immediately.

[0017] <Configuration of Moving Body> FIG. 2 is a diagram showing the configuration of an embodiment of a moving body to which the present technology is applied.

[0018] The moving body 1 includes an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging) 11, a moving object instance segmentation processing unit 12, a moving object movement prediction unit 13, a static environment mapping unit 14, a trajectory planning unit 15, a self-position estimation unit 16, a control unit 17, a GPS (Global Positioning System) 18, an IMU (Inertial Measurement Unit) 19, a barometric pressure sensor 20, a geomagnetic sensor 21, a temperature sensor 22, ESCs (Electronic Stability Controls) 23-1 to 23-4, and motors 24-1 to 24-4.

[0019] 2 may be configured so that all of the units of the moving body 1 are included in the moving body 1, or some of them may be configured as other devices. For example, some or all of the moving body instance segmentation processing unit 12, the moving body movement prediction unit 13, the static environment mapping unit 14, and the trajectory planning unit 15 may be mounted on an information processing device other than the moving body 1, and data may be exchanged between the moving body 1 and the information processing device.

[0020] The operation of the moving body 1 will be described with reference to the flowchart of FIG.

[0021] In step S11, a scan is performed by the FMCW LiDAR 11. The FMCW LiDAR 11 performs ranging using the FMCW LiDAR method (a frequency continuous modulation method). In the FMCW LiDAR method, ranging is performed by frequency modulating a light source and reading the change in frequency between the modulated transmitted light and its reflected light. Here, the measurement principle by the FMCW LiDAR 11 will be explained with reference to FIGS. 4 and 5 . As shown in FIG. 4 , it is assumed that the FMCW LiDAR 11 irradiates transmitted light (TX light) and receives received light (RX light) as reflected light when measuring the distance R to an object 101 to be ranged and the relative speed v of the object 101.

[0022] Figure 5 shows the relationship between transmitted light and received light, with the vertical axis representing the laser frequency and the horizontal axis representing time. In Figure 5, the solid triangular wave represents the transmitted light (TX light), and the dashed triangular wave represents the received light (RX light).

[0023] In this case, the chirp includes a period in which the optical frequency is decreased (down chirp) and a period in which the optical frequency is increased (up chirp), and the combined period of the down chirp and up chirp (Tmod in Figure 4) corresponds to the distance measurement of one point. A LiDAR method that uses this type of chirp light for distance measurement is called an FMCW LiDAR method.

[0024] By measuring the beat frequency in the down chirp (fdown in FIG. 5) and the beat frequency in the up chirp (fup in FIG. 5), the distance R from the FMCW LiDAR 11 to the object 101 and the relative velocity v between the FMCW LiDAR 11 and the object 101 can be calculated using the following equations (1) and (2).

[0025] fdown+fup=4γR / c...(1)

[0026] In equation (1), R represents the distance [m] from the FMCW LiDAR 11 to the object 101. γ represents the chirp rate [Hz / s], and c represents the speed of light [m / s]. Note that the ToF in FIG. 5 can be expressed as τ = 2R / c, and the right-hand side of equation (1) is derived from the relationship between τ and γ.

[0027] fdown-fup=2v / λlaser...(2)

[0028] In equation (2), v represents the relative velocity [m / s] between the FMCW LiDAR 11 and the object 101. λ represents the central wavelength [nm] of the light source. ΔfDoppler in FIG. 5 can be expressed as ΔfDoppler = (v / c)f, and the right-hand side of equation (2) is derived from these relationships.

[0029] The distance R and the relative velocity v can be calculated using equations (1) and (2). Measurements by the FMCW LiDAR 11 also provide information about the direction of movement of the object 101. Because the FMCW LiDAR 11 uses a method for estimating distance from changes in frequency, it is highly robust against adverse weather conditions such as fog and rain. The FMCW LiDAR 11 can also estimate movement information for each light ray direction from Doppler information.

[0030] Estimation of motion information will be described with reference to Fig. 6. A motion vector 111 (referred to as F motion vector 111) on a light ray connecting a light ray from the FMCW LiDAR 11 and the object 101 is motion information obtained from the FMCW LiDAR 11. When the object 101 is moving in a direction and at a speed represented by a motion vector 112, this motion vector 112 (referred to as actual motion vector 112) represents the actual motion of the object 101 as seen from the FMCW LiDAR 11.

[0031] The motion component obtained by projecting the actual motion vector 112 onto the light ray becomes the F motion vector 111. The FMCW LiDAR 11 acquires the F motion vector 111 in the light ray direction as motion information, and can acquire information on the motion direction and speed of the object 101, i.e., information represented by the actual motion vector 112, from the motion information.

[0032] Referring to FIG. 7 , we will explain how the motion components differ depending on the direction of movement of the moving body 1 equipped with the FMCW LiDAR 11 relative to the object. A in FIG. 7 shows a situation in which the moving body 1 is moving toward a stationary object 102. When the direction of movement of the moving body 1 and the surface of the stationary object 102 facing the moving body 1 are nearly perpendicular to each other, a motion component in the ray direction as indicated by the arrow is obtained. In this case, the motion component in the ray direction, i.e., the vector corresponding to the F motion vector 111 described with reference to FIG. 6 , can be obtained with high accuracy. By obtaining the F motion vector 111 with high accuracy, object motion information, i.e., information on the actual motion vector 112, can also be obtained with high accuracy.

[0033] 7B shows a situation in which the moving object 1 is moving parallel to the stationary object 103, for example, the ground. When the moving direction of the moving object 1 and the surface of the stationary object 103 on the moving object 1 side are nearly parallel, a motion component in the ray direction as indicated by the arrow is obtained. In this case, the motion component in the ray direction, i.e., the vector corresponding to the F motion vector 111 described with reference to FIG. 6, becomes small (close to 0), and the motion information of the object, i.e., the information on the actual motion vector 112, also becomes small.

[0034] For this reason, when there is an object approaching the moving body 1, the direction of movement and speed of the approaching object can be detected with high accuracy, but it is thought that the accuracy of detecting the direction of movement and speed of an object that is parallel to the moving body 1 may be reduced. For example, when there is an object that may approach the moving body 1, when detecting the object and taking evasive action to avoid a collision with the object, it is possible to use the FMCW LiDAR 11 to detect objects that may result in a collision, but not detect objects that are unlikely to result in a collision.

[0035] The mobile unit 1 equipped with the FMCW LiDAR 11 can detect the direction and speed of movement of surrounding objects (obstacles), determine whether there is a possibility of approaching the detected object, and if there is a possibility of approaching, plan a trajectory that will prevent the mobile unit from approaching the detected object and move based on the planned trajectory. Furthermore, since such object detection can be performed using a single frame (one frame), as will be described later, the time required from detection to trajectory planning can be shortened.

[0036] Returning to the description of the flowchart shown in Fig. 3, in step S11, a scan is performed by the FMCW LiDAR 11, and ranging data obtained as a result of the scan is supplied as a point cloud to the moving object instance segmentation processing unit 12 and the self-position estimation unit 16. The FMCW LiDAR 11 also outputs information corresponding to a motion component in the light ray direction, i.e., an F motion vector 111, and the motion component in the light ray direction is supplied to the moving object instance segmentation processing unit 12 and the moving object motion prediction unit 13.

[0037] In step S12, the moving object instance segmentation processing is performed by the moving object instance segmentation processing unit 12. The processing performed by the moving object instance segmentation processing unit 12 will be described with reference to FIG.

[0038] 8A shows an image within the angle of view of the FMCW LiDAR 11 of the moving object 1, depicting a state in which moving objects 131-1 and 131-2 are flying. In this state, when a scan is performed by the FMCW LiDAR 11 and point cloud data and movement components in the ray direction are acquired, the moving object instance segmentation processing unit 12 uses the data to perform processing for dividing the image into moving object regions and static object regions.

[0039] The aircraft speed is cancelled from the motion components of each point cloud obtained from the FMCW LiDAR 11, and the area where the motion components remain becomes the moving body area, and the area where no motion components remain becomes the static area. As shown in B of Figure 8, the area corresponding to the moving body 131-1 is detected as the moving body area 132-1, and the area corresponding to the moving body 131-2 is detected as the moving body area 132-2. The area other than the moving body area 131-1 and the moving body area 131-2 is detected as the static area 133.

[0040] Since point cloud data is obtained from the FMCW LiDAR 11, as shown in C of Fig. 8, a cluster of point clouds that are close to each other and located in an area determined to be a moving object area 131, or a cluster of point clouds in an area with close motion components, is treated as a single instance 135. In the example shown in C of Fig. 8, the cluster of point clouds in moving object area 132-1 is treated as instance 135-1, and the cluster of point clouds in moving object area 132-2 is treated as instance 135-2.

[0041] In step S12, the moving object instance segmentation processing unit 12 performs moving object instance segmentation, and when an instance 135 is detected as described with reference to Fig. 8, the process proceeds to step S113. In step S113, the static environment mapping unit 14 creates a static environment map.

[0042] The static environment mapping unit 14 is supplied with a ray direction motion component from the FMCW LiDAR 11, and a moving object list and moving object region information from the moving object instance segmentation processing unit 12. For example, as shown in C of Fig. 8, when instance 135-1 and instance 135-2 are detected, the moving object list and moving object region information are lists that state that the two instances have been detected as moving objects, and are information that can uniquely identify the region (a cluster of point cloud data) detected as instance 135.

[0043] The static environment mapping unit 14 generates a static environment map by removing points in moving object regions from the point cloud obtained from the FMCW LiDAR 11 and accumulating the remaining points in stationary regions over time. This generated static environment map may be a point cloud map (accumulation of points using the vehicle's own position) or may be an occupancy grid map, which is a map divided into voxels and calculated for the occupancy probability of each voxel. The static environment map generated by the static environment mapping unit 14 is supplied to the trajectory planning unit 15.

[0044] In step S13, once the static environment map is generated by the static environment mapping unit 14, the process proceeds to step S14. In step S14, the moving object movement prediction unit 13 determines whether or not a moving object is present. This determination is made by referring to the moving object list, and if a moving object is listed in the moving object list, it is determined that a moving object is present, and if a moving object is not listed in the moving object list, it is determined that a moving object is not present.

[0045] If it is determined in step S14 that no moving object exists, the process proceeds to step S15. In step S15, a trajectory is planned by the trajectory planning unit 15. The trajectory planning unit 15 determines the direction and speed of the aircraft to travel, using the static environment map supplied from the static environment mapping unit 14, the moving object list and moving object movement information supplied from the moving object instance segmentation processing unit 12, and the aircraft's own position and aircraft speed supplied from the control unit 17.

[0046] In this case, since no moving object is detected, the moving object list and moving object movement information supplied from the moving object instance segmentation processing unit 12 indicate that no moving object has been detected. The trajectory planning unit 15 refers to the static environment map, determines a trajectory and speed that will avoid approaching static objects, generates control instructions, and supplies them to the control unit 17.

[0047] In step S16, the control unit 17 controls the airframe based on the control instructions supplied from the trajectory planning unit 15. The control unit 17 controls the attitude, speed, etc. of the airframe. The control unit 17 calculates its own position and speed using information obtained from the self-position estimation unit 16, GPS 18, IMU 19, barometric pressure sensor 20, geomagnetic sensor 21, temperature sensor 22, etc. The control unit 17 controls the ESCs 23-1 to 23-4, and the motors 24-1 to 24-4 are driven based on the control of the ESCs 23-1 to 23-4, thereby controlling the attitude and speed of the airframe.

[0048] On the other hand, if the moving object motion prediction unit 13 determines in step S14 that a moving object is present, the process proceeds to step S17. In step S17, one moving object (instance) is selected from the moving object list. The moving object motion prediction unit 13 selects one instance from the instances listed in the moving object list supplied from the moving object instance segmentation processing unit 12, and makes it a processing target.

[0049] The moving object movement prediction unit 13 predicts the moving direction and speed of the moving object, i.e., the movement of the moving object, in the processes of steps S18 to S25. The process related to the prediction of the movement of the moving object will be described with reference to Fig. 9 and subsequent figures.

[0050] FIG. 9A is the same as FIG. 8C and shows an example of detected instances. As shown in FIG. 9A, instances 135-1 and 135-2 are detected, and information about these two instances is entered in the moving object list. When the moving object movement prediction unit 13 selects instance 135-1 from this moving object list as the selection target, as shown in FIG. 9B, the point cloud clusters that make up instance 135-1 are used to estimate the direction and speed at which instance 135-1 moves.

[0051] Estimation of the moving direction and speed of an object will be described with reference to Figure 10. As described with reference to Figure 6, the FMCW LiDAR 11 acquires a motion component in the light ray direction, and this motion component in the light ray direction becomes a vector obtained by projecting a motion vector representing the direction and speed in which the object is actually moving onto the light ray. Figure 10A shows a situation in which light is irradiated from the FMCW LiDAR 11 onto an object (dynamic obstacle) 131, and motion vectors (F motion vectors) 201-1 to 201-3 in the light ray direction of the light are obtained.

[0052] The F motion vector 201-1 is a component obtained by projecting the actual motion vector (real motion vector) 203-1 (hereinafter, the actual motion vector is indicated by a dotted arrow) of the moving object 131 in the light direction. The F motion vector 201-2 is a component obtained by projecting the actual motion vector 203-2 of the moving object 131 in the light direction. The F motion vector 201-3 is a component obtained by projecting the actual motion vector 203-3 of the moving object 131 in the light direction.

[0053] In this way, an actual motion vector 203 representing the actual movement of the moving object 131 can be obtained from multiple F motion vectors 201 obtained by irradiating light onto the moving object 131. The velocity component obtained from the Doppler information of each point cloud of the FMCW LiDAR 11 is a component obtained by projecting the relative motion vector with respect to the actual moving object 131 in the light ray direction, and therefore there is a possibility of an error in the actual relative movement. For this reason, by using Doppler information obtained from multiple points, three pieces of information in the example shown in Figure 10, the actual relative movement can be estimated, thereby making it possible to obtain relative movement with little error.

[0054] FIG. 10B is a diagram in which the starting points of the F motion vectors 201-1 to 201-3 obtained by the FMCW LiDAR 11 are aligned. The smaller the angle between the light direction (F motion vector 201) and the actual motion vector 203, the longer the F motion vector 201. From this, it can be estimated that the moving object 131 is moving in the direction in which the F motion vector 201 is longest. In the example shown in FIG. 10B, the F motion vector 201-3 is the longest, so it can be estimated that the moving object 131 is moving in the direction of this F motion vector 201-3.

[0055] The example shown in Fig. 10 shows a case where the moving object 131 is moving in an upward direction relative to the FMCW LiDAR 11 (the moving object 1 on which the FMCW LiDAR 11 is mounted). With reference to Fig. 11, a description will be given of a case where the moving object 131 is moving in a direction toward the FMCW LiDAR 11 (the moving object 1 on which the FMCW LiDAR 11 is mounted).

[0056] FIG. 11A shows a situation in which light is irradiated from the FMCW LiDAR 11 onto an object (dynamic obstacle) 131, and F motion vectors 211-1 to 211-3 are obtained in the ray direction of the light. The F motion vector 211-1 is a component of the actual motion vector 213-1 of the moving object 131 projected in the ray direction. The F motion vector 211-2 is a component of the actual motion vector 213-2 of the moving object 131 projected in the ray direction, and in FIG. 11A, the F motion vector 211-2 and the actual motion vector 211-2 are in the same direction, so they overlap. The F motion vector 211-3 is a component of the actual motion vector 213-3 of the moving object 131 projected in the ray direction.

[0057] 11B is a diagram in which the starting points of the F motion vectors 211-1 to 211-3 obtained by the FMCW LiDAR 11 are aligned. In the example shown in FIG. 11B, the F motion vector 211-2 is the longest, so it can be estimated that the moving object 131 is moving in the direction of this F motion vector 211-2. The F motion vector 211-2 is moving in the direction toward the moving object 1, and in this case, the F motion vector 211 is the largest.

[0058] With reference to FIG. 12, a case where the moving object 131 is moving downward relative to the FMCW LiDAR 11 (the moving object 1 on which the FMCW LiDAR 11 is mounted) will be described.

[0059] 12A shows a situation in which light is irradiated from the FMCW LiDAR 11 onto an object (dynamic obstacle) 131, and F motion vectors 221-1 to 221-3 are obtained in the ray direction of the light. The F motion vector 221-1 is a component of the actual motion vector 223-1 of the moving object 131 projected in the ray direction. The F motion vector 221-2 is a component of the actual motion vector 223-2 of the moving object 131 projected in the ray direction. The F motion vector 221-3 is a component of the actual motion vector 223-3 of the moving object 131 projected in the ray direction.

[0060] 12B is a diagram showing the alignment of the start points of the F motion vectors 221-1 to 221-3 obtained by the FMCW LiDAR 11. In the example shown in FIG. 12B, the F motion vector 221-1 is the longest, so it can be estimated that the moving object 131 is moving in the direction of this F motion vector 221-1.

[0061] In this way, by obtaining the F motion vector from a plurality of distance measurement points, the direction of movement of a moving object can be detected.

[0062] <When there is one ranging point> We will explain the calculation of the object's movement direction and speed, and the trajectory planning based on the calculated movement direction and speed, for the cases where the F motion vector is obtained from one ranging point, two ranging points, and three ranging points.

[0063] A case where an F motion vector is obtained from one ranging point will be described with reference to Fig. 13. The situation shown in Fig. 13A is a case where light is irradiated from the FMCW LiDAR 11 onto an object (dynamic obstacle) 131, and an F motion vector 251 in the ray direction of the light is obtained, and this shows a case where there is one ranging point.

[0064] When the F motion vector 251 obtained by the FMCW LiDAR 11 is from only one ranging point, the actual motion vectors from which the F motion vector 251 may be calculated are distributed on a plane perpendicular to the ray direction. In the example shown in A of Figure 13, actual motion vectors 253-1 to 253-7 are shown. When any of these actual motion vectors 253-1 to 253-7 is projected onto the ray, it becomes the F motion vector 251.

[0065] The actual motion vectors 253-1 to 253-7 are vectors extending from the ranging point to a plane a that intersects perpendicularly with the ray, in other words, the F motion vector 251. In other words, the actual motion vector 253 that represents the moving direction and speed of the moving object 131 is not determined to be a single vector, but all vectors that start at the ranging point of the moving object 131 and reach the plane a are candidates.

[0066] 13B, when the starting point of the F motion vector 251 obtained by the FMCW LiDAR 11 and the starting points of the actual motion vectors 253-1 to 253-5 that are candidates for the actual motion vector of the moving object 131 are overlapped, one plane a is determined at the position where the vector reaches. From the F motion vector 251, it can be estimated that the moving object 131 is moving in the direction indicated by plane a. Furthermore, from the F motion vector 251, the relative velocity with respect to the moving object 131 can also be calculated.

[0067] When the F motion vector is obtained from a single distance measurement point, the object's moving direction and relative speed can be estimated to be on a plane. A method for avoiding obstacles in such a case, in other words, trajectory planning, will be described with reference to FIG.

[0068] As shown in Figure 14, if there is only one ranging point, ranging point 255, on object 131, there are an infinite number of motion vectors that can be candidates for actual motion vector 253. Candidate actual motion vectors 253-1 to 253-4 are shown in Figure 14. As described with reference to Figure 13, plane a can be estimated from the candidate actual motion vectors, and it can be estimated that moving object 131 is moving in the direction and at the speed of plane a.

[0069] If the candidate actual motion vector contains a component pointing toward the moving body 1 equipped with the FMCW LiDAR 11, it indicates that the plane a may be moving in a direction toward the moving body 1.

[0070] When scanning the moving object 131, if a motion vector can be detected from only one ranging point, it is possible to estimate whether the moving object 131 is moving toward the moving object 1 (own aircraft). In this way, if there is only one ranging point and a motion component toward the own aircraft is detected, if the moving object 1 moves while maintaining its moving direction and speed, the moving object 1 and the moving object 131 may come close to each other, and in the worst case scenario, they may collide.

[0071] In such a case, a route that includes stopping, decelerating, etc. is planned as the trajectory route of the moving body 1. For example, a deceleration command is issued to the moving body 1 until a plurality of light rays hit the moving body 131, in other words, until the number of ranging points reaches three or more.

[0072] A situation in which a motion vector is obtained from only one ranging point is thought to occur, for example, when the moving object 131 is located at a distance from the moving object 1 or when the moving object 131 is small. When the moving object 1 decelerates and the moving object 131 approaches, the number of ranging points increases to two or three, and as the number of ranging points increases, it is possible to more accurately estimate the moving speed and direction of the moving object 131, as will be described later. For this reason, when there is only one ranging point and it is determined that the moving object 131 may be heading towards the own aircraft, an instruction is issued to slow down the speed of the own aircraft or to remain in place.

[0073] In cases where only one ranging point is obtained because the moving object 131 is located at a distance from the moving object 1, the speed of the moving object 1 may be set in accordance with the speed of the moving object 131. Even if the moving object 131 is located at a distance from the moving object 1, if the speed of the moving object 131 is fast, in other words, if the movement component is large, the time until the moving object 131 approaches the moving object 1 will be short. In such cases, control is performed to slow down the speed of the moving object 1. On the other hand, if the moving object 131 is located at a distance from the moving object 1 and the speed of the moving object 131 is slow, in other words, if the movement component is small, the time until the moving object 131 approaches the moving object 1 will be long. In such cases, control is performed so that the speed of the moving object 1 is not slowed down too much.

[0074] In this way, it is possible to configure the speed of the moving object 1, for example, the upper limit speed of the moving object 1, to be changed depending on the magnitude of the movement component of the moving object 131.

[0075] <When there are two ranging points> Next, a case where there are two ranging points will be described. Fig. 15A shows a state where light is irradiated from the FMCW LiDAR 11 onto an object (dynamic obstacle) 131, and F motion vectors 261-1 and 261-2 in the ray direction of the light are obtained, and shows a case where there are two ranging points.

[0076] The actual motion vectors that can be calculated from the F motion vector 261-1 obtained by the FMCW LiDAR 11 are distributed on a plane a that is perpendicular to the ray direction. In the example shown in A of Figure 15, only the actual motion vector 263-1 is shown, but when any of the actual motion vectors 263-1 that start from the ranging point and reach the plane a are projected onto the ray, they become the F motion vector 261-1.

[0077] Similarly, actual motion vectors that can be calculated from the F motion vector 261-2 obtained by the FMCW LiDAR 11 are distributed on a plane b perpendicular to the ray direction. In the example shown in A of Fig. 15, only the actual motion vector 263-2 is shown, but when any actual motion vector 263-2 that starts at the ranging point and reaches the plane b is projected onto the ray, it becomes the F motion vector 261-2.

[0078] In other words, when there are two ranging points, F motion vector 261-1 and F motion vector 261-2 are obtained from the two ranging points, and plane a consisting of candidate actual motion vector 263-1 for F vector 261-1 and plane b consisting of candidate actual motion vector 263-2 for F vector 261-2 can be estimated.

[0079] 15B, when the starting points of the F motion vectors 261-1 and 261-2 obtained by the FMCW LiDAR 11 and the starting points of the actual motion vectors 263-1 and 263-2 that are candidates for the actual motion vector of the moving object 131 are overlapped, planes a and b are determined at the positions where the vectors reach. From planes a and b, an intersection line 265 where planes a and b intersect can be estimated.

[0080] A vector pointing to a point on the intersection line 265 is an actual motion vector 263 that represents the moving direction and speed of the moving object 131. When the F motion vector is obtained from two ranging points, it can be estimated that the moving direction and relative speed of the object lie on the intersection line 265. A method for avoiding an obstacle in such a case, in other words, trajectory planning, will be described with reference to FIG.

[0081] 16 shows the case where one scan line hits the moving object 131. Range-finding points 267-1, 267-2, and 267-3 shown on the moving object 131 are on one scan line (one straight line). In this case, planes a, b, and c can be estimated from the three range-finding points 267, and for example, two intersection lines can be estimated: an intersection line ab where planes a and b intersect, and an intersection line ac where planes a and c intersect, but the intersection lines ab and ac may be approximately the same straight line.

[0082] In such a case, it can be estimated that the movement direction and relative velocity of the moving object 131 lie on an intersecting line, as in the case described with reference to Figure 15. In the example shown in Figure 16, it is shown that actual motion vectors 263-1 to 263-4, which are candidates for the actual motion vector of the moving object 131, lie on an intersecting line 265.

[0083] 16, the horizontal direction of the actual motion vector 263 (the same direction as the scan line) can be uniquely estimated as the direction in which the intersection line 265 is located, but the vertical direction (the direction perpendicular to the scan line) can only be estimated to be one of the directions on the intersection line 265. In this case, it is not possible to accurately estimate in which direction the moving object 131 is moving in the up and down direction.

[0084] In such a case, the trajectory planning unit 15 plans a moving path 269 that moves in a direction that avoids the direction in which the intersection line 265 is located. FIG. 17 is a diagram showing a situation when viewed from above, which corresponds to the situation shown in FIG. 16 when viewed from the side. As shown in FIG. 17, the direction that is estimated as the moving direction of the moving object 131 is the direction in which the intersection line 265 is located, that is, the direction toward the lower left in the figure. A trajectory that does not move in a direction that intersects with this intersection line 265 is planned. In the example shown in FIG. 17, the moving object 131 is estimated to move in the lower left direction in the figure, so a trajectory is planned for the moving object 1 to move in the upper right direction in the figure.

[0085] In this way, when an object is scanned, if a motion vector is detected from ranging points on one scan line, in other words, if a motion vector is not detected from ranging points on multiple scan lines, or in other words, if only two ranging points hit the object, it is not possible to identify the movement component in a direction perpendicular to the scan plane, but it is possible to at least estimate the direction in which the object is moving on the scan plane. In such cases, since it is possible to estimate the direction in which the object is moving on the scan plane where the object hit, if the route is changed in a direction that avoids the estimated direction, it is possible to avoid the moving object 131 without impeding its movement.

[0086] <When there are three ranging points> Next, a case where there are three ranging points will be described. The situation shown in A of Fig. 18 shows a state in which light is irradiated from the FMCW LiDAR 11 onto an object (dynamic obstacle) 131, and F motion vectors 271-1, 271-2, and 271-3 are obtained in the ray direction of the light, and shows a case in which there are three ranging points. These three ranging points are not on a straight line.

[0087] The actual motion vectors that can be calculated from the F motion vector 271-1 obtained by the FMCW LiDAR 11 are distributed on a plane a that is perpendicular to the ray direction. In the example shown in A of Figure 18, only the actual motion vector 273-1 is shown, but when any of the actual motion vectors 273-1 that start from the ranging point and reach the plane a are projected onto the ray, they become the F motion vector 271-1.

[0088] Similarly, actual motion vectors that can be calculated from the F motion vector 271-2 obtained by the FMCW LiDAR 11 are distributed on a plane b perpendicular to the ray direction. In the example shown in A of Fig. 18, only the actual motion vector 273-2 is shown, but when any actual motion vector 273-2 that starts at the ranging point and reaches the plane b is projected onto the ray, it becomes the F motion vector 271-2.

[0089] Similarly, actual motion vectors that can be calculated from the F motion vector 271-3 obtained by the FMCW LiDAR 11 are distributed on a plane c that is perpendicular to the ray direction. In the example shown in A of Fig. 18, only the actual motion vector 273-3 is shown, but when any actual motion vector 273-3 that starts from the ranging point and reaches the plane c is projected onto the ray, it becomes the F motion vector 271-3.

[0090] In other words, if there are three ranging points, F motion vectors 271-1 to 271-3 can be obtained from the three ranging points, and plane a consisting of candidate actual motion vector 273-1 for F vector 271-1, plane b consisting of candidate actual motion vector 273-2 for F vector 271-2, and plane c consisting of candidate actual motion vector 273-3 for F vector 271-3 can be estimated.

[0091] 18B, when the starting points of the F motion vectors 271-1 to 271-3 obtained by the FMCW LiDAR 11 and the starting points of the actual motion vectors 273-1 to 273-3 that are candidates for the actual motion vector of the moving object 131 are overlapped, planes a, b, and c are determined at the positions where the vectors reach. From planes a, b, and c, an intersection 275 where these planes intersect can be estimated.

[0092] The intersection point 275 where the planes a to c intersect can be determined from the following equation.

[0093] In the equation of plane a, n 1x is a coefficient, and d1 represents the offset amount. Similarly, in the equation for plane b, n 2x is a coefficient, d2 represents the offset amount, and in the equation of plane c, n 3x is a coefficient, and d3 represents the offset amount. These coefficients and offset amounts can be uniquely determined because planes a to c have been determined. By solving these simultaneous equations, the point (x, y, z) where the three planes intersect can be determined. This determined point (x, y, z) is the intersection point 275, which represents the direction of the actual motion vector 273.

[0094] In addition, taking into account the possibility of measurement errors, by using an optimization method such as RANSAC from a set of four or more ranging points, it is possible to more accurately determine the coordinates of the intersection point 275 and the direction of the actual motion vector 273.

[0095] The vector pointing to the intersection 275 is the actual motion vector 273 that represents the moving direction and speed of the moving object 131. When the F motion vector is obtained from three ranging points that are not on a single line, it can be uniquely estimated that the moving direction and relative speed of the object are on the intersection 275. A method for avoiding an obstacle in such a case, in other words, trajectory planning, will be described with reference to FIG.

[0096] The example shown in Figure 19 shows a case where multiple scan lines (two in Figure 19) hit the moving object 131. Range-finding points 277-1 to 277-3 shown on the moving object 131 are on one scan line, and range-finding points 277-4 to 277-6 are on another scan line. In this case, from among range-finding points 277-1 to 277-6, three range-finding points 277 that are not on a straight line can be estimated to be planes a, b, and c, respectively, and an intersection 275 where planes a, b, and c intersect can be estimated.

[0097] 18 , the moving direction and relative velocity of the moving object 131 can be estimated to be on the intersection 275, and can be estimated in one direction. In other words, one actual motion vector 273 can be estimated as the moving direction of the moving object 131.

[0098] In such a case, the trajectory planning unit 15 plans a movement path 279 that avoids the direction indicated by the actual motion vector 273. If the situation shown in FIG. 19 is considered to be a situation when viewed from the side, the corresponding situation when viewed from above can be estimated to be the situation shown in FIG. 17. Referring again to FIG. 17, the direction of movement of the moving object 131 is the direction in which the actual motion vector 273 (the direction in which the intersection line 265 is located in FIG. 17) faces. A trajectory is planned that does not move in a direction that intersects with this actual motion vector 273. In the example shown in FIG. 17, the moving object 131 is estimated to move in the lower left direction in the figure, and therefore a trajectory is planned for the moving object 1 to move in the upper right direction in the figure.

[0099] Alternatively, as shown in FIG. 19, when the actual motion vector 273 is pointing downward, a movement path 279 that moves above the moving object 131 and avoids the moving object 131 may be planned.

[0100] In this way, when an object is scanned and motion vectors are detected from multiple distance measurement points that are not on a straight line, it is also possible to identify the motion component in a direction perpendicular to the scan plane. In such cases, the direction in which the object is moving can be uniquely estimated, and if the route is changed to a direction that avoids the estimated direction, it becomes possible to avoid the moving object 131 without impeding its movement.

[0101] Although the case where there are three distance measurement points has been described as an example, if there are three or more distance measurement points, the direction and speed of movement of the object can be determined with higher accuracy.

[0102] In this way, the accuracy of estimating the object's moving direction and the planned trajectory differ depending on whether the number of ranging points is one, two, or three or more. Therefore, in order to predict the object's movement with different prediction accuracy depending on the number of ranging points, the processing is branched depending on the number of ranging points. Returning to the explanation with reference to the flowchart in Figure 3.

[0103] In step S17, one moving object to be processed is selected from the moving object instance list, and in step S18, it is determined whether or not there are three or more ranging points on the moving object that are not on a straight line. If it is determined in step S18 that there are three or more ranging points on the moving object that are not on a straight line, the process proceeds to step S19.

[0104] In step S19, the three-dimensional movement of the moving object is estimated by the moving object movement prediction unit 13. The process of step S19 is the process when there are three or more ranging points, as described with reference to FIG.

[0105] In step S19, since there are three or more ranging points, the movement of the object can be estimated in three dimensions. In step S20, the predicted trajectory of the moving object is defined as a risk area on the static environment map. After processing in step S20, the process proceeds to step S21.

[0106] In step S21, it is determined whether or not risk assessment has been completed for all animal objects in the animal object instance list. If it is determined in step S21 that risk assessment has been completed for all animal objects in the animal object instance list, the process proceeds to step S15.

[0107] In this way, when the process proceeds to step S15 via the process of step S19, the trajectory planning for the moving object for which the process of step S19 has been performed is performed as described with reference to FIG.

[0108] On the other hand, if it is determined in step S21 that there is an animal object in the animal object instance list that has not been risk-assessed, processing returns to step S17, the animal object that has not been risk-assessed becomes the new processing target, and the subsequent processing is repeated.

[0109] On the other hand, if it is determined in step S18 that there are no three or more ranging points on the moving object that are not on a straight line, the process proceeds to step S22. In step S22, it is determined whether there are only one scan line or two ranging points on the moving object. If it is determined in step S22 that there are only one scan line or two ranging points on the moving object, the process proceeds to step S23.

[0110] In step S23, a candidate motion vector of the moving object is calculated. The process of step S23 is the process when there are two ranging points or when there are points on one scan line, as described with reference to Figures 15 and 16, and calculating the candidate motion vector of the moving object involves finding an intersection line 265 and finding a motion vector pointing toward a point on that intersection line 265.

[0111] In step S23, since the distance measurement points are on two points or one scan line, it is possible to estimate the object's movement toward a point on the intersection line 265. In step S23, the trajectory prediction range of the moving object is defined as a risk area on the static environment map. After processing in step S24, the process proceeds to step S21.

[0112] In step S21, it is determined whether or not risk assessment has been completed for all animal objects in the animal object instance list. If it is determined in step S21 that risk assessment has been completed for all animal objects in the animal object instance list, the process proceeds to step S15.

[0113] In this way, when the process proceeds to step S15 via the process of step S23, the trajectory planning for the moving object for which the process of step S23 has been performed is performed as described with reference to FIGS.

[0114] On the other hand, if it is determined in step S22 that the number of ranging points on the moving object is not one scan line or not only two points, i.e., if it is determined that the number of ranging points on the moving object is one point, processing proceeds to step S25.

[0115] In step S25, the upper limit speed is changed depending on the magnitude of the detected movement component. The process in step S25 is the process when there is only one distance measurement point as described with reference to Fig. 13. For example, if the detected movement component is large, the upper limit speed of the moving object 1 is set to a speed close to 0 (a speed at which the moving object 1 stays in place), and if the detected movement component is small, the upper limit speed of the moving object 1 is set to a speed slightly slower than the upper limit speed set at that time.

[0116] After step S25, the process proceeds to step S21. In step S21, it is determined whether risk assessment has been completed for all animal objects in the animal object instance list. If it is determined in step S21 that risk assessment has been completed for all animal objects in the animal object instance list, the process proceeds to step S15.

[0117] In this way, when the process proceeds to step S15 via step S25, the trajectory planning for the moving object for which the process of step S25 has been performed is performed as described with reference to FIG.

[0118] The trajectory planning unit 15 (FIG. 2) selects the optimal route toward the goal while avoiding obstacles, and supplies control instructions to be taken at the current time to the control unit 17. If there are no moving obstacles, i.e., in the case of processing based on the flowchart shown in FIG. 3, it is determined in step S14 that there are no moving objects, and the processing proceeds to step S15 where trajectory planning is performed, the trajectory planning unit 15 plans, on a static environment map, the shortest and optimal route for flying toward the goal while avoiding obstacles. A search algorithm called A* or the like can be applied as appropriate to this planning.

[0119] When a moving obstacle with a collision risk is present, the trajectory planning unit 15 executes the processing of steps S17 to S21 based on the flowchart shown in Fig. 3. As described above, in the processing of steps S17 to S21, the accuracy of the motion prediction of the moving object varies depending on the number of ranging points of the FMCW LiDAR 11 present on the moving object, so a trajectory planning appropriate for that accuracy is performed. Ultimately, a path that avoids the moving object and heads toward the goal is selected by taking into account the motion prediction results of multiple moving objects while avoiding obstacles on the static environment map and selecting the path with the lowest risk.

[0120] In this way, the moving direction and speed of the object are estimated from the motion vector information obtained by performing ranging using the FMCW LiDAR 11. During the estimation, different estimation processes are performed depending on the number of ranging points, and trajectory planning is performed based on the estimation processes.

[0121] The process leading up to this trajectory planning, i.e., the process shown in the flowchart in Figure 3, is performed for one frame. With this technology, once one frame of data is obtained, the direction and speed of an object's movement can be estimated and a trajectory plan can be created that avoids the object's movement direction. Therefore, even in an environment where a moving object is present, it is possible to instantly detect the moving object, determine its direction of movement, and select a safe route. This also reduces delays and enables instantaneous response to complex changes in the object's movement.

[0122] <Second embodiment> Fig. 20 is a diagram showing an example of the configuration of a moving body 401 in a second embodiment. The moving body 401 shown in Fig. 20 has a configuration in which a camera 411 and a calibration data supply unit are added to the moving body 1 shown in Fig. 2, but is otherwise configured in the same way.

[0123] 20, an image is captured by a camera 411, and the acquired image data is supplied to the moving object instance segmentation processing unit 12. In addition, sensor placement information is also supplied to the moving object instance segmentation processing unit 12 from a calibration data supply unit 412.

[0124] The processing by the moving object instance segmentation processing unit 12 will be further described with reference to Fig. 21 . A in Fig. 21 shows an example of an image captured by the camera 411. The image captures moving objects 131-1 and 131-2. When image data of such an image is supplied from the camera 411 to the moving object instance segmentation processing unit 12, the moving object instance segmentation processing unit 12 analyzes the image based on the supplied image data, as shown in B in Fig. 21 , and extracts instances 135-1 and 135-2. Machine learning can be used for the image analysis.

[0125] The moving object instance segmentation processing unit 12 is also supplied with a point cloud from the FMCW LiDAR 11. As shown in C of FIG. 21 , by using the inter-sensor arrangement information to project the LiDAR point cloud onto an image, it is possible to determine which instance each point belongs to. For example, it can be seen that four LiDAR points belong to instance 135-1, and four LiDAR points belong to instance 135-2. The LiDAR points determined to belong to this instance 135 are used as moving object information and are used in the above-mentioned processes.

[0126] By providing a camera 411 and using image data captured by the camera 411, instances can be detected using the image data, and as a result, LiDAR points located within the area detected as an instance can be detected and used as information about the moving object.

[0127] When detecting instances using only LiDAR points from the FMCW LiDAR 11 described with reference to Fig. 8, it was necessary to calculate the movement components of each LiDAR point in one frame in the process of dividing the area into dynamic and static regions, but by using image data from the camera 411, it is possible to omit the process of calculating the movement components of each LiDAR point and reduce the amount of processing involved in detecting instances. Therefore, it is possible to detect moving objects and calculate a route to avoid the moving objects more quickly.

[0128] The operation of the moving object 401 shown in Fig. 20 will be described with reference to the flowchart of Fig. 22. In step S201, a scan is performed by the FMCW LiDAR 11, and a point cloud of LiDAR points is supplied to the moving object instance segmentation processing unit 12.

[0129] In step S202, the camera 411 captures an image, and the image data is supplied to the moving object instance segmentation processing unit 12.

[0130] In step S203, instance segmentation by machine learning is performed using the image as input by the moving object instance segmentation processing unit 12. The processing in step S203 is performed as described with reference to Figs. 21A and 21B.

[0131] In step S204, the LiDAR point cloud data is transformed into the camera coordinate system, and in step S205, moving object instance segmentation is performed in the LiDAR point cloud data. The processing of steps S204 and S205 is performed as described with reference to C of FIG.

[0132] When an instance of a moving object is detected in this way, processing is performed on the detected instance. In step S206, avoidance and trajectory planning processing is performed. The avoidance and trajectory planning processing performed in step S206 is the processing of steps S15 to S25 in the flowchart of Fig. 3, and since this processing has already been described, its description will be omitted.

[0133] In this way, it is also possible to configure the system so that an image is captured by the camera 411 and an instance is detected using the image.

[0134] Rotating objects such as drone propellers and car tires can become noise. For such objects, the positions of the propellers and tires can be identified by image recognition using images acquired by the camera 411, and such areas can be excluded from the motion estimation area when estimating the motion of the object. In this way, it becomes possible to perform motion estimation with noise removed, thereby enabling more accurate motion estimation.

[0135] If a camera 411 is provided and moving objects are detected by analyzing the captured image, for example, when processing an image captured by the camera 411 of the moving object 401 in which two moving objects overlap, it becomes possible to detect the presence of two moving objects.

[0136] Here, we have explained an example in which a camera 411 is provided and images from the camera 411 are used, but it is also possible to configure the system so that light reflection intensity images obtained by the FMCW LiDAR 11 are used instead of images from the camera 411.

[0137] In the above-described embodiment, an example has been described in which there is one FMCW LiDAR 11, but there may be provided a plurality of FMCW LiDARs 11. For example, it is also possible to provide an FMCW LiDAR 11 in each of the front, rear, left and right directions of the mobile body 11, and to configure the above-described processing to be executed in each FMCW LiDAR 11.

[0138] Third Embodiment A case where the above-mentioned moving body 1 (moving body 401) tracks an object will be described with reference to Fig. 23. As shown in Fig. 23, a case is considered in which the moving body 131 is a vehicle, and the moving body 1 photographs the moving body 131 while maintaining a constant distance from the moving body 131.

[0139] The state in which a constant distance is maintained is a state in which the relative speed between moving object 1 and moving object 131 is maintained at 0. When moving object 131 is moving to the right in the drawing at a speed v, moving object 1 can also move to the right in the drawing at a speed v, thereby enabling tracking to be performed in a state in which the relative speed is maintained at 0.

[0140] As described above, the moving object 1 can calculate the relative speed with respect to the moving object 131 from the Doppler information of each point cloud of the FMCW LiDAR 11. By controlling the moving object 1 so that the calculated relative speed is always 0, highly accurate tracking and photography can be performed.

[0141] For example, the moving body 1 can be configured to include an FMCW LiDAR 11 on the moving body 131 side and an FMCW LiDAR 11 on the side in the direction of travel. In such a configuration, the FMCW LiDAR 11 provided on the moving body 131 side can be used to perform processing to keep the relative speed with respect to the moving body 131 at 0, and the FMCW LiDAR 11 provided on the side in the direction of travel can be used to perform processing to move while avoiding obstacles in the direction of travel.

[0142] By applying this technology, it becomes possible to estimate the movement of a moving object in a single frame, enabling high-speed tracking and close-range tracking.In addition, even if the moving object being tracked moves nonlinearly, it becomes possible to quickly respond to and track that movement.

[0143] <Regarding the Recording Medium> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0144] 24 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes using a program. In the computer, a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, and a RAM (Random Access Memory) 2003 are interconnected by a bus 2004. An input / output interface 2005 is also connected to the bus 2004. An input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010 are connected to the input / output interface 2005.

[0145] The input unit 2006 includes a keyboard, a mouse, a microphone, etc. The output unit 2007 includes a display, a speaker, etc. The storage unit 2008 includes a hard disk, a non-volatile memory, etc. The communication unit 2009 includes a network interface, etc. The drive 2010 drives removable media 2011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0146] In a computer configured as described above, the CPU 2001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 2008 into the RAM 2003 via the input / output interface 2005 and the bus 2004 and executing it.

[0147] The program executed by the computer (CPU 2001) can be provided by being recorded on, for example, a removable medium 2011 such as a package medium. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0148] In the computer, the program can be installed in the storage unit 2008 via the input / output interface 2005 by inserting the removable medium 2011 into the drive 2010. The program can also be received by the communication unit 2009 via a wired or wireless transmission medium and installed in the storage unit 2008. Alternatively, the program can be installed in advance in the ROM 2002 or the storage unit 2008.

[0149] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0150] In this specification, a system refers to an entire device made up of multiple devices.

[0151] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0152] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0153] The present technology can also be configured as follows: (1) An information processing method comprising: acquiring information for predicting the movement of a surrounding object from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging), detecting the object using the information, and predicting the movement of the object with a prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the object. (2) The information processing method described in (1) above, in which, when the FMCW LiDAR has one ranging point on the object, the predicted result of the object's movement is a direction in which a plane made up of candidate vectors that may be motion vectors representing the direction in which the object is moving is located. (3) The information processing method described in (1) or (2) above, in which, when the FMCW LiDAR has two ranging points on the object, the predicted result of the object's movement is a direction in which an intersection line between a first plane obtained from a first ranging point and a second plane obtained from a second ranging point is located, the plane made up of candidate vectors that may be motion vectors representing the direction in which the object is moving. (4) When the FMCW LiDAR has three or more ranging points on the object, the information processing method according to any one of (1) to (3) above, wherein the direction of an intersection of a first plane obtained from a first ranging point, a second plane obtained from a second ranging point, and a third plane obtained from a third ranging point is determined as a predicted result of the object's motion. (5) The information processing method according to any one of (1) to (4) above, wherein the information is a motion component in a light ray direction, and the object is detected by detecting an area where the motion component is present. (6) The information processing method according to (2) above, wherein the upper limit of the speed of a moving object is changed according to the magnitude of the motion component of the object obtained by the FMCW LiDAR at the ranging points. (7) The information processing method according to (3) above, wherein a trajectory of a moving object is planned that avoids the direction in which the intersection line is located. (8) The information processing method according to (4), further comprising planning a trajectory of the moving body that avoids a direction in which the intersection is located.(9) The information processing method according to any one of (1) to (8), analyzing an image captured by a camera to detect an object, and predicting movement of the object using data from the FMCW LiDAR at ranging points within an area where the detected object is located. (10) The information processing method according to any one of (1) to (9), controlling a moving body so that a relative velocity with respect to the object becomes 0. (11) The information processing method according to any one of (6), (7), (8), and (10), wherein the moving body is a drone. (12) An information processing device comprising: a detection unit that acquires information for predicting movement of objects located around the moving body from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging) provided in the moving body, and detects the object using the information; and a prediction unit that predicts movement of the object with prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the object detected by the detection unit. (13) The information processing device according to (12), further comprising: a trajectory planning unit that plans a trajectory for the moving body to move in a direction in which the object will not move, based on the movement of the object predicted by the prediction unit. (14) The information processing device according to (13), further comprising: a control unit that controls the moving body to move on the trajectory planned by the trajectory planning unit. (15) A program for causing a computer to execute processing including steps of: acquiring information for predicting the movement of surrounding objects from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging), detecting the object using the information, and predicting the movement of the object with prediction accuracy that differs depending on the number of ranging points of the FMCW LiDAR on the object.

[0154] 1 Moving object, 11 FMCW LiDAR, 12 Moving object instance segmentation processing unit, 13 Moving object movement prediction unit, 14 Static environment mapping unit, 15 Trajectory planning unit, 16 Self-position estimation unit, 17 Control unit, 18 GPS, 20 Barometric pressure sensor, 21 Geomagnetic sensor, 22 Temperature sensor, 23 ESC, 24 Motor, 101 Object, 111 Motion vector, 112 Actual motion vector, 131 Moving object, 133 Static region, 135 Instance, 201 F motion vector, 203 Actual motion vector, 211 F motion vector, 213 Actual motion vector, 221 F motion vector, 223 Actual motion vector, 251 F motion vector, 253 Actual motion vector, 255 Ranging point, 261 F motion vector, 263 actual motion vector, 265 intersection line, 267 ranging point, 269 movement path, 271 F motion vector, 273 actual motion vector, 275 intersection point, 277 ranging point, 279 movement path, 401 moving body, 411 camera, 413 calibration data supply unit

Claims

1. An information processing method, comprising: acquiring information for predicting the movement of surrounding objects from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging); detecting the objects using the information; and predicting the movement of the objects with a prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the objects.

2. The information processing method according to claim 1, wherein, when the FMCW LiDAR has a single ranging point on the object, the direction in which a plane consisting of candidate vectors that are likely to be motion vectors representing the direction in which the object is moving is located is used as a prediction result of the object's movement.

3. The information processing method according to claim 1, wherein when the FMCW LiDAR has two ranging points on the object, a plane consisting of candidate vectors that are likely to be motion vectors representing the direction in which the object is moving is determined as a prediction result of the movement of the object, and the direction of the intersection line between a first plane obtained from a first ranging point and a second plane obtained from a second ranging point is determined as the direction of the intersection line of the first plane obtained from a first ranging point and a second plane obtained from a second ranging point.

4. The information processing method according to claim 1, wherein when there are three or more ranging points of the FMCW LiDAR on the object, a plane consisting of candidate vectors that may be motion vectors representing the direction in which the object is moving is determined as a prediction result of the movement of the object, and the direction is determined as an intersection point where a first plane obtained from a first ranging point, a second plane obtained from a second ranging point, and a third plane obtained from a third ranging point intersect.

5. The information processing method according to claim 1, wherein the information is a motion component in a light ray direction, and the object is detected by detecting an area in which the motion component exists.

6. The information processing method according to claim 2, further comprising changing an upper limit of a speed of a moving object according to a magnitude of a motion component of the object obtained by the FMCW LiDAR at the range measurement point.

7. The information processing method according to claim 3, further comprising the step of planning a trajectory of a moving object that avoids a direction in which said intersection line is located.

8. The information processing method according to claim 4, further comprising the step of planning a trajectory of a moving object that avoids a direction in which said intersection point is located.

9. The information processing method according to claim 1, further comprising: analyzing an image captured by a camera, detecting the object; and predicting the movement of the object using data from the FMCW LiDAR at a ranging point within an area in which the detected object is located.

10. The information processing method according to claim 1, further comprising controlling a moving object so that a relative velocity between said moving object and said moving object becomes zero.

11. The information processing method according to any one of claims 6, 7, 8 and 10, wherein the moving object is a drone.

12. An information processing device comprising: a detection unit that acquires information for predicting the movement of objects located around a moving body from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging) equipped on the moving body and detects the object using the information; and a prediction unit that predicts the movement of the object with a prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the object detected by the detection unit.

13. The information processing device according to claim 12, further comprising a trajectory planning unit which plans a trajectory for the moving body to move in a direction in which the object does not move, based on the movement of the object predicted by the prediction unit.

14. The information processing device according to claim 13, further comprising a control unit that controls the moving body so that it moves along the trajectory planned by the trajectory planning unit.

15. A program for causing a computer to execute a process including the steps of: acquiring information for predicting the movement of surrounding objects from an FMCW LiDAR (Frequency Modulated Continuous Wave Light Detection and Ranging); detecting the object using the information; and predicting the movement of the object with a prediction accuracy that varies depending on the number of ranging points of the FMCW LiDAR on the object.

Citation Information

Patent Citations

  • Object detection system and vehicle equipped with object detection system

    JP2009041981A

  • Radar device

    JP2017003386A

  • Target detector and target detection method

    JP2019200154A

  • Method and system for controlling an autonomous vehicle using a coherent range Doppler optical sensor

    JP2022058483A

  • Flight management mode transitioning for aircraft trajectory management

    US20160358481A1