Method for performing a control function of an autonomous vehicle

Abstract

Method for performing a control function of an autonomous vehicle (10), the method comprising: Receiving a first point cloud and at least a second time-separated point cloud, based on three-dimensional position data (402) of a scene located outside the autonomous vehicle (10), from at least one sensor (42n) of the autonomous vehicle (10); Position alignment, via at least one processor (44), of at least one static aspect of the scene in the first point cloud with at least one static aspect of the scene in the at least second cloud, to obtain position-aligned first and second point clouds; Determine, via at least one processor (44), a velocity of at least one moving object in the position-oriented first and second point clouds; Performing, via at least one processor (44), a control function of the autonomous vehicle (10) using the speed of the at least one moving object; and Segmenting the first and second point clouds into moving (413a,b) or static data points (411a,b), wherein the position alignment step uses the static data points as the at least one static aspect.

Description

[0001] This description relates generally to vehicles and, more particularly, to systems and methods related to determining the speed of moving objects to control autonomous vehicles.

[0002] An autonomous vehicle is a vehicle capable of sensing its environment and navigating with little or no user input. This occurs through the use of sensing devices such as radar, sensors, image sensors, and the like. Autonomous vehicles also utilize information from global positioning systems (GPS), navigation systems, vehicle-to-vehicle communications, vehicle infrastructure technologies, and / or wired systems to navigate the vehicle.

[0003] While autonomous vehicles offer many potential advantages over conventional vehicles, in certain circumstances it may be desirable for autonomous vehicles to operate better, such as by processing effective ways to determine the speed of moving objects.

[0004] US 8 989 944 B1 describes a method comprising receiving a first set of points based on the detection of an environment of an autonomous vehicle during a first period of time and selecting a plurality of points from the first set of points that form a first point cloud representing an object in the environment, receiving a second set of points based on the detection of the environment during a second period of time that is after the first period of time, selecting a plurality of points from the second set of points that form a second point cloud representing the object in the environment, determining a transformation between the selected points from the first set of points and the selected points from the second set of points, wherein the transformation is used to determine a speed of the object,and providing instructions for controlling the autonomous vehicle based at least in part on the speed of the object.

[0005] US 8 886 387 B1 describes a computing device that can identify an object in an environment of a vehicle and receive a first three-dimensional (3D) point cloud representing a first view of the object. The computing device can determine a reference point on the object in the first 3D point cloud and receive a second 3D point cloud representing a second view of the object. The computing device can determine a transformation between the first view and the second view and estimate a projection of the reference point from the first view relative to the second view based on the transformation to track the reference point from the first view to the second view. The computing device can determine one or more motion characteristics of the object based on the projection of the reference point.

[0006] US 2016 / 0 068 114 A1 describes methods and systems for vision-based calculation of self-motions.

[0007] US 9 892 328 B2 describes computer-assisted methods that can be carried out by a driver assistance system while the driver's own vehicle is driving. The driver assistance system comprises a camera that can be connected to a processor. The first and second image sections are captured from the camera's field of view. Corresponding pixels on the road are tracked from the first image frame to the second image frame. Image movements between the corresponding pixels on the road are processed in order to detect a hazard on the road. It is determined that the corresponding pixels represent a moving shadow on the road in order to avoid a false positive detection of a hazard on the road, or it is determined that the corresponding pixels do not represent a moving shadow on the road in order to verify the detection of a hazard on the road.

[0008] US 9 043 069 B1 describes methods and apparatus for estimating a course of a target vehicle. An exemplary method may include determining a first point cloud representing the location of a target vehicle in a first time period and a second point cloud representing the location of the target vehicle in a second time period. Using a computing device, an initial comparison may be determined between the first point cloud and the second point cloud based on an estimate of a speed for the target vehicle and a time difference between the first time period and the second time period. Furthermore, the initial comparison may be revised based on minimizing a distance between points of the first point cloud and corresponding points of the second point cloud. Based on the revised comparison, an estimate of the heading of the target vehicle may then be determined.

[0009] It can be considered a task to provide an improved method for determining the speed of moving objects.

[0010] According to the invention, a method for performing a control function of an autonomous vehicle includes receiving first and second time-spaced point clouds based on three-dimensional position data of a scene located external to the autonomous vehicle from a sensor of the autonomous vehicle. The method includes position aligning, via a processor, a static aspect of the scene in the first point cloud with a static aspect of the scene in the second cloud to obtain first and second point clouds whose position is aligned. The method includes determining, via a processor, a speed of a moving object in first and second point clouds whose position is aligned. The method includes performing a control function of the autonomous vehicle via a process using the speed of the moving object.The method involves segmenting the first and second point clouds into moving and static data points. The position alignment step may use static data points as the static aspect.

[0011] In embodiments, the determining step includes determining a distance traveled by the moving object between the first or second position-aligned point clouds. The speed of the moving object is determined based on the distance traveled.

[0012] In embodiments, the determining step includes using an elapsed time between the first and second point clouds to determine the speed. The first and second point clouds may have a temporal separation defined by a frame rate of the sensor.

[0013] In embodiments, the method comprises identifying a cluster of data points in the temporally spaced first and second point clouds that correspond to the moving object. The method may comprise matching an identified cluster of data points located in the point cloud with an identified cluster of data points located in the second point cloud. The matching step may comprise determining a spatial transformation of a cluster of data points in the first point cloud with a cluster of data points in the second point cloud. The determining step may comprise determining the speed of the object based on a distance traveled between matched clusters of data points in the first and second point clouds, with their position aligned. The determining step may derive the distance traveled from the spatial transformation.

[0014] In embodiments, the method includes the determining step using moving data points in the first and second point clouds whose positions are aligned to determine the speed of the object. The preceding step of identifying a cluster of data points in both the first and second temporally spaced point clouds corresponding to a moving object may use the moving data points.

[0015] In embodiments, the method includes iteratively calculating the closest point to determine a spatial transformation between clusters of moving data points. The spatial transformation allows the distance traveled for the moving object to be determined, wherein the speed of the moving object is determined based on the distance traveled. In other embodiments, the method includes generating a mesh around the data points in the first and second point clouds corresponding to the moving object. The mesh may be arranged around an identified cluster of moving data points according to the segmentation and identification steps described above.The method may include determining a spatial transformation of the mesh in the first and second point clouds whose positions are aligned, thereby enabling the distance traveled to be determined. The speed is determined based on the distance traveled.

[0016] In embodiments, the first and second point clouds correspond to consecutive frames of the scene captured by sensors. The acquisition, position alignment, and detection steps can be performed for each set of consecutive frames of the scene captured by the sensor.

[0017] In some embodiments, the sensor is a lidar sensor or other ranging sensor, such as a radar sensor. The sensor can also be an optical camera. Fig. 1 is a functional block diagram illustrating an autonomous vehicle; Fig. 2 is a functional block diagram illustrating a transport system with one or more autonomous vehicles from Fig. 1 illustrates; Fig. 3 is a functional block diagram illustrating an autonomous driving system (ADS) in conjunction with an autonomous vehicle; Fig. 4 is a diagram illustrating a module of a point cloud processing system for determining the speed of objects and the data flow between the modules; and Fig. 5 is a flowchart for a control method for determining velocities of objects based on point cloud data.

[0018] As mentioned with reference to Fig. 1, a control system, generally depicted at 100, is associated with a vehicle 10 according to various embodiments. Generally, the system 100 provides low-level processing of three-dimensional images of the surroundings of the vehicle 10 in the form of point clouds to estimate the speed of surrounding objects for use in controlling the vehicle 10.

[0019] As in Fig. 1, the vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially encloses the other components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The wheels 16-18 are each rotatably coupled to the chassis 12 near a respective corner of the body 14.

[0020] In various embodiments, the vehicle 10 is an autonomous vehicle, and the system 100 and / or components thereof are integrated into the autonomous vehicle 10 (hereinafter referred to as the autonomous vehicle 10). The autonomous vehicle 10 is, for example, a vehicle that is automatically controlled to transport passengers from one location to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be noted that any other vehicle, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), vessels, aircraft, and the like, may be used.

[0021] In an exemplary embodiment, the autonomous vehicle 10 corresponds to a level four or level five automation system according to the Society of Automotive Engineers (SAE) standard taxonomy of automated driving levels, "J3016." Using this terminology, a level four system indicates "high automation," referring to a driving mode in which the automated driving system assumes all aspects of the dynamic driving task, even if a human driver does not adequately respond to a request for intervention. A level five system, on the other hand, exhibits "full automation," referring to a driving mode in which the automated driving system performs all aspects of the dynamic driving task under all road and environmental conditions that a human driver can handle.It should be understood that the embodiments according to the present subject matter are not limited to any particular taxonomy or rubric of automation categories. Furthermore, systems according to the present embodiment may be used in conjunction with any autonomous or other vehicle that utilizes a navigation system and / or other systems for route guidance and / or implementation.

[0022] As illustrated, the autonomous vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. The propulsion system 20, in various embodiments, may include an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transfer power from the propulsion system 20 to the vehicle wheels 16 and 18 according to selectable gear ratios. According to various embodiments, the transmission system 22 may include a stepped-ratio automatic transmission, a continuously variable transmission, or other suitable transmission.

[0023] The braking system 26 is configured to provide braking torque to the vehicle wheels 16 and 18. In various embodiments, the braking system 26 may include friction brakes, brake-by-wire, a regenerative braking system such as an electric machine, and / or other suitable braking systems.

[0024] The steering system 24 influences a position of the vehicle wheels 16 and / or 18. While illustrated as a steering wheel 25 for illustrative purposes in some embodiments within the scope of the present description, the steering system 24 may not include a steering wheel.

[0025] The sensor system 28 includes one or more sensor devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radars, lidars, global positioning systems, optical cameras, thermal imaging cameras, ultrasonic sensors, and / or other sensors. The actuator system 30 includes one or more actuator devices 42a-42n that control one or more vehicle features, such as, but are not limited to, the propulsion system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the autonomous vehicle 10 may also include vehicle interior and / or exterior features not included in Fig. 1, such as various doors, trunk and cabin features, such as air, music, lighting, touchscreen display components (such as those used in conjunction with navigation systems) and the like.

[0026] The data storage device 32 stores data for use in automatically controlling the autonomous vehicle 10. In various embodiments, the data storage device 32 stores defined maps of the navigable environment. In various embodiments, the defined maps are predefined and retrieved from a remote system (in further detail with respect to Fig. 2). For example, the defined maps may be assembled by the remote system and communicated to the autonomous vehicle 10 (wirelessly and / or wired) and stored in the data storage device 32. Route information may also be stored in the data device 32—i.e., a series of road segments (geographically linked to one or more of the defined maps) that together define a route the user may take from a starting location (e.g., the user's current location) to a destination location. Also, in various embodiments, the data storage device 32 stores processing algorithms and data for processing three-dimensional point clouds to determine the speed of objects in the environment on a frame-by-frame basis.As can be seen, the data storage device 32 may be part of the controller 34, separate from the controller 34, or part of the controller 34 and part of a separate system.

[0027] The controller 34 includes at least one processor 44 and a computer-readable storage device or media 46. The processor 44 may be a custom-made or off-the-shelf processor, a central processing unit (CPU), a graphics processing unit (GPU) among multiple processors connected to the controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a combination thereof, or generally any device for executing instructions. The computer-readable storage device or media 46 may include volatile and non-volatile memory in read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operating variables while the processor 44 is powered off.The computer-readable storage device or media 46 may be implemented using any of a number of known storage devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or any other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represent executable instructions used by the controller 34 in controlling the autonomous vehicle 10.

[0028] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. The instructions, when executed by processor 44, receive and process signals from sensor system 28, perform logic, calculations, methods, and / or algorithms to automatically control the components of autonomous vehicle 10, and generate control signals that are transmitted to actuator system 30 to automatically control the components of autonomous vehicle 10 based on the logic, calculations, methods, and / or algorithms. Although in Fig. 1, embodiments of the autonomous vehicle 10 may include any number of controllers 34 that communicate and cooperate via any suitable communication medium or combination of communication media to process the sensor signals, perform logic, calculations, methods, and / or algorithms, and generate control signals to automatically control the functions of the autonomous vehicle 10. In one embodiment, as discussed in detail below, the controller 34 is configured to process three-dimensional imaging data of the environment of the vehicle 10 in the form of point clouds to determine speed on a frame-by-frame basis. The controller 34 is further configured to perform autonomous control functions based on a determined speed.

[0029] The communication system 36 is configured to wirelessly transmit information to and from other units 48, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote transportation systems, and / or user devices (relating to Fig. 2). In an exemplary embodiment, the wireless communication system 36 is configured to communicate over a wireless local area network (WLAN) using the IEEE 802.11 standard, via Bluetooth, or via mobile data communication. However, additional or alternative communication methods, such as a dedicated short-range radio communication (DSRC) channel, are also contemplated within the scope of this description. DSRC channels refer to one-way or two-way short- to mid-range radio communication channels specifically designed for automotive applications and a corresponding set of protocols and standards.

[0030] With further reference to Fig. 2 in various embodiments, the autonomous vehicle 10 described with reference to Fig. 1, may be suitable for use as part of a taxi or shuttle service within a specific geographic area (e.g., a city, a school or business campus, a shopping mall, an amusement park, an event center, or the like). For example, the autonomous vehicle 10 may be associated with an autonomous vehicle-based transportation system. Fig. Figure 2 illustrates an exemplary embodiment of an operating environment, shown generally at 50, including an autonomous vehicle-based transportation system (or simply “remote transportation system”) 52, which, as described with reference to Fig. 1, is associated with one or more autonomous vehicles 10a-10n. In various embodiments, the operating environment 50 (which may fully or partially correspond to the Fig. 1) further comprises one or more user devices 54 that communicate with the autonomous vehicle 10 and / or the remote transportation system 52 via a communications network 56.

[0031] The communications network 56 supports communication between devices, systems, and components supported by the operating environment 50 (e.g., via physical communication links and / or wireless communication links). For example, the communications network 56 may include a wireless carrier system 60, such as a cellular phone system, including a plurality of cellular towers (not shown), one or more mobile switching centers (MSCs) (not shown), and any other network components necessary to connect the wireless carrier system 60 to the fixed network. Each cellular tower includes transmit and receive antennas and a base station, with the base stations of various cellular towers connected to the MSCs, either directly or through intermediate devices, such as a base station controller.The wireless carrier system 60 may implement any suitable communications technology, for example, digital technologies such as CDMA (e.g., CDMA2000), LTE (e.g., 4G LTE or 5G LTE), GSM / GPRS, or other current or emerging wireless technologies. Other cell tower / base station / MSC arrangements are possible and could be used with the wireless carrier system 60. For example, the base station and cell tower could be co-located or remote from each other, each base station could serve a single cell tower, a single base station could serve multiple cell towers, or multiple base stations could be coupled to a single MSC, to name a few of the possible arrangements.

[0032] In addition to using the wireless carrier system 60, a second wireless carrier system in the form of a satellite communications system 64 may be used to provide unidirectional or bidirectional communications with the autonomous vehicle 10a-10n. This may be accomplished using one or more communications satellites (not shown) and an uplink transmitting station (not shown). Unidirectional communications may include, for example, satellite radio services, wherein programmed content data (news, music, and the like) is received from the transmitting station, packaged for upload, and then transmitted to the satellite, which broadcasts the programming to subscribers. Bidirectional communications may include, for example, satellite telephone services that utilize the satellite to relay telephone communications between the vehicle 10 and the station.Satellite telephony can be used either in addition to or instead of the mobile operator system 60.

[0033] A fixed-line communication system 62 may include a conventional fixed-line telecommunications network connected to one or more landline telephones and connecting the wireless carrier system 60 to the remote transport system 52. For example, the fixed-line communication system 62 may be a public switched telephone network (PSTN) such as those used to provide wired telephony, packet-switched data communications, and the Internet infrastructure. One or more segments of the fixed-line communication system 62 could be implemented using a standard wired network, a fiber optic or other optical network, a cable network, power lines, other wireless networks such as wireless local area networks (WLANs) or networks providing wireless broadband access (BWA), or any combination thereof.Furthermore, the remote transport system 52 need not be connected via the fixed network communication system 62, but could include radiotelephone equipment so that it can communicate directly with a wireless network, such as the wireless carrier system 60.

[0034] Although in Fig. 2, embodiments of the operating environment 50 may support any number of user devices 54, including multiple user devices 54 owned, operated, or otherwise used by a person. Each user device 54 supported by the operating environment 50 may be implemented using any suitable hardware platform. In this regard, the user device 54 may be implemented in a common form factor, including: a desktop computer; a mobile computer (e.g., a tablet computer, a laptop computer, or a netbook computer); a smartphone; a video game device; a digital media player; a component of a home entertainment device; a digital camera or video camera; a wearable computing device (e.g., a smart watch, smart glasses, smart clothing); or the like.Each user device 54 supported by operating environment 50 is embodied as a computer-implemented or computer-based device having the hardware, software, firmware, and / or processing logic necessary to perform the various techniques and methods described herein. For example, user device 54 includes a microprocessor in the form of a programmable device that includes one or more instructions stored in an internal memory structure and is operable to receive binary inputs and generate binary outputs. In some embodiments, user device 54 includes a GPS module that can receive GPS satellite signals and generate GPS coordinates based on those signals.In further embodiments, user device 54 includes cellular communication functionality such that the device performs voice and / or data communications over communication network 56 using one or more cellular communication protocols, as explained herein. In various embodiments, user device 54 includes a visual display, such as a graphical touchscreen display or other display.

[0035] The remote transportation system 52 includes one or more backend server systems (not shown) that may be cloud-based, network-based, or resident at the particular campus or geographic location served by the transportation system 52. The remote transportation system 52 may be staffed with a live advisor, an automated advisor, an artificial intelligence system, or a combination thereof. The remote transportation system 52 may communicate with the user devices 54 and the autonomous vehicles 10a-10n to schedule trips, dispatch autonomous vehicles 10a-10n, and the like. In various embodiments, the remote transportation system 52 stores account information, such as subscriber authentication data, vehicle license plates, profile records, biometric data, behavior patterns, and other related subscriber information.In one embodiment, as described in more detail below, the remote transportation system 52 includes a route database 53 in which information about the routes of the navigation system is stored, including lane markings for lanes along the various routes, and whether and to what extent certain route sections are affected by construction areas or other possible hazards or obstacles detected by one or more autonomous vehicles 10a-10n.

[0036] According to a typical use case workflow, a registered user of the remote transportation system 52 may create a ride request via the user device 54. The ride request typically specifies the passenger's desired pickup location (or current GPS location), the desired destination (which may identify a predefined vehicle stop and / or a user-defined passenger destination), and a pickup time. The remote transportation system 52 receives the ride request, processes the request, and dispatches a selected one of the autonomous vehicles 10a-10n (when and if available) to pick up the passenger at the designated pickup location and time. The transportation system 52 may also generate and send an appropriately configured confirmation message or notification to the user device 54 to notify the passenger that a vehicle is en route.

[0037] As can be seen, the subject matter disclosed herein provides certain enhanced features and functionality for what may be considered a standard or baseline autonomous vehicle 10 and / or an autonomous vehicle-based transportation system 52. To this end, an autonomous vehicle-based transportation system may be modified, enhanced, or otherwise supplemented to provide the additional functionality described in more detail below.

[0038] According to various embodiments, the controller 34 implements an autonomous drive system (ADS) 70, as shown in Fig. 3. That is, suitable software and / or hardware components of the controller 34 (e.g., the processor 44 and the computer-readable storage medium 46) are used to provide an autonomous propulsion system 70 used in conjunction with the vehicle 10.

[0039] In various embodiments, the instructions of the autonomous driving system 70 may be structured according to function or system. For example, the autonomous driving system 70 may, as shown in Fig. 3, include a sensor fusion system 74, a positioning system 76, a steering system 78, and a vehicle control system 80. As can be appreciated, in various embodiments, the instructions may be divided into any number of systems (e.g., combined, further subdivided, etc.), as the description is not limited to the present examples.

[0040] In various embodiments, the sensor fusion system 74 synthesizes and processes sensor data and predicts the presence, location, classification, and / or history of objects and features in the environment of the vehicle 10. In various embodiments, the sensor fusion system 74 may include information from multiple sensors, including, but not limited to, cameras, lidars, radars, and / or any number of other types of sensors.

[0041] The positioning system 76 processes sensor data along with other data to determine a position (e.g., a local position relative to a map, an exact position relative to the lane of a road, vehicle direction, speed, etc.) of the vehicle 10 relative to its surroundings. The guidance system 78 processes sensor data along with other data to determine a route for the vehicle 10 to follow. The vehicle control system 80 generates control signals for controlling the vehicle 10 according to the determined route.

[0042] In various embodiments, the controller 34 implements machine learning techniques to support the functionality of the controller 34, such as feature detection / classification, obstacle mitigation, route traversal, mapping, sensor integration, ground truth determination, and the like.

[0043] The autonomous drive system is configured to generate drive outputs that are communicated to the actuator system 30. In an exemplary embodiment, the actuators 42 include a steering control, a shift control, a throttle control, and a brake control. The steering control may, for example, control a steering system 24, as shown in Fig. 1. The gearshift control may, for example, control a transmission system 22 as shown in Fig. 1. The throttle control may, for example, control a drive system 20 as shown in Fig. 1. The brake control may, for example, control the wheel brake system 26, as shown in Fig. 1 shown.

[0044] As briefly mentioned above, the system 100 is made of Fig. 1 Part of the autonomous propulsion system 70, for example, as a system for processing lidar data or other three-dimensional position data, point clouds, to determine the speed of surrounding objects. The systems and methods for determining the speed described below with reference to the Fig. 4 and Fig. 5 are incorporated into the positioning system 76 in embodiments and may be incorporated downstream of the sensor fusion system 74.

[0045] For example, with reference to Fig. 4 and further reference to Fig. 3, the system 100 includes a point cloud processing system 400. The point cloud processing system 400 of Fig. 4 is implemented in various embodiments by at least one processor 44, which executes the instructions received from the computer-readable medium 46. The point cloud processing system 400 is configured to process frames from the point clouds obtained by the sensor system 28 to determine the speed of surrounding objects for use in the autonomous propulsion system 70, particularly as part of post-tracking functions of the positioning system 76 and as a cross-check localization of the vehicle 10 as determined by the positioning system 76.

[0046] The sensor system 28 receives three-dimensional imaging data 402 of the surroundings of the vehicle 10 through the sensor devices 42n. The three-dimensional imaging data 402 may be obtained from a lidar device, a radar device, or other ranging device 42n, or an optical camera. The three-dimensional data 402 may be divided into frames, with the sensor device 42n capturing the time-spaced images of each scene surrounding the vehicle 10. The scene surrounding the vehicle 10 includes objects (not shown) that can be identified and identified by the methods described herein with reference to the Fig. 4 and Fig. The speed of the object can be determined using the processing method described in Figure 5. The three-dimensional imaging data 402 can be obtained by the sensor fusion system 74.

[0047] The sensor device 42n generally has a frame rate-defining time elapsed between acquired images. For example, in the case of the rotating lidar device 42n, consecutive frames are acquired in successive rotations of the lidar device 42n. The three-dimensional data 402 obtained from the sensor system 28 can be processed by the sensor fusion system 74, which can be integrated therein as part of a data receiving module 404, to generate point cloud data 406. The point cloud data 406 includes an array of points distributed in the x and y dimensions, each of which includes range data (and other data fields, such as intensity) in the z direction. Although the Cartesian coordinate system is described herein, other three-dimensional coordinate systems, such as polar coordinates, can be used.Frames of point cloud data 406 are obtained, each frame being based on frames of the three-dimensional imaging data 402. . Fig. 4 shows exemplary first and second frames of the point cloud data 406a, 406b that can be used to determine the speed of the moving objects in the point cloud data 406. However, it should be understood that one or more of the most recent frames 406a can be processed as at least a second frame of the point cloud data 406b with respect to a plurality of the previous frames of the point cloud data.

[0048] The point cloud processing system 400 includes a static / moving segmentation module 408 configured to obtain the first point cloud and at least one second point cloud 406a, 406b and to segment moving or static data points in the first point cloud 406a. The static / moving segmentation module 408 outputs first and second point clouds 410a, 410b, each including segmented moving data points 413a,b and segmented static data points 411a,b (the second point cloud 410b that was previously segmented). That is, each data point can include an identification field of either static or moving data. The moving data points 413a,b are divided into clusters of moving data points 413a,b.

[0049] The static / moving segmentation module 408 may use the velocity parameters determined from the previous point cloud analysis, as described below with reference to the velocity calculation module 424. Once the velocity parameters are derived from previous point clouds 406b, they may be grouped (i.e., segmented) to have the same derived rotational / linear velocities. Static objects have rotational / linear velocities close to 0. Accordingly, previous velocity information may be transposed to a current scene of data points to infer current velocities, thus allowing segmentation of static and moving data.

[0050] The point cloud processing system 400 includes a static scene alignment module 414 configured to use the segmented static data points 411a,b to positionally align the first point cloud 410a with at least one second point cloud 410b, which are less recent point clouds. This means that the static features identified by the static data points 411a,b in the first point cloud 410a are aligned with corresponding static features in the second point cloud 410a.The static scene alignment module 414 is configured to output the positionally aligned first and second point clouds 416 whose moving data points 413a from the first point cloud 410a are relatively displaced compared to the moving data points 413b from the second point cloud 410b in a common static frame provided by the positionally aligned first and second point clouds 416. The relatively moving data points 413a,b form clusters of moving data points 413a from the first point cloud 410a that are displaced to a position relative to the clusters of moving data points 413b from the second point cloud when viewed in a static reference frame provided by the positionally aligned first and second point clouds 416.

[0051] The shifted position of the point clusters 413a,b of the moving data is a three-dimensional translation in the x, y, and z dimensions and may also include rotational movements. A distance traveled is defined by the relative displacement of the position, which can be used to determine a speed of movement of an object corresponding to each moving cluster of data points 413a,b.

[0052] In one embodiment, the static scene alignment module 414 is configured to register a static map (created in advance) with more recent point clouds 416. Registering the point cloud to the static map may use methods such as iterative nearest points or other optimization techniques to minimize the difference between the point clouds 416 and the static map. The static map may consist of closely spaced point clouds that have been preprocessed and aligned relative to each other (and possibly meshed).

[0053] In another embodiment, the static scene alignment module 414 is configured to register the point clouds 416 based on visual odometry methods, in which two frames are compared and the difference between them is minimized. Such methods are capable of removing errors from inert sensors while simultaneously building a high-resolution local map.

[0054] The point cloud processing system 400 includes an object matching module 418 configured to match or register corresponding clusters of motion data points 413a, 413b in the positionally aligned first and second point clouds 416. This means that corresponding clusters of moving points 413a, 413b are identified, and matching and registration are performed to register or align a cluster of motion data points 413a from the first positionally aligned point cloud with a cluster of motion data points 413b from the second positionally aligned point cloud.

[0055] There are a number of possibilities for identifying matched motion data points 413a, 413b. For example, grouping can be performed based on similar rotation / translation velocities. The grouping can be based on similar shapes between current point clouds 416. The grouping can be based on shapes in a predetermined obstacle set. For example, one element of this obstacle set could be a particular vehicle model that tends to look the same regardless of where and when it is seen. Another possibility would be grouping based on proximity to a cluster seen in previous point clouds, often with a prediction that states where it is now based on where it was in the past and how it was moving at that time.

[0056] The matching or registration procedure performed by the object matching module 418 generates a transformation matrix t1..t n for each identified cluster of movement data points o1..o n , which correspond to the respective moving object in real space. The task between is configured to create a transformation data set 420 based on the adaptation processes performed by the object between 418 with transformation data t1..t n for spatially transforming, to each other, first and second cluster of moving data points o1..o n , which are considered to be corresponding.

[0057] The alignment or registration method includes, in one embodiment, an iterative nearest point algorithm. The iterative nearest point algorithm is employed to minimize a positional difference between an identified cluster of motion data points 413a from the first point cloud 410 and an identified cluster of data points 413b from the second point cloud 410b. The positional difference is determined with reference to a static reference frame, such as the one determined by the static scene alignment module 414. In the iterative nearest point algorithm, one of the clusters of data points 413a, 413b, which is taken as the reference, is kept as fixed, while the other cluster of data points 413a, 413b, the source, is transformed to best match the reference.The algorithm revises the transformation iteratively (using a combination of translation and rotation) to minimize the distance from the source of the reference cluster of data points 413a, 413b. The algorithm outputs a fine-tuned transformation t1...t for each identified matching cluster of motion data points 413a, 413b. n out of.

[0058] In one implementation of the iterative nearest point algorithm, an iterative process is performed as follows. For each point in the source point cluster 413a, 413b, the nearest point in the reference point cluster 413a, 413b is found. The algorithm then estimates a combination of rotation and transformation using a cost function, such as a mean squared error, that will align each source point to its matching reference point in the best possible way. The source points are then transformed using the obtained transformation.

[0059] The matching method and the registration method, in various embodiments, comprise a mesh matching algorithm. An exemplary mesh matching algorithm converts the first and second clusters of motion data points 413a,b into search- and template-related polygonal and triangular mesh models that represent a topology of the cluster of motion data points 413a,b. Each mesh surface of the template is matched to a search surface, and intermediate transformation parameters are determined. In one embodiment, a perpendicular distance between the matching template and search surfaces is determined, following a normal vector of each surface. Collecting the transformation parameters, e.g., perpendicular distances, constitutes a transformation matrix t1.t n represents.

[0060] The conversion data t1.t nare determined for each pair of matching clusters of data points 413a, 413b, which combine to form the transformation data 420 output by the object derivation module 418.

[0061] The point cloud processing system 400 from Fig. 4 includes a distance module 422 configured to determine a distance traveled for each object from the transformation data 420. The transformation data 420 represents a positional relationship between each identified and matching cluster of motion data points 413a, 413b, from which a scalar or vector distance parameter d1..d n can be derived from the distance module 422. The distance parameter d1..d n represents a distance traveled for each matching cluster of motion data points 413a, 413b.

[0062] The point cloud processing system 400 includes a velocity calculation module 424 configured to calculate the velocity v 1..v n each identified moving cluster of data points based on the distance parameters d1..d n for the distance traveled. The speed calculation module can retrieve the frame rate data f (for example) from the data storage device 32 to determine the speed v1..v n to calculate. The frame rate data provides an elapsed time between the frames of three-dimensional imaging data 402 that corresponds to the time elapsed between the first or second point clouds 406a, 406b. The distance parameters d1..d n and the time dimension frame rate f are combined by the velocity calculation module to calculate the velocity v1..v n to determine the speed v1..v ncan be a vector or a scalar parameter. The velocity parameters can include rotational or transverse velocity components.

[0063] The speed parameters v1..v n for corresponding clusters of motion data points 413a, 413b output by the speed calculation module 424 are used by the autonomous drive system 70. The autonomous drive system 70 can calculate the speeds v1..v n for example, in the context of object tracking as part of the positioning system 76 and / or in the context of localization cross-checking as part of the positioning system 76. In any case, the velocities v1..v n used by the autonomous drive system 70 to control at least one autonomous vehicle control feature.

[0064] Example applications of the speed parameters by the autonomous propulsion system 70 include inferring the future motion of identified objects. Such inferring may involve using a Kalman filter based on a pre-determined motion model or a generative model that has been taught how similar-looking obstacles have moved in the past, e.g., pedestrians tend to ignore the traffic light at this intersection, etc. Based on the inferred future motion, the autonomous propulsion system 70 may generate one or more autonomous control commands considering the anticipated future motion of the identified object.

[0065] Another exemplary use of the velocity parameters can be to remove errors in other systems. For example, a carrier sensor may have noise or another error that causes the autonomous drive system 70 to detect an erroneous movement of the vehicle 10, e.g., more or less than in reality. By registering the current point cloud with at least one of the previous point clouds or a static map composed of previous point clouds, it is possible to counteract the aftereffects of such an error.

[0066] The present description proposes methods and systems that allow object velocities to be determined from the point cloud data 406 without necessarily requiring object classification and high-level processing by, for example, the positioning system 76, which generally includes determining and tracking bounding boxes over other high-level processing operations. The present description allows for the processing-efficient determination of the velocity of objects based on frames of the point cloud data 406.

[0067] With reference now to Fig. 5 and continued reference to the Fig. 1-4, a flowchart illustrates a control method 500 performed by the systems 100 of Fig. 1 according to the present description, which comprises the autonomous drive system 70 of Fig. 3 and the point cloud processing system 400 from Fig. 5. As can be seen from the description, the sequence of operations within the procedures is not as in Fig. 5 is not limited to sequential execution, but may be executed in one or more different orders as appropriate, as described herein. In various embodiments, method 400 may be scheduled for execution based on one or more predefined events and / or executed continuously during operation of autonomous vehicle 10.

[0068] The method 500 includes a step 502 in which a scene surrounding the vehicle 10 is captured and point cloud data 406 is generated. Specifically, the sensor system 28, and in particular a ranging sensor 42n thereof, captures three-dimensional images 402 at a frame rate of the sensor 42n. Based on the three-dimensional images 402, the first and at least second frames of the three-dimensional point cloud data 406a, 406b are generated by the data preservation module and the sensor fusion and preprocessing module 74.

[0069] The method 500 includes a step 504 of segmenting the point cloud data, in particular the first, most recent frame 406a thereof, into static data points 411a,b and motion data points 413a,b. In step 504, clusters of the motion data points 413a,b are defined with respect to the respective frames of the point cloud data 406a,b. Step 504 is performed by the static / moving segmentation module 408 and uses a segmentation algorithm, as described above, to classify the static and motion data points.

[0070] Method 500 includes a step 506 of aligning a static chord of a first point cloud 406a with a static scene of at least one second point cloud 406b. The second point cloud 406 may be a static map consisting of a plurality of pre-registered frames of the static point cloud data. In this way, any relative movement of clusters of the motion data points 413a,b in the same reference frame can be identified. The static scene aligning step 506 uses the static data points 411a,b of the first and second point clouds 406a,b for position-based alignment of the static scenes. In particular, the alignment step 506 is performed by the static scene alignment module using an alignment algorithm, as described above, to generate the position-based alignment of the first and second point clouds 416.

[0071] The method 500 includes a step 508 in which clusters of motion data points 413a,b are matched. Moving a cluster of motion data points 413a in the first point cloud 410a relative to a cluster of motion data points 413b in at least one second point cloud 410b is disclosed when the first and second point clouds 410a,b are positionally aligned in step 506. Clusters of motion data points 413a,b that correspond to the same object in real space are identified in the matching step 508. A registration or matching algorithm is employed in step 508 to derive a spatial transformation from a reference cluster of motion data points 413a to a target cluster of motion data points 413b. The registration or matching algorithm may be an iterative nearest point algorithm or a mesh matching algorithm in exemplary embodiments.The matching step 508 is performed by the object matching module 418 and generates the conversion data 420.

[0072] The method 500 includes a step 510 of determining the distance traveled1..d n each cluster identified as corresponding in step 508. In particular, the transformation data 420 represents a spatial relationship between clusters of motion data points 413a, 413b that have moved in the positionally aligned static scenes 416 that make up a static reference frame. Such a spatial relationship allows a distance parameter d1..d n in scalar or vector form. The step 510 of determining the distance traveled d1..d n is performed by the distance module 422.

[0073] Method 500 includes a step 512 of calculating the speed v1..v neach cluster of motion data points 413a, b, which corresponds to a velocity of an object in real space. Based on the distance parameter d1..d n and the frame rate f of the sensor device 42n, the speed v1..v n The speed calculation from step 512 is performed by the speed calculation module 424.

[0074] Method 500 includes a step 514 of using the calculated velocity v1..v n from step 512 as an input for the autonomous drive system 70 for generating at least one control function for the vehicle 10. In particular, the calculated speed v1..v n in controlling tracking and / or localization as described above.

[0075] In the preceding description, the first and second temporally spaced point clouds are described with reference to the segmentation of static / moving data, static alignment, and object matching processes. It is contemplated that these steps may be performed based on one or more current point clouds with more than one previous point cloud. For example, a static map of combined and aligned previous point clouds could be generated and used as a reference for comparison with one or more current point clouds.

Claims

[1] A method for performing a control function of an autonomous vehicle (10), the method comprising: Receiving a first point cloud and at least one second time-spaced point cloud based on three-dimensional position data (402) of a scene located outside the autonomous vehicle (10) from at least one sensor (42n) of the autonomous vehicle (10); Position aligning, via at least one processor (44), at least one static aspect of the scene in the first point cloud with at least one static aspect of the scene in the at least second cloud to obtain position-aligned first and second point clouds; Determining, via at least one processor (44), a speed of at least one moving object in the position-aligned first and second point clouds; Performing, via at least one processor (44), a control function of the autonomous vehicle (10) using the speed of the at least one moving object; and Segmenting the first and second point clouds into moving (413a,b) or static data points (411a,b), wherein the position alignment step uses the static data points as the at least one static aspect. [2] The method of claim 1, wherein the determining step comprises determining a distance traveled for the at least one moving object between the position-aligned first and second point clouds (410a,b), and wherein the speed of the at least one moving object is determined based on the distance traveled. [3] The method of claim 1, wherein the determining step comprises using an elapsed time between the first and second time-spaced point clouds (413a,b) to determine the velocity. [4] The method of claim 1, comprising identifying at least one cluster of data points in the temporally spaced first and second point clouds (413a,b) corresponding to at least one of the moving objects. [5] The method of claim 4, comprising matching at least one cluster of data points in the point cloud with at least one cluster of data points in the at least one second point cloud (413a,b), wherein the determining step comprises determining the speed of the at least one moving object based on a distance traveled between matched clusters of data points of the position-aligned first and second point clouds (413a,b). [6] The method of claim 5, wherein the matching comprises an iterative nearest point or a mesh matching algorithm. [7] The method of claim 1, wherein the determining step uses motion data points in the first and second point clouds (413a,b) whose position is aligned to determine the speed of the at least one moving object.

Images