Method and system for generating importance occupancy grid map

Abstract

The present invention discloses a system and method for generating an importance occupancy grid map (OGM) for a vehicle. The method comprises: receiving a three-dimensional (3D) point cloud; receiving a binary map, wherein the binary map is associated with a set of GPS coordinates of the vehicle; receiving information representing a planned path of the vehicle; and generating an importance OGM using a map generation module based on the 3D point cloud, the binary map and the planned path of the vehicle.

Description

Technical Field

[0001] The present invention relates to generating a map for a vehicle using a point cloud. In particular, the present invention relates to generating an importance occupancy grid map (OGM) using a point cloud generated from one or more sensors on a vehicle. Background Art

[0002] An autonomous vehicle can use different sensors (such as Light Detection and Ranging (LiDAR) sensors and camera sensors) to sense its surrounding environment, and can use a perception module to process the raw sensor data received from the different sensors to perform object detection, classification, regression, and segmentation to detect and identify objects of interest (such as pedestrians or other cars).

[0003] The information collected from the sensors is provided as sensor data by various sensors to the planning system of the autonomous vehicle, and the planning system uses the sensor data for path planning and navigation of the autonomous vehicle. In order to efficiently plan the path and safely navigate the autonomous vehicle in any environment, it is very important to know the location information of any object (static and dynamic) in the environment. Occupancy grid maps (OGMs) generated from various sensor data are usually used to represent the environment around the autonomous vehicle.

[0004] A single OGM can represent the environment at a specific point in time. The environment in which an autonomous vehicle travels is usually dynamic and unstructured. One way to cope with such an environment is to update the current OGM every time sensor data is received from the vehicle's sensors, and then update the planned path accordingly.

[0005] Three-dimensional (3D) point clouds collected from one or more sensors on a vehicle (e.g., from 3D range sensors such as LiDAR units) are typically processed to generate one or more OGMs when identifying various dynamic targets on the road. However, continuously processing 3D point cloud data to accurately detect, track, and segment dynamic targets requires a lot of computing resources; and existing methods also tend to process the entire point cloud domain of the target of interest, resulting in relatively slow target detection.

[0006] An improved technical solution is needed that can efficiently process raw point cloud data to detect dynamic objects on the road, thereby navigating an autonomous vehicle in a dynamic environment using less computing resources than traditionally required. Summary of the invention

[0007] According to one aspect of the present invention, a method for generating an importance occupancy grid map (OGM) for a vehicle is provided. The method comprises: receiving a 3D point cloud; receiving a binary map from a Global Positioning System (GPS) unit of the vehicle, wherein the binary map is associated with a GPS location; receiving path command information representing a planned path of the vehicle from a path planning module of the vehicle; and a map generation module generating an importance OGM based on the 3D point cloud, the binary map, and the path command information representing the planned path of the vehicle. Such an importance OGM can be used to filter target detection regions and reduce the computational effort of target detection, segmentation, and tracking components. By considering the planned path of the vehicle, the method can extract the regions most relevant to the current route of the vehicle from the 3D point cloud and the binary map. Thus, the above method is used to limit most of the data processing to regions that may conflict with the planned trajectory of the vehicle, and only focus on target detection in these regions, thereby improving computational efficiency at runtime.

[0008] The simplified binary map can be used to enhance the object detection, segmentation or tracking machine learning process, thereby saving computing resources when generating the importance OGM for runtime object recognition in real time or near real time. Compared with high-definition (HD) maps, binary maps can include less information and only include key road data such as lane geometry and lane boundaries. Binary maps can also include lane groups.

[0009] According to the above aspect, the importance OGM includes one or more regions of interest based on the planned path of the vehicle, and the one or more regions of interest are respectively represented by target masks, and the target masks indicate that the vehicle may collide on the planned path of the vehicle.

[0010] According to any of the above aspects, the method further includes: providing the importance OGM to a path planning module of the vehicle to generate a new planned path for the vehicle according to the one or more regions of interest in the importance OGM.

[0011] According to any of the above aspects, the importance OGM includes a road mask determined according to the planned path of the vehicle, and the width of the road mask is related to the angular velocity or the vehicle speed of the vehicle.

[0012] According to any of the above aspects, the method also includes: generating an input occupancy grid map (OGM) based on the received 3D point cloud; generating a binary OGM by combining the binary map and the input OGM; and generating a set of map feature vectors representing one or more regions of interest using a map feature extractor neural network based on the binary OGM.

[0013] According to any of the above aspects, the method also includes: using a path feature extractor neural network to generate a set of path feature vectors representing the vehicle trajectory, wherein the set of map feature vectors and the set of path feature vectors have the same dimension; merging the set of map feature vectors and the set of path feature vectors to obtain a set of merged feature vectors.

[0014] According to any of the above aspects, the method includes: generating a set of prediction features using a recursive neural network based on the merged feature vector, wherein the prediction features represent the one or more regions of interest and the vehicle trajectory.

[0015] According to any of the above aspects, the method comprises: generating the importance OGM from the prediction features using a decoder neural network

[0016] According to any of the above aspects, the planned path information representing the planned path includes the angular velocity and the vehicle speed of the vehicle.

[0017] According to at least some of the above aspects, the input OGM is generated based on at least one of the following contents: height values ​​of a grid or polygon surrounding a group of points in the 3D point cloud, average height values ​​of one or more columns of points in the 3D point cloud, intensity values ​​of one or more points in the 3D point cloud, value ranges of one or more height values ​​based on one or more points in the 3D point cloud, and point density values ​​of one or more points in the 3D point cloud.

[0018] According to another aspect, a method for training a map generation module to generate an importance OGM based on a 3D point cloud, a binary map, and path command information representing a planned path of a vehicle is provided. The map generation module includes a plurality of neural networks that are trained together using a supervised learning algorithm to learn parameters (e.g., weights) of each of the plurality of neural networks. An exemplary neural network is a recursive neural network, such as a long short-term memory (LSTM) network. During the training of the map generation module, the plurality of neural networks generate importance road masks for one or more objects in the 3D point cloud based on a ground truth OGM that has been previously generated based on historical data including previously obtained point clouds, binary maps, and path command information. Importance road masks are generated for one or more objects in the 3D point cloud based in part on whether the objects are sufficiently close to the vehicle in the planned path of the vehicle. Using this method, the 3D point cloud can be intelligently and efficiently cropped to retain the most important objects for further processing in a shorter time using fewer computing resources.

[0019] According to another aspect, a processing system for generating an importance occupancy grid map (OGM) for a vehicle is provided. The processing system includes: a processing unit; a memory coupled to the processing unit, wherein the memory stores machine executable instructions, and when the machine executable instructions are executed by the processing unit, the processing system performs the following operations: receiving a 3D point cloud; receiving a binary map from a GPS unit of the vehicle, wherein the binary map is associated with a set of GPS coordinates; receiving information representing a planned path of the vehicle from a path planning module of the vehicle; and a map generation module generates an importance OGM based on the 3D point cloud, the binary map, and the information representing the planned path of the vehicle.

[0020] According to another aspect, a processor-implemented method is provided for training a map generation module to generate an importance occupancy grid map (OGM) based on a point cloud, a binary map, and path command information. The processor-implemented method includes: receiving a set of labeled training data within n consecutive times {t–n, t–n+1, t–n+2, ..., t2, t1, t0}, wherein the set of labeled training data includes labeled training data corresponding to each time ti in the n consecutive times, each labeled training data including a 3D point cloud, a binary map, path command information representing a planned path of a vehicle, and a ground truth importance OGM. The processor-implemented method also includes: for each time ti in the n consecutive times, where i = {–n, –n+1, ..., 2, 1, 0}, providing the labeled training data corresponding to time ti to the map generation module to train the map generation module in the following manner: generating an input occupancy grid map (OGM) based on the 3D point cloud included in the labeled training data corresponding to time ti map, OGM); generate a binary OGM by combining the binary map included in the labeled training data corresponding to time ti and the input OGM; generate a set of map feature vectors representing one or more regions of interest associated with time ti using a map feature extractor neural network; generate a set of path feature vectors associated with time ti using the path feature extractor neural network according to the path command information representing the planned path of the vehicle included in the labeled training data corresponding to time ti, wherein the set of path feature vectors represents the vehicle trajectory, and the set of map feature vectors and the set of path feature vectors have the same dimension; merge the set of map feature vectors associated with time ti and the set of path feature vectors associated with time ti to obtain a set of merged feature vectors associated with time ti quantity; based on the merged feature vector associated with time ti, using a recursive neural network to generate a set of prediction features associated with time ti, wherein the prediction features represent the one or more regions of interest and the vehicle trajectory; using a decoder neural network to generate an importance OGM associated with time ti from the prediction features associated with time ti; calculating an error signal based on the ground truth OGM included in the labeled training data corresponding to time ti and the importance OGM associated with ti; back-propagating the error signal to the map feature extractor neural network, the path feature extractor neural network, the recursive neural network and the decoder neural network to adjust the parameters of the map feature extractor neural network, the path feature extractor neural network, the recursive neural network and the decoder neural network.

[0021] According to the above aspect, the path command information representing the planned path includes the angular velocity and the vehicle speed of the vehicle.

[0022] According to any of the above aspects, the input OGM is generated also based on the path command information included in the labeled training data corresponding to time ti, and the path command information includes at least one of the following information: information representing the height value of a grid or polygon surrounding a group of points in the 3D point cloud associated with time ti, information representing the average height value of one or more columns of points in the 3D point cloud associated with time ti, information representing the intensity value of one or more points in the 3D point cloud associated with time ti, information based on the value range of one or more height values ​​of one or more points in the 3D point cloud associated with time ti, and information representing the point density value of one or more points in the 3D point cloud associated with time ti.

[0023] According to another aspect, a processor-implemented method is provided for generating a ground truth importance occupancy grid map (OGM). The processor-implemented method includes: receiving a plurality of 3D point clouds, wherein the plurality of 3D point clouds are respectively associated with corresponding times in a set of continuous times, and the plurality of 3D point clouds each include one or more targets; processing the plurality of 3D point clouds to obtain a unique target ID and target type for each of the one or more targets; receiving a plurality of GPS locations of a vehicle, wherein the plurality of GPS coordinates are respectively generated at corresponding times in the set of continuous times; receiving a plurality of binary maps, wherein the plurality of binary maps are respectively associated with corresponding times in the set of continuous times and are based on the GPS coordinates of the vehicle at the corresponding times; for each corresponding time in the set of continuous times, generating a target map based on the 3D point cloud associated with the corresponding time, wherein the target map includes a representation of each of the one or more targets; for each of the one or more targets, generating a track map associated with the unique target ID for the target at all times in the set of continuous times based on the target map associated with each time in the set of continuous times, wherein the track map includes each instance of the target at all times in the set of continuous times, Each instance of the target is generated according to the GPS position of the target at the corresponding time in the set of continuous times; according to the trajectory map of the target, trajectory paths are determined for the one or more targets respectively; according to the trajectory paths of the one or more targets, a related target list is generated for the vehicle; for each corresponding time in the set of continuous times, a trajectory map including the trajectory path of the vehicle is generated for the vehicle, wherein the trajectory path of the vehicle is determined according to the binary map associated with the corresponding time and the GPS position of the vehicle associated with the corresponding time, and the trajectory path of the vehicle is represented by a road mask; for each corresponding time in the set of continuous times, a ground truth importance OGM is generated according to the trajectory path of the vehicle associated with the corresponding time and the trajectory path of each target in the related target list, wherein the ground truth importance OGM includes: a road mask representing the trajectory path of the vehicle associated with the corresponding time, a target mask of each target in the related target list, and a distance between the vehicle in the trajectory path of the vehicle and each target mask of each target in the related target list.

[0024] According to the above aspect, the related target list includes at least one target among the one or more targets, and the at least one target has a minimum Euclidean distance with the vehicle within the set of continuous times, and the minimum Euclidean distance is greater than a predetermined threshold.

[0025] According to any of the above aspects, the road mask for representing the trajectory path of the vehicle has a width determined according to one or more of the following: the road type of the trajectory path, the speed of the vehicle, and the type of the vehicle.

[0026] According to any of the above aspects, the target mask of each target in the list of related targets has a shape, size, radius, width or length determined according to the kinematic information of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Reference will now be made, by way of example, to the accompanying drawings which show exemplary embodiments of the present application, in which:

[0028] Figure 1 is a block diagram of some components in an exemplary autonomous vehicle;

[0029] Figure 2 is a block diagram of an exemplary map generation module within a perception module on a vehicle;

[0030] Figure 3 It is trained using a supervised learning algorithm Figure 2 The map generation module is used to learn Figure 2 A block diagram of the weights of multiple neural networks in the map generation module;

[0031] Figure 4 is a block diagram of the operations performed by a trained map generation module during inference to generate an occupancy grid map (OGM);

[0032] Figure 5 shows a series of exemplary target maps generated from a series of previously observed 3D point clouds;

[0033] Figure 6 Shown according to Figure 5 A plurality of exemplary trajectory maps generated from a series of target maps;

[0034] Figure 7 is a schematic diagram of an exemplary ground truth importance OGM generated using a series of target maps and trajectory maps;

[0035] Figure 8 is a flowchart of an exemplary method performed by a map generation module in reasoning for generating an importance occupancy grid map (OGM);

[0036] Fig. 9A and Fig. 9B It is trained using a supervised learning algorithm Figure 2 The map generation module is used to learn Figure 2 A flowchart of an exemplary method for mapping weights of multiple neural networks in a module.

[0037] Fig. 10A An exemplary 3D point cloud that is divided into small pieces to generate one or more 2D input OGMs is shown;

[0038] Fig. 10B An exemplary binary mask and an exemplary set of multi-layer input OGMs are shown.

[0039] Like reference numerals are used in different drawings to denote like components. DETAILED DESCRIPTION

[0040] The present invention is with reference to the accompanying drawings, in which embodiments are shown. However, many different embodiments may be used, and therefore description should not be interpreted as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention thorough and complete. Similar numbers in the present invention refer to similar elements, and in alternative embodiments, prime symbols are used to represent similar elements, operations or steps. The separate frame or separation of the functional elements of the illustrated system and device does not necessarily require the physical separation of these functions, because the communication between these elements can occur by means of message passing, function call, shared memory space, etc. without any such physical separation. Therefore, although functions are shown separately herein for ease of explanation, these functions do not need to be implemented in physically or logically separated platforms. Different devices may have different designs, so that although some devices implement some functions in fixed-function hardware, other devices may implement these functions in a programmable processor, which has a code obtained from a machine-readable medium.

[0041] For convenience, the present invention describes exemplary embodiments of methods and systems for positioning an autonomous vehicle. An autonomous vehicle may be any type of vehicle, such as a motor vehicle (e.g., a car, truck, bus), a boat or ship, a submarine, an airplane, storage equipment, construction equipment, a tractor, or other farm equipment. The teachings of the present invention are not limited to vehicles or any particular type of vehicle, but may be applied to other real or virtual targets as well as non-passenger vehicles and passenger vehicles. The teachings of the present invention may also be implemented in non-vehicle mobile robots, including but not limited to intelligent vacuum cleaners, rovers, lawn mowers, unmanned aerial vehicles (UAVs), and other targets. Even though the vehicle control system described herein has facilitated semi-automatic or fully autonomous driving, it may also be used for vehicles in non-autonomous driving mode.

[0042] Figure 1Selected components of an autonomous vehicle 100 provided by an exemplary embodiment of the present invention are shown. The vehicle 100 may include a vehicle control system 115 connected to a sensor or sensor system 110, a drive control system 150, and a mechanical system 190. The vehicle 100 also includes various structural elements, such as a frame, doors, panels, seats, windows, mirrors, etc., which are known in the art but have been omitted from the present invention to avoid obscuring the teachings of the present invention. The sensor 110 is mounted to a structural element in the vehicle 100 at different locations in the vehicle 100.

[0043] The vehicle control system 115 includes a processor 102 coupled to a plurality of internal components in the vehicle 100 via a communication bus (not shown). The processor 102 is coupled to a random access memory (RAM) 122, a read only memory (ROM) 124, a persistent (non-volatile) memory 126 such as an erasable programmable read only memory (EPROM) (flash memory), one or more wireless transceivers 130 for exchanging radio frequency signals with a wireless network, a satellite receiver 132 for receiving satellite signals from a satellite network, and a real-time clock 134. The vehicle control system 115 is also coupled to other components in the vehicle 100, including sensors 110, a touch screen 136, one or more speakers 138, one or more microphones 140, a drive control system 150, and a mechanical system 190.

[0044] The one or more wireless transceivers 130 may include one or more cellular (RF) transceivers for communicating with a plurality of different wireless access networks (e.g., cellular networks) using different wireless data communication protocols and standards. The vehicle control system 115 may communicate with any of a plurality of fixed base transceiver stations of a wireless WAN (e.g., a cellular network) within its geographic coverage area. The one or more wireless transceivers 130 may send and receive signals over the wireless WAN. The one or more wireless transceivers 130 may include a multi-band cellular transceiver that supports a plurality of radio frequency bands.

[0045] The one or more wireless transceivers 130 may also include a wireless local area network (WLAN) transceiver for communicating with a WLAN (not shown) via a WLAN access point (AP). The WLAN may include a wireless local area network (WLAN) that complies with the IEEE 802.11x standard (sometimes referred to as a wireless local area network). ) or other communication protocols.

[0046] The one or more wireless transceivers 130 may also include a short-range wireless transceiver, such as The wireless transceiver 130 may also include other short-range wireless transceivers, including but not limited to near field communication (NFC), IEEE802.15.3a (also known as ultra-wideband (UWB)), Z-Wave, ZigBee, ANT / ANT+, or infrared (e.g., Infrared Data Association (IrDA) communication).

[0047] Real time clock 134 may include a crystal oscillator that provides accurate real time information, such as that provided by Atmel Corporation.

[0048] The touch screen 136 includes a display such as a color liquid crystal display (LCD), a light-emitting diode (LED) display, or an active-matrix organic light-emitting diode (AMOLED) display, with a touch-sensitive input surface or cover connected to the electronic controller. Other input devices (not shown) coupled to the processor 102 may also be provided in the vehicle 100, including buttons, switches, and dials.

[0049] The vehicle control system 115 also includes one or more speakers 138, one or more microphones 140, and one or more data ports 142, such as serial data ports (e.g., Universal Serial Bus (USB) data ports). The system may also include other sensors, such as tire pressure sensors (TPS), door contact switches, light sensors, proximity sensors, etc.

[0050] The drive control system 150 is used to control the movement of the vehicle 100. The drive control system 150 includes a steering unit 152, a brake unit 154, and a throttle (or acceleration) unit 156, which can all be implemented as software modules or control blocks within the drive control system 150. When in fully automatic or semi-automatic driving mode, the steering unit 152, the brake unit 154, and the throttle unit 156 process the path information received from the path planning system 174 stored in the memory 126 of the vehicle control system 115, and generate control signals to control the steering, braking, and throttle of the vehicle 100 to travel on the planned path. The drive control system 150 may include other components for controlling other aspects of the vehicle 100 (including controlling turn signals and brake lights, etc.).

[0051] The mechanical system 190 receives control signals from the drive control system 150 to operate mechanical components in the vehicle 100. The mechanical system 190 affects the physical operation of the vehicle 100. The mechanical system 190 includes an engine 192, a transmission 194, and wheels 196. The engine 192 can be a gasoline-powered engine, a battery-powered engine, a hybrid engine, an electric engine, etc. Other components can be included in the mechanical system 190, including, for example, turn signals, brake lights, fans, and windows.

[0052] A graphical user interface (GUI) may be rendered by the processor 102 and displayed on the touch screen 136. The user may interact with the GUI using the touch screen and optionally other input devices (e.g., buttons, dials) to display relevant information, such as navigation information, driving information, parking information, media player information, climate control information, etc. The GUI may include a series of traversable menus for specific content.

[0053] The memory 126 in the vehicle control system 115 stores operating system software 160 executed by the processor 102. In addition to the GUI, the memory 126 also stores a plurality of software modules, collectively referred to as an autonomous driving system (ADS) 162, wherein each module in the autonomous driving system 162 is software including machine-readable instructions executable by the processor 102. The modules in the autonomous driving system 162 include a vehicle positioning module 164, a parking assistance module 166, an autonomous parking module 168, an assisted driving module 170 for semi-autonomous driving, a path planning module 174, a perception module 176, and other modules 178. Other modules 178, including mapping modules, navigation modules, climate control modules, media player modules, phone modules, and messaging modules, are also stored in the memory 126. In some embodiments, the perception module 176 (also referred to as a perception module) is used to perform the operations of the methods described herein when executed by the processor 102.

[0054] Although the parking assist module 166 , the autonomous parking module 168 , the assisted driving module 170 for semi-autonomous driving, the autonomous driving module 172 , the path planning module 174 or the perception module 176 are shown as separate modules, in other embodiments, they may be used in combination with one or more other software modules.

[0055] The memory 126 also stores various data 180. The data 180 may include sensor data 182 sensed by one or more sensors 110, user data 184 including user preferences, settings, and optional personal media files (e.g., music, videos, routes, etc.), and a download cache 186 including data downloaded via the wireless transceiver 130. For example, the memory 126 may store image data received from the camera 112, LiDAR data (e.g., three-dimensional data representing a three-dimensional point cloud d) received from the LiDAR scanning system 114, SAR data received from the SAR unit 116, odometry data from the wheel odometry unit 117 and / or the inertial measurement unit (IMU) 118, position data from the global positioning system (GPS) 119, and data from other sensors 120. The odometry data received from the wheel odometry unit 117 includes rotation data representing the rotation of the vehicle 100 and translation data representing the translation of the vehicle 100. The odometry data received from the IMU 118 includes the three-axis angular velocity of the vehicle 100 and the three-axis acceleration of the vehicle 100 .

[0056] In some embodiments, the processor 102 receives sensor data from the LiDAR unit 114 mounted on the vehicle 100 and generates a 3D point cloud based on the sensor data received from the LiDAR unit 114. In some embodiments, the perception module 176 can be implemented as a software system as part of a software stack (“ADS software stack”) of the autonomous driving system 160. The perception module 176 can receive 2D images and 3D point clouds from the processor 102 and output targets detected in the 2D images or 3D point clouds to other software systems of the ADS software stack, such as the path planning module 174.

[0057] The LiDAR unit 114 can capture information about the vehicle 100 in a wide view (e.g., a 360° view). The LiDAR unit 114 captures three-dimensional (3D) information about the environment and generates a set of data points in a 3D coordinate system. The 3D data points are collectively referred to as a 3D point cloud.

[0058] It should be understood that the 3D point cloud may be generated based on sensing data from any 3D range sensor (eg, LiDAR unit 114 ), radar unit, or stereo camera.

[0059] Using the various sensing units 112, 114, 116, 117, 118, 119, the sensor system 110 can collect information about the local external environment of the vehicle 100 (e.g., any surrounding obstacles) as well as information about a larger range of neighboring areas (e.g., the LiDAR unit 114 can collect information from an area with a radius of 100 meters or more around the vehicle 100). The sensor system 110 can also collect information about the position and orientation of the vehicle 100 relative to a reference coordinate system (e.g., using the GPS unit 119). The sensor system 110 can also collect information about the vehicle 100 itself. In this case, the vehicle 100 itself can be considered to be part of the sensed environment. For example, the sensor system 110 can collect information from sensing units (e.g., accelerometers, speedometers, odometers, and / or inertial measurement units) to determine the state of the vehicle 100, such as the linear velocity, angular velocity, acceleration, and tire grip of the vehicle 100. These sensing units may or may not be part of the sensor system 110. The sensor system 110 may repeatedly (eg, periodically) receive information from its sensing units in real time. The sensor system 110 may also provide sensor data to other components in the vehicle 100 in real time or near real time.

[0060] The sensor system 110 communicates with the perception module 176 via the processor 102 to provide the sensor data, including the 3D point cloud, to the perception module 176. The perception module 176 has been implemented to detect and identify targets in the external environment, such as for detecting and identifying pedestrians or another car. The perception module 176 can use any suitable module (e.g., using a machine learning process) to perform target detection, classification, regression, and segmentation on the 3D point cloud to detect and identify targets. The perception module 176 in this example includes a module that generates an importance OGM using one or more neural networks. The one or more neural networks are trained using weakly supervised machine learning techniques, as described in detail below.

[0061] The modules in the perception module 176 can be implemented using software. These software can include any number of independent or interconnected modules. For example, the perception module 176 can be implemented using a dedicated image processor, or can be implemented using one or more general-purpose processors of a vehicle controller (not shown) in the vehicle 100. The perception module 176 can repeatedly (e.g., periodically) receive sensor data from the sensor system 110 and perform target detection, classification, regression, and segmentation to detect and identify targets in real time or near real time. The output of the perception module 176 can include, for example, recognition results of detected targets in 2D and / or 3D coordinate systems, including target categories, target bounding boxes, target segmentation fragments (e.g., target masks), target positions, and target boundaries.

[0062] The sensor data from the sensor system 110 and the output from the perception module 176 can be provided to the path planning module 174. The path planning module 174 performs path planning for the vehicle 100. The output of the path planning module 174 is a path command signal S, which can be provided to the drive control system 150. The path command signal S can also be provided to other modules in the vehicle control system 115.

[0063] The download cache 186 may be deleted periodically (e.g., after a predetermined time). System software, software modules, specific device applications, or portions thereof may be temporarily loaded into volatile memory, such as RAM 122 for storing runtime data variables and other types of data or information. Data received by the vehicle control system 115 may also be stored in RAM 122. Although specific functions are described for various types of memory, this is only an example, and different functions may be assigned to different types of memory.

[0064] The vehicle control system 115 includes a satellite receiver 132, which can determine its position using signals received by the satellite receiver from multiple satellites in a satellite network. The satellite network typically includes multiple satellites that are part of at least one Global Navigation Satellite System (GNSS). At least one GNSS provides autonomous geospatial positioning within global coverage. For example, the satellite network can be a group of GNSS satellites. Exemplary GNSS include the U.S. NAVSTAR Global Positioning System (GPS) or the Russian Global Orbiting Navigation Satellite System (GLONASS). Other satellite navigation systems that have been deployed or are under development include the European Union's Galileo Positioning System, China's BeiDou Navigation Satellite System (BDS), the Indian Regional Satellite Navigation System, and the Japanese Satellite Navigation System.

[0065] The vehicle 100 may include other components not shown, including a user interface system and a wireless communication system (e.g., including an antenna). These other components may also provide input to the above systems and / or receive output from the above systems. The vehicle 100 may communicate with external systems (e.g., external map databases). The vehicle 100 may also communicate with a network such as a vehicle network that enables communication between autonomous, semi-autonomous, or non-autonomous vehicles.

[0066] Recently, computer vision methods in autonomous vehicles tend to use high-definition (HD) maps to augment point cloud data to enable object recognition and help trajectory planning, but processing these HD maps requires a lot of computation, especially when using such data-rich maps as input to one or more neural networks. In addition, existing methods for processing point clouds and HD maps using deep learning typically focus on outputting objects and their expected routes on the road. In addition, existing deep learning technical solutions for autonomous driving often do not take into account the planned trajectory of the vehicle, so unnecessary areas in the environment are often checked even if they are unlikely to affect the vehicle's trajectory. The technical solution proposed in the present invention trains and implements one or more neural networks to avoid unimportant areas and focus the algorithm on parts of the map that may interact with the planned trajectory of the vehicle at a certain point in time.

[0067] Finally, the highly detailed HD maps used by existing methods are not always available and are often obtained using a lot of time and resources. For example, HD maps include detailed information such as stop signs, traffic lights, exit lanes, turn lanes, etc., which can be fed into the machine learning algorithm as multi-layer data or multi-channel input. A lot of data chewing is involved, so a lot of calculations are required. The technical solution proposed in the present invention uses a simplified binary map instead of an HD map to enhance the machine learning process, thereby saving computing resources when generating an importance occupancy grid map (OGM) for dynamic target recognition at runtime in real time or near real time.

[0068] Figure 2 is a block diagram of an exemplary map generation module 350 within the perception module 176 on the vehicle 100 .

[0069] The LiDAR unit 114 performs a 3D scan of the environment in which the vehicle 100 is located by scanning multiple laser beams across the surrounding environment of the vehicle 100, and generates a 3D point cloud 251 based on the 3D scan. The 3D point cloud 251 can also be referred to as 3D data, which is a collection of multiple data points. Each data point in the 3D point cloud can be represented by a set of 3D coordinates of the sensing target in a 3D coordinate system (e.g., x, y, and z values ​​in meters). Typically, each data point in the 3D point cloud can be represented as a vector including x, y, and z coordinate values, and optionally represented as other values ​​(e.g., other information such as reflected light intensity or detection time).

[0070] The 3D point cloud corresponds to a specific time point and may include a timestamp throughout the present invention. That is, the 3D point cloud may be associated with a time frame. Multiple 3D point clouds may be associated with a corresponding one of a set of continuous time frames, respectively. The data points in the 3D point cloud may be irregularly spaced, depending on the external environment. In some embodiments, in addition to the 3D coordinates, each data point in the 3D point cloud may also include other information, such as reflected light intensity or detection time. In some embodiments, the 3D point cloud may be preprocessed to generate other point cloud information, which may include but is not limited to: point cloud information representing the height value of a grid or polygon surrounding a set of data points in the 3D point cloud, point cloud information representing the average height value of one or more columns of points in the 3D point cloud, point cloud information representing the intensity value of each of one or more data points in the 3D point cloud, point cloud information representing the range of one or more height values ​​based on one or more data points in the 3D point cloud, and point cloud information representing the point density value of one or more data points in the 3D point cloud. Each of the other point cloud information mentioned above may be part of the input OGM (e.g., a channel) used to generate the final importance OGM, as described in detail below.

[0071] The vehicle drive control system 115 acquires the binary map 252 in real time or near real time based on the GPS position of the vehicle 100. The vehicle drive control system 115 receives the GPS position of the vehicle from the GPS unit 119 of the vehicle 100, including a set of GPS coordinates (e.g., longitude and latitude coordinates). Compared with the HD map, the binary map 252 includes less information and only includes key road data, such as the geometry of the lane and the lane boundaries. The binary map 252 may also include a lane group. For example, a typical HD map usually includes high-density data, including: high-density map data, intersection information, road boundaries, signal lights and road signs (e.g., traffic lights, traffic signs), point clouds of static targets (e.g., trees, buildings), metadata (e.g., names, lane numbers, routes). The simplified binary map 252 may include just enough information to indicate whether each individual cell (which may be a pixel or a pixel group) belongs to a road. If the individual cell is part of the road, the cell may include a value of 1, otherwise it includes a value of 0. In some cases, the binary map may include other data, such as intersections or lane numbers.

[0072] As described above, the binary map 252 typically includes less information than the HD map and only includes key road data, such as the geometry of the lanes and lane boundaries. The binary map 252 may also include lane groups. In some exemplary embodiments, the binary map 252 may be simply presented as a grid map including a plurality of cells, wherein each cell has a corresponding binary value (0 or 1): 0 represents an area that does not belong to a lane, and 1 represents a portion of a lane. Each cell may include one or more pixels. In other embodiments, the binary map 252 may include other information such as road signs, intersections, and lane groups. In some embodiments, the vehicle control system 115 obtains the binary map 252 based on the GPS location of the vehicle 100 by sending a query including the GPS location of the vehicle 100 to a database to retrieve the binary map 252. The database is stored in the memory of the vehicle 100 or on a remote server in real time or near real time. The database including binary maps of different regions of the world receives a query including the GPS location of the vehicle 100, retrieves a binary map associated with the GPS location, and sends the retrieved binary map to the vehicle control system 115. In some embodiments, the binary map may be derived from a corresponding HD map based on GPS location using the rich spatial and structural information represented by the HD map. In some embodiments, the binary map may be derived from the corresponding HD map through a pre-processing step taking into account the planned path of the vehicle 100, so that only information related to the planned path of the vehicle 100 needs to be obtained from the HD map. In some embodiments, the binary map 252 may also be generated from an aerial map based on GPS location or downloaded from an online application programming interface (API) that can be accessed in real time, rather than being derived from the HD map.

[0073] The perception module 176 may receive information representing a planned path for the vehicle 100 from the path planning module 174 in the vehicle control system 115. The path command information 253 representing the planned path for the vehicle 100 includes one or more of the planned angular velocity ω and the planned vehicle speed v of the vehicle 100. The planned angular velocity ω of the vehicle 100 includes information representing a planned turn (if any), such as the direction and magnitude of the planned turn of the vehicle 100. The planned angular velocity ω of the vehicle 100 may be represented by a vector. In some embodiments, the path command information 253 includes a value equivalent to ω / v (angular velocity / vehicle speed) of the vehicle 100. In some embodiments, the path command information 253 may include other information, such as a rotation angle of the planned path for the vehicle 100. The path command information 253 is associated with a time frame when the perception module 176 generates or retrieves the path command information 253.

[0074] The map generation module 350 receives the 3D point cloud 251, the binary map 252 and the path command information 253, and generates an importance occupancy grid map (OGM) Mi 260, as described in detail below. The map generation module 350 includes a map feature extractor neural network (NN) 351, a path feature extractor NN 353, a recursive neural network 355 and a decoder network 357. The neural networks 351, 353, 355, 357 are pre-trained and optimized using a set of labeled training data, wherein each labeled training data corresponds to a time frame in a series of continuous time frames. The generated importance OGM Mi 260 includes one or more regions of interest based on the planned path of the vehicle 100, wherein the one or more regions of interest are respectively represented by a target mask, which indicates that the vehicle 100 may have a collision on the planned path of the vehicle 100. The generated importance OGM Mi 260 may then be sent to the path planning module 174 to generate a new planned path for the vehicle 100 , wherein the new planned path minimizes the risk of collision with the areas of interest identified in the importance OGM Mi 260 .

[0075] In some embodiments, the new planned path may be sent to drive control system 150 to control the operation of vehicle 100. Mechanical system 190 receives control signals from drive control system 150 to operate mechanical components in vehicle 100 to cause vehicle 100 to travel on the new planned path.

[0076] Figure 4 is a block diagram of the operations performed by the trained map generation module 350 in inference. The trained map generation module 350 is used to generate the importance OGM 260 based on the current 3D point cloud 351, the current binary map 252, and the current path command information 253 received from the path planning system 174, as described in detail below. For the purpose of the present invention, the current 3D point cloud is a 3D point cloud received from the LiDAR unit 114 when the LiDAR unit 114 performs a scan of the surrounding environment of the vehicle 100. The current binary map 252 is a binary map obtained using a query including the GPS position of the vehicle 100. The GPS position of the vehicle 100 is obtained when the 3D point cloud is received from the LiDAR unit 114. The current path command information 253 is the path command information obtained from the path planning module in the vehicle 100 based on the current GPS position of the vehicle 100 when the 3D point cloud is received from the LiDAR unit 114. As Figure 4As shown, the submodule 360 ​​receives the current 3D point cloud 251 and generates the input OGM ML 254 according to the current 3D point cloud 251. The submodule 360 ​​pre-processes the current 3D point cloud 251 to generate path planning information, including: information representing the height value of a grid or polygon surrounding a group of data points in the 3D point cloud, information representing the average height value of one or more columns of data points in the 3D point cloud, information representing the intensity value of each of one or more data points in the 3D point cloud, information representing the value range of one or more height values ​​based on one or more data points in the 3D point cloud, and information representing the point density value of one or more data points in the 3D point cloud. The above information included in the path planning information 253 is respectively stored in the corresponding channels of the input OGM ML 254.

[0077] Reference now Fig. 10A , showing that the submodule 360 ​​of the map generation module 350 divides the exemplary 3D point cloud 351 into blocks 1050a, 1050b. The 3D point cloud 351 is divided into a plurality of individual blocks 1050a, 1050b in order to generate Fig. 10B One or more individual input OGMs 1200a, 1200b, 1200c are shown. Each individual block 1050a, 1050b has a corresponding intensity value and a corresponding height value 1100. The individual blocks 1050a, 1050b can have a voxel shape. A group of individual blocks with the same height value 1100 can be combined into a layer of blocks 1400a, 1400b. Therefore, each layer of blocks 1400a, 1400b has a corresponding layer height value generated based on the height value of each individual block 1400a, 1400b in the layer (for example, the average value of the height values ​​100 of all blocks 1400a, 1400b in the layer). Each layer of blocks 1400a, 1400b has a width Wc 1020 and a length Lc 1030. A layer of blocks 1400 a and 1400 b may also be referred to as a layer of 3D point cloud 251 , and the width Wc 1020 or length Lc 1030 of each layer may be referred to as the width Wc 1020 or length Lc 1030 of the 3D point cloud 251 .

[0078] Fig. 10B A set of exemplary multi-layer OGMs including separate input OGMs 1200a, 1200b, 1200c are shown, and these separate input OGMs can be collectively constituted by Figure 41020 of the 3D point cloud 351, and the length Li 1230 is the same as the length Lc 1030 of the 3D point cloud 351. A single layer 1400a, 1400b of the 3D point cloud 351 can be used to generate a single 2D input OGM 1200a, 1200b, 1200c. Thus, a 3D point cloud 351 having multiple layers 1400a, 1400b can be used to generate multiple 2D input OGMs 1200a, 1200b, 1200c, each of which can be used to form a corresponding portion of the input OGM ML 254. Each individual cell in the 2D input OGM 1200a, 1200b, 1200c is generated based on the corresponding individual block in the corresponding layer 1400a, 1400b of the 3D point cloud 351, and the individual cell can have a cell width and a cell length, wherein the cell width is the same as the width of the corresponding block in the corresponding layer, and the cell length is the same as the length of the corresponding block in the corresponding layer.

[0079] Reference again Figure 4 , the second submodule 363 receives the input OGM ML 254 and the binary map 252, and combines the binary map 252 and the input OGM ML 254 to generate a binary OGM MB 257 provided to the map feature extractor NN 351. In some embodiments, the second submodule 363 generates the binary OGM MB 257 by merging the binary map 252 and the input OGM ML 254 as other layers or other channels. Fig. 10B As shown, the binary mask 252 has a width Wb 1520 and a length Lb 1530, wherein the width Wb 1520 is the same as the width Wi 1210 of the individual 2D input OGMs 1200a, 1200b, 1200c, and the length Lb 1530 is the same as the length Li 1230 of the individual 2D input OGMs 1200a, 1200b, 1200c.

[0080] The map feature extractor NN 351 includes learning parameters (e.g., weights). The parameters of the map feature extractor NN 351 are learned using a supervised machine learning algorithm during the training process of the map generation module 350, as described in detail below. The map feature extractor NN 351 including the learned parameters is hereinafter referred to as the trained map feature extractor NN 351. The trained map feature extractor NN 351 receives the binary OGM MB 257 as input, generates a set of map feature vectors 310 representing one or more regions of interest in the binary OGM MB 257, and outputs the set of map feature vectors 310. The one or more regions of interest may include areas where the vehicle 100 and other objects on the road may collide.

[0081] The path feature extractor NN 353 includes learning parameters (e.g., weights). The parameters of the path feature extractor NN 353 are learned using a supervised machine learning algorithm during the training process of the map generation module 350, as described in detail below. The path feature extractor NN 353 including the learning parameters is referred to as the trained path feature extractor NN 353 hereinafter. The trained path feature extractor NN 353 receives the path command information 253 including at least the angular velocity of the vehicle 100, generates a set of path feature vectors 312 representing the vehicle trajectory based on the path command information 253, and outputs the set of path feature vectors 312. In some embodiments, the path command information 253 includes a value equivalent to ω / v (angular velocity / vehicle speed) of the vehicle 100. The set of map feature vectors 310 and the set of path feature vectors 312 have the same dimensions. The angular velocity indicates the direction of the vehicle's expected path, and if a turning action is planned for the vehicle 100, the angular velocity indicates the magnitude or degree of the turning action.

[0082] Then, the merging submodule 365 merges a set of map feature vectors 310 and a set of path feature vectors 312 to obtain a set of merged feature vectors 313. This set of merged feature vectors 313 is sent as input to a recurrent neural network (RNN) 355. RRN 355 includes learning parameters (e.g., weights). The parameters of RNN 355 are learned using a supervised machine learning algorithm during the training process of the map generation module 350, as described in detail below. The RNN 355 including the learning parameters is referred to as the trained RNN 355 hereinafter. The trained RNN 355 receives the merged feature vector 313, predicts features 315 based on the merged feature vector 313, and outputs predicted features 315. The predicted features 315 are a set of vectors that include information representing one or more areas of interest from a set of map feature vectors 310 and vehicle trajectories from a set of path feature vectors 312. The trained RRN 355 may be a Long-Short Term Memory (LSTM) neural network or a Gated Recurrent Unit (GRU) neural network.

[0083] The decoder NN 357 also includes learning parameters. The parameters of the decoder NN 357 are learned using a supervised machine learning algorithm during the training process of the map generation module 350, as described in detail below. The decoder NN 357 including the learning parameters is referred to as the trained decoder network 357 below. The trained decoder NN 357 receives the predicted features 315, converts the predicted features 315 into importance OGM Mi 260, and outputs OGM Mi 260. OGM Mi 260 can be used to filter the target detection space and to reduce the amount of computation of the target detection, segmentation and tracking module (not shown) of the perception system 176 in the ADS 162.

[0084] Figure 7Importance in OGM Mi 260 (hereinafter referred to as "Mi 260") shows that the vehicle 100 is traveling along a trajectory path, wherein the trajectory path is represented by a road mask 710, a road boundary 705, and various dynamic targets, and the various dynamic targets include a first vehicle 730, a first pedestrian 740, a second pedestrian 750, a second vehicle 760, a third pedestrian 770, and a third vehicle 780. Mi 260 includes one or more regions of interest based on the planned path of the vehicle 100, wherein the one or more regions of interest are represented by corresponding target masks 720a, 720b, 720c, 720d, respectively. The target mask is used to indicate that the vehicle 100 may have a collision on the planned path of the vehicle 100. In some embodiments, the region of interest is located around a dynamic target that has a minimum Euclidean distance with the vehicle 100 at any point in time (i.e., in any time frame) along its driving path, for example, the first vehicle 730 has a minimum Euclidean distance 790a with the vehicle 100, the first pedestrian 740 has a minimum Euclidean distance 790b with the vehicle 100, the second pedestrian 750 has a minimum Euclidean distance 790c with the vehicle 100, and the second vehicle 760 has a minimum Euclidean distance 790d with the vehicle 100. The minimum Euclidean distance generally refers to the shortest straight-line distance between two points or two lines. The minimum Euclidean distance generally needs to be less than a certain distance threshold before the neural network considers the dynamic target to be sufficiently important (i.e., close enough to the driving path of the vehicle to affect its planned path). The value of the distance threshold itself has been learned and set by the neural network during its training process, and may depend on many factors, including: the angle and / or linear speed of the dynamic target, the angle and / or linear speed of the ego vehicle, the planned path of the ego vehicle, the type of dynamic target, the vehicle type of the ego vehicle, the neighborhood, and any other suitable factors. The distance threshold may also be manually set by an administrator of the ADS 162, where appropriate.

[0085] Each region of interest is represented by a target mask, such as a target mask 720a corresponding to the first vehicle 730, a target mask 720b corresponding to the first pedestrian 740, a target mask 720c corresponding to the second pedestrian 750, and a target mask 720d corresponding to the second vehicle 760. The target masks 720a, 720b, 720c, 720d may have a predetermined 2D shape, such as a voxel shape, an ellipse, a square, a circle, a rectangle, an octagon, a hexagon, or a triangle. In some embodiments, each target may have a shape determined according to the type of dynamic target. For example, a pedestrian target may have an elliptical target mask; a car target may have a rectangular target mask. In some embodiments, the size (e.g., width) of the road mask may be related to or determined by one or more kinematic information about the dynamic target and / or the vehicle 100 within the road mask. For example, the diameter or width of the target mask may be determined according to the angular velocity or vehicle speed of the vehicle 100 and / or the linear velocity of the dynamic target associated with the target mask.

[0086] In contrast, the third pedestrian 770 and the third vehicle 780 are far enough away from the vehicle 100 along the path of the vehicle 100 that the neural network does not consider them "important" or "relevant" and are therefore not associated with the region of interest or the target mask. In other words, the pedestrian 770 and the vehicle 780 have a minimum Euclidean distance greater than the above distance threshold from the vehicle 100 on the path represented by the road mask 710. In some embodiments, any dynamic targets that do not have a target mask can be excluded from the importance OGM Mi 260 because they are unlikely to affect the planned path of the vehicle 100.

[0087] Figure 3is a block diagram of training a map generation module 350 using a supervised learning algorithm to learn parameters (e.g., weights) of a map feature extractor NN 351, a path feature extractor NN 353, a RNN 355, and a decoder NN 357. The map generation module 350 is trained using a set of labeled training data 355 (hereinafter referred to as a labeled training dataset 355). The labeled training dataset 355 includes labeled training data 450 in n consecutive times, where t-n is the earliest time and t0 is the latest time. Each labeled training data 450 in the labeled training dataset 355 has a timestamp {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0} corresponding to a given time ti. The labeled training data 450 with a given timestamp ti includes a 3D point cloud 251, a binary map 252, a path command information 253, and a ground truth importance OGM 455 (also referred to as a ground truth OGM) corresponding to the given time ti. The 3D point cloud 251 is a point cloud 251 observed at a given time ti, the binary map 252 is a binary map obtained based on the GPS position of the vehicle 100 at the given time ti, and the path command information 253 includes information representing the planned path of the vehicle 100 at the given time ti. The ground truth OGM 455 at a given time ti is generated using the 3D point clouds, binary maps, and path command information at n previous times in n consecutive times {t–n, t–n+1, t–n+2, t2, t1, t0}, as described in detail below. After the ground truth OGM 455 at a given time ti is generated, the labeled training data 450 at the given time ti is generated by associating the generated ground truth OGM 455 at the given time ti with the point cloud 251, binary map 252, and path command information 253 at the given time ti. Then, the labeled training data 450 is added to the labeled training data set 355. After generating the ground truth OGM 455 for each time ti in n consecutive times {t–n, t–n+1, t–n+2, ..., t2, t1, t0}, the labeled training data set 355 of n consecutive times {t–n, t–n+1, t–n+2, ..., t2, t1, t0} is stored in a memory for training the map generation module 350, as described in detail below.

[0088] Reference again Figure 3, training the map generation module 350 using the labeled training data set 355 will now be described. For each time ti, where i = {-n, -n+1, ​​..., 2, 1, 0}, a first submodule 360 ​​is used to generate an input OGM ML 254 based on the 3D point cloud 251. The first submodule 360 ​​pre-processes the 3D point cloud 251 to determine: height values ​​of a grid or polygon surrounding a set of data points in the 3D point cloud 251, an average height value of one or more columns of data points in the 3D point cloud 251, an intensity value of one or more data points in the 3D point cloud 251, a range of values ​​based on one or more height values ​​of one or more data points in the 3D point cloud 251, or a point density value of one or more data points in the 3D point cloud 251. The above contents can be stored in corresponding channels of the input OGMML 254, respectively.

[0089] The second submodule 363 receives the binary map 252 and the OGM ML 254 provided by the first submodule 360, and combines the binary map 252 and the input OGM ML 254 to generate a binary OGM MB 257. The binary OGM MB 257 is provided to the map feature extractor NN 351. In some embodiments, the second submodule 363 generates the binary OGM MB 257 by appending the binary map 252 to the input OGM ML 254 as other channels.

[0090] The map feature extractor NN 351 receives the binary OGM MB 257 as input, generates a set of map feature vectors FVi (where i = {–n, –n+1, –n+2, ..., 2, 1, 0}) based on the binary OGM MB 257, and outputs the set of map feature vectors FVi, where each vector FVi 310 represents one or more regions of interest based on the 3D point cloud 251. Each vector FVi 310 is generated for each time ti in the continuous time {t–n, t–n+1, t–n+2, ..., t2, t1, t0}. The one or more regions of interest may include areas where the vehicle 100 and other objects on the road may collide.

[0091] The path feature extractor NN 353 receives the path command information 253 including at least the angular velocity of the vehicle, generates a set of path feature vectors FV'i (where i = {–n, –n+1, –n+2, ..., 2, 1, 0}) based on the path command information 253, and outputs this set of path feature vectors FV'i, where each vector FV'i 312 represents the vehicle trajectory. Each vector FV'i 312 is generated for each time ti in the continuous time {t–n, t–n+1, t–n+2, ..., t2, t1, t0}. This set of map feature vectors FVi 310 and this set of path feature vectors FV'i 312 have the same dimension. The angular velocity represents the direction of the expected path of the vehicle 100. If a turning action is planned for the vehicle, the angular velocity represents the amplitude or degree of the turning action. In some embodiments, the path command information 253 is represented by a value equivalent to ω / v (angular velocity / vehicle speed) of the vehicle 100.

[0092] The merging submodule 365 merges a set of map feature vectors FVi 310 and a set of path feature vectors FV'i 312 at time frame ti to obtain a set of merged feature vectors 313. Then, this set of merged feature vectors 313 is provided to a recurrent neural network (RNN) 355. RNN 355 receives the merged feature vector 313, predicts features 315 at time ti based on the merged feature vector 313, and outputs predicted features 315. The predicted features 315 are a set of vectors that include information representing one or more regions of interest in a set of map feature vectors FVi 310 and vehicle trajectories in a set of path feature vectors FV'i 312. As described above, RNN 355 can be a long short-term memory (LSTM) neural network or a gated recurrent unit (GRU) neural network.

[0093] The decoder neural network 357 receives the predicted features 315, converts the predicted features 315 into the importance OGM Mi 260 at the time frame ti, and outputs the importance OGM Mi 260. The loss calculation submodule 370 then compares the importance OGM Mi 260 with the ground truth OGM 455 in the labeled training data 450 with the timestamp ti. The loss calculation submodule 370 calculates the error 380 based on the importance OGM Mi 260 and the ground truth OGM Mi 455. The error is then back-propagated to each of the neural networks 351, 353, 355, 357 to update the weights of the network.

[0094] In some embodiments, the training map generation module 350 is stopped when the labeled training data 455 corresponding to each time {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0} is processed. In some embodiments, the training map generation module 350 is stopped when the error 380 reaches a value less than a threshold, which indicates that the generated importance OGM Mi 260 is close enough to the ground truth importance OGM 455 included in the labeled training data 455.

[0095] After the map generation module 350 is trained using the labeled training data set 455, the neural networks 351, 353, 355, 357 all have learned parameters (i.e., weights). The map generation module 350 in which the neural networks 351, 353, 355, 357 all have learned parameters (i.e., weights) is referred to herein as a trained map generation module 350. The trained map generation module 350 may also be referred to as a machine-learned map generation module 350. The training of the map generation module 350 is also referred to below in conjunction with Fig. 9A and 9B Description, can be performed off-site (i.e., not on the vehicle 100) by a computing system (not shown); after training, the map generation module 350 including the trained NNs 351, 353, 355, and 357 (i.e., NNs 351, 353, 355, 357 all including learning parameters) can be deployed to the vehicle 100 (i.e., stored in the memory 126 of the vehicle control system 115) and operated during the inference process, as described above in conjunction with Figure 4 The map generation module 350 may be repeatedly trained using multiple sets of labeled training data 355 (eg, labeled training data sets 355).

[0096] In some embodiments, the map generation module 350 may be pre-trained using a labeled training dataset including labeled training data 450 to learn parameters of the NNs 351, 353, 355, 357 before being trained using a labeled training dataset 355 including the generated ground truth OGM 455. The map generation module 350 may be trained using a labeled training dataset 355 generated from data obtained from an existing database that stores various information about the historical driving records of one or more vehicles. For example, a sequential labeled dataset of dynamic objects and associated 3D point clouds may be obtained from an existing database, such as the KITTI visual benchmark suite of object tracking boxes. TM dataset and processed to generate labeled training data 450 to be included in labeled training dataset 355 .

[0097] Reference now Figures 5 to 7 , a method of generating a ground truth OGM 455 and a labeled training dataset 455 is shown. Figures 5 to 7 The method shown in is performed offline by a processing system (not shown). Figure 5 , a series of exemplary target maps 500a, 500b, 500c, 500d based on 3D point clouds 251 captured in continuous time {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0} are shown. As described above, a set of labeled training data 355 is used to train the map generation module 350 of the perception system 176. The labeled training data 450 at any given time ti includes the ground truth OGM 455 generated according to the method described below at the given time ti. The labeled training data set 355 includes labeled training data 450 corresponding to the continuous time {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0}, which can be used to train the NNs 351, 353, 355, 357 in the map generation module 350 to optimize the parameters (e.g., weights) of the respective NNs 351, 353, 355, 357.

[0098] In order to generate the ground truth OGM 455 at a given time ti 457 based on the point cloud 251, the binary map 252 and the path command information 253 at a given time ti, it is necessary to first define the importance criteria. The importance criteria are used to determine whether a dynamic or static target should be included in the ground truth OGM 455. The ground truth OGM 455 is used as a label for the labeled training data 450 corresponding to the given time ti. For example, the importance criteria determine whether and when the target trajectory is important relative to the driving path of the vehicle. In some embodiments, the minimum Euclidean distance can be used to determine the potential collision between the vehicle and the target. The minimum Euclidean distance usually needs to be less than a certain distance threshold before determining that the target is important enough (i.e., close enough to the driving path of the vehicle to affect its planned path). The value of the distance threshold can depend on many factors, including: the angular velocity and / or linear velocity of the target, the angular velocity ω and / or linear velocity v of the vehicle, the planned path of the vehicle, the type of the target, the vehicle type of the vehicle, the neighboring area, and any other suitable factors. The distance threshold can be manually set for each target type, and optionally further adjusted for each vehicle type of the ego vehicle. For example, if the target type is a pedestrian with an average walking speed, the distance threshold can be set to 5 meters. For another example, if the target type is an SUV traveling at 50KM / h on a path parallel to the ego vehicle's travel path, the distance threshold can be set to 20 meters.

[0099] A processing system (not shown) generates a ground truth OGM 455 for time ti based on the 3D point cloud 251, the binary map 252, and the path command information 253 included in the labeled training data 450 corresponding to time ti. The 3D point cloud 251 may include one or more targets, each having a unique target ID and a target type. The 3D point cloud 251 is processed to determine the GPS position of each target at each time frame based on each target ID. The target can be static or dynamic. The binary map 252 can use binary values ​​1 or 0 to indicate whether a pixel or grid on the map 252 belongs to a road ("1") or does not belong to a road ("0"). The binary map 252 is associated with the GPS position of the ego vehicle recorded at a given time ti. The binary map 252 can be retrieved from an existing database storing binary maps using a query including the GPS position of the ego vehicle at a given time ti.

[0100] Next, the processing system (not shown) generates a target map 500a, 500b, 500c, 500d for each corresponding time ti in a set of n consecutive times {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0} according to the 3D point cloud 251 associated with the corresponding time ti. Figure 5 As shown, the target maps 500a, 500b, 500c, 500d at a given time ti include a representation of each target at ti based on the GPS position of the corresponding target at the given time ti. Figure 5 In order to facilitate explanation and distinguish different target types, circles 510a, 510b, 510c, 510d are used to represent pedestrians on each target map 500a, 500b, 500c, 500d; squares 520a, 520b, 520c, 520d are used to represent cars on each target map 500a, 500b, 500c, 500d; triangles 530a, 530b, 530c, 530d are used to represent SUVs on each target map 500a, 500b, 500c, 500d. Other shapes can be used to represent targets. At time t=-3, target map 500a shows pedestrians 510a, cars 520a, and SUVs 530a at different locations. At time t=-2, target map 500b shows pedestrian 510b in the same position as target map 500a, car 520b heading east (right), and SUV 530b heading north (up). At time t=-1, target map 500c shows pedestrian 510c still not moving, car 520c heading east again, and SUV 530c heading north again. At time t=0, target map 500d shows pedestrian 510d still not moving, car 520d heading east again, and SUV 530d heading north again.

[0101] Reference now Figure 6 , for each target, a processing system (not shown) can now generate a trajectory map 610, 620, 630 for each target 510, 520, 530 associated with a unique target ID at all times {t3, t2, t1, t0}. Each trajectory map 610, 620, 630 shows a trajectory path 640a, 640b, 640c based on a specific GPS position of each target map 500a, 500b, 500c, 500d of a single target 510, 520, 530 at continuous times {t3, t2, t1, t0}. Each trajectory map 610, 620, 630 includes an instance based on the GPS position of the corresponding target 510, 520, 530 at each time frame ti from t3 to t0. For example, the trajectory map 610 includes a pedestrian target 510 (target ID 001). The pedestrian target 510 is stationary in all time frames from t3 to t0. Therefore, the track path 640a may be a fixed point of the target 510 on the track map 610. The track map 620 includes four instances 520a, 520b, 520c, 520d of the sedan target 520 (target ID 003) on the track path 640b, and the sedan target 520 is traveling eastward from t3 to t0. The track map 630 includes four instances 530a, 530b, 530c, 530d of the SUV target 530 (target ID 005) on the track path 640c, and the SUV target 530 is traveling northward from t3 to t0.

[0102] A processing system (not shown) then determines the driving path of the ego vehicle based on a set of GPS positions recorded by the ego vehicle at each time {t3, t2, t1, t0}. The driving path can be further processed using one or more binary maps 252 of the same time. For example, at any given time ti, if and when a binary map 252 is available, the driving path of the ego vehicle can be augmented using the road boundary or lane group information in the corresponding binary map 252 at the given time ti. The driving path of the ego vehicle and the binary map 252 can be processed to generate the following: Figure 7 A road mask 710 is shown, which generally represents the driving path of the vehicle within the road boundary.

[0103] Based on the trajectory paths 640a, 640b, 640c in each trajectory map 610, 620, 630, a processing system (not shown) determines a list of important or relevant targets based on the driving path of the ego vehicle. The targets can be determined to be relevant based on a set of rules, which can be predetermined. For example, if at any point along the driving path 640b of the target 520 at time {t3, t2, t1, t0}, the minimum Euclidean distance 560b between the target 520 and the ego vehicle 100 is less than a certain threshold, then the target 520 can be said to be relevant. That is, the shortest Euclidean ("straight line") distance between any point on the driving path 560b of the target 520 and any point on the driving path of the ego vehicle 100 from t=-3 to t=0 is the minimum Euclidean distance. For each target 510, 520, 530, if the minimum Euclidean distance 560a, 560b, 560c to the ego vehicle 100 is less than a set threshold, the target 520, 530 is considered to be relevant for the purpose of calculating the training map 455. The threshold can be set based on many factors, such as: the target's dynamic information (such as angular velocity and linear velocity), the ego vehicle's dynamic information, the road type (such as a highway or an urban road), the road structure (such as whether there is an intersection), historical data on traffic accidents, the type of target (such as a pedestrian or a vehicle), the speed limit of the road, and the type of ego vehicle (such as a car or a truck). For example, if the target type is a pedestrian and the pedestrian target walks at a speed of 1 meter per second, the threshold can be set to 5 meters. For another example, if the target type is a car and is traveling at a speed of 30 km / hr, the threshold can be set to 20 meters.

[0104] Next, the processing system (not shown) generates a ground truth importance OGM 455 (also referred to as ground truth OGM 455) for each time ti in a set of n consecutive times {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0} based on the trajectory path of the ego vehicle associated with the corresponding time ti and the trajectory path 640b, 640c of each target 520, 530 in the associated target list. The ground truth OGM 455 is as follows. Figure 7As shown, the road mask 710 represents the trajectory path of the vehicle 100 associated with the corresponding time ti, the target mask 720a, 720b, 720c, 720d of each target 730, 740, 750, 760 in the related target list, and the distance 790a, 790b, 790c, 790d between the vehicle 100 and each target mask 720a, 720b, 720c, 720d. In some embodiments, the road mask 710 for representing the trajectory path of the vehicle has a width determined according to one or more of a plurality of factors, such as: the dynamic information of the vehicle 100 at the time frame ti (such as angular velocity or linear velocity), the road type (such as a highway or an urban road), the road structure (such as an intersection, a single lane or multiple lanes), the type of the vehicle 100 (such as a car or a truck), and the like.

[0105] In some embodiments, the target masks 720a, 720b, 720c, 720d corresponding to the related targets 730, 740, 750, 760 have a shape, size, radius, width or length determined according to one or more factors, the one or more factors including the target type of the target, the kinematic information of the target at time ti, the direction in which the target is moving, etc. For example, Figure 7 The target mask for all targets is shown to be elliptical. In other embodiments, pedestrian targets may be represented by circular road masks, while car targets may be represented by square road masks.

[0106] Once the ground truth OGM 455 for a given time ti is generated, the ground truth OGM can be stored in a database. Then, labeled training data 450 corresponding to each of n consecutive times {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0} can be generated, including the point cloud 251, binary map 252, path command information 253 and the generated ground truth OGM for each time ti. Then, the labeled training data 455 for n consecutive times is stored for training the map generation module 350 as described above.

[0107] Reference now Figure 8 , a flow chart of an exemplary method 800 for generating importance OGM Mi 260 executed in inference by the trained map generation module 350 is shown. The method 800 may be implemented in software executed by a processor 102 in a vehicle control system 115, etc. The coding of the software is well within the scope of ordinary skill in the art in view of the present invention. The method 800 may include more or fewer processes than those shown and described, and may be performed in a different order. The software includes computer readable code or instructions executable by the processor 102 in the vehicle control system for performing the method 800.

[0108] In 810 , the vehicle control system 115 receives the 3D point cloud 251 at the current time ti from the LiDAR unit 114 of the vehicle 100 (or a different sensor such as a radar unit or a stereo camera).

[0109] In 820 , the submodule 360 ​​of the map generation module 350 receives the binary map 252 for the current time ti , wherein the binary map 252 is associated with the GPS position received from the GPS unit 119 in the vehicle 100 .

[0110] In 830 , the path feature extractor NN 351 with learned parameters (eg, weights) receives path command information 253 representing the planned path of the vehicle 100 at the current time ti from the path planning module 174 in the vehicle 100 .

[0111] In 840, the map generation module 350 generates the importance OGM 260 of the current time ti according to the 3D point cloud 251, the binary map 252 and the path command information 253, as described above in conjunction with Figure 4 described.

[0112] In 850 , the importance OGM Mi 260 at the current time ti is provided to the path planning system 174 , and the path planning system 174 generates a new planned path for the vehicle 100 according to one or more regions of interest in the importance OGM Mi 260 .

[0113] refer to Fig. 9A and Fig. 9B , a flowchart showing an exemplary method of training the map generation module 350 using a supervised learning algorithm to learn parameters (eg, weights) of the plurality of neural networks 351 , 353 , 355 , 357 in the map generation module 350 .

[0114] In 910, the map generation module 350 receives multiple sets of input labeled training data 355 of n consecutive time frames, wherein each labeled training data 450 in a set of labeled training data 355 is associated with a corresponding time ti in n consecutive times {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0}. Each labeled training data 450 includes: a 3D point cloud 251, a binary map 252, path command information 253 representing a planned path of the vehicle, and an expected (i.e., ground truth) importance OGM 455. In 930, for each time ti in n consecutive times, where i = {-n, -n+1, ​​..., 2, 1, 0}, the map generation module 350 generates the importance OGM Mi260 associated with the time frame ti in an iterative manner as described below through steps 9310 to 9390.

[0115] In 9310, the submodule 360 ​​of the map generation module 350 receives the 3D point cloud 351 of the labeled training data 355 corresponding to time ti, and generates an input occupancy grid map (OGM) ML 254 based on the 3D point cloud 251 at time ti. In 9320, the second submodule 363 of the map generation module 350 generates a binary OGM MB 257 at time ti by combining the binary map 252 corresponding to time ti and the generated input OGM ML 254 at time ti. In 9330, the map feature extractor NN 351 receives the binary OGM MB 257 at time ti, generates a set of map feature vectors FVi, where i = {-n, -n+1, ​​-n+2, ..., 2, 1, 0}, and outputs the set of map feature vectors FVi. Each map feature vector FVi 310 represents one or more regions of interest based on the 3D point cloud 251.

[0116] In 9340, the path feature extractor NN 353 receives the path command information 253 of the labeled training data 355 corresponding to the time frame ti and generates a set of path feature vectors FV'i 312. The set of path feature vectors FV'i 312 represents the vehicle trajectory, and the set of map feature vectors FVi 310 and the set of path feature vectors FV'i 312 have the same dimension.

[0117] In 9350 , the merging submodule 365 of the map generation module 350 merges a set of map feature vectors FVi 310 and a set of path feature vectors FV′i 312 to obtain a set of merged feature vectors 313 .

[0118] In 9360, the RNN 351 of the map generation module 350 generates a set of predicted features 315 based on the merged feature vector 313, wherein the predicted features 315 represent one or more regions of interest and vehicle trajectories.

[0119] In 9370, the decoder NN 357 receives a set of predicted features 315, generates the importance OGM Mi 260 of time ti based on the predicted features 315, and outputs the importance OGM Mi 260 of time ti. In 9380, the loss calculation submodule 370 calculates the error 380 based on the ground truth OGM 455 in the labeled training data 450 corresponding to time ti and the importance OGM Mi 260 of time ti output by the decoder NN 357.

[0120] In 9390 , the map generation module 350 back-propagates the error 380 to the map feature extractor NN 351 , the path feature extractor NN 353 , the recursive NN 355 , and the decoder NN 357 .

[0121] In 9395, the map generation module 350 adjusts (i.e., updates or modifies) one or more parameters (e.g., weights) of each of the NNs 351, 353, 355, 357 according to the error 380 so as to optimize the NNs 351, 353, 355, 357 of the map generation module 350. Then, the map generation module 350 starts again at the next time frame ti+1 at step 9310, and continues to perform the same steps 9310 to 9395 at each time ti in the n consecutive times {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0}. In some embodiments, the training of the map generation module 350 is stopped when the labeled training data 450 of all times {t-n, t-n+1, ​​t-n+2, ..., t2, t1, t0} (i.e., all labeled training data 450 in the labeled training data set 355) is processed. In some embodiments, the training of the map generation module 350 is stopped when the error 380 has a value less than a threshold value, which indicates that the generated importance OGM Mi 260 is close enough to the ground truth OGM 455 .

[0122] Although the present invention describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be omitted or changed as appropriate. Where appropriate, one or more steps may be performed in an order other than the order described.

[0123] Although the present invention is at least partially described in terms of methods, it will be understood by those of ordinary skill in the art that the present invention is also directed to various components for performing at least some aspects and features of the methods, whether by hardware components, software, or any combination thereof. Accordingly, the technical solution of the present invention may be embodied in the form of a software product. Suitable software products may be stored in a pre-recorded storage device or other similar non-volatile or non-transient computer-readable medium, including a DVD, CD-ROM, USB flash drive, removable hard disk, or other storage medium, etc. The software product includes instructions tangibly stored thereon, which enable a processing device (e.g., a personal computer, a server, or a network device) to perform examples of the methods disclosed herein.

[0124] Without departing from the subject matter of the claims, the present invention may be implemented in other specific forms. The exemplary embodiments described are merely illustrative and non-restrictive in all respects. Features selected from one or more of the above-described embodiments may be combined to create optional embodiments that are not explicitly described, and features suitable for such combinations may be understood within the scope of the present invention.

[0125] All values ​​and subranges within the disclosed range are also disclosed. Moreover, although the system, equipment and process disclosed and shown herein may include a specific number of elements / assemblies, the system, equipment and assembly may also be modified to include more or less of these elements / assemblies. For example, although any disclosed element / assembly may be used as a singular reference, the embodiments disclosed herein may be modified to include a plurality of these elements / assemblies. The subject matter described herein is intended to encompass and include all appropriate changes in technology.

Claims

1. A method for generating an importance occupancy grid map for a vehicle, characterized in that: The method comprises: Receive 3D point cloud; receiving a binary map from a global positioning system unit of the vehicle, wherein the binary map is associated with a GPS location; receiving, from a path planning module of the vehicle, path command information representing a planned path for the vehicle; An importance occupancy grid map is generated based on the 3D point cloud, the binary map and the path command information representing the planned path of the vehicle; the importance occupancy grid map includes a road mask determined according to the planned path of the vehicle, and the width of the road mask is related to the angular velocity or vehicle speed of the vehicle.

2. The method according to claim 1, characterized in that The importance occupancy grid map includes one or more regions of interest based on a planned path of the vehicle, the one or more regions of interest are respectively represented by a target mask, and the target mask indicates that the vehicle may collide on the planned path of the vehicle.

3. The method according to claim 1 or 2, characterized in that: The method further includes providing the importance occupancy grid map to a path planning module of the vehicle to generate a new planned path for the vehicle according to one or more regions of interest in the importance occupancy grid map.

4. The method according to claim 1, characterized in that: The method further comprises: generating an input occupancy grid map based on the received 3D point cloud; generating a binary occupancy grid map by combining the binary map and the input occupancy grid map; A set of map feature vectors representing one or more regions of interest are generated using a map feature extractor neural network based on the binary occupancy grid map.

5. The method according to claim 4, characterized in that The method further comprises: generating a set of path feature vectors representing vehicle trajectories using a path feature extractor neural network, wherein the set of map feature vectors and the set of path feature vectors have the same dimension; The set of map feature vectors and the set of path feature vectors are merged to obtain a set of merged feature vectors.

6. The method according to claim 5, characterized in that The method further comprises: A set of prediction features is generated using a recurrent neural network based on the merged feature vector, wherein the prediction features represent the one or more regions of interest and the vehicle trajectory.

7. The method according to claim 6, characterized in that The method also includes generating the importance occupancy grid map from the predicted features using a decoder neural network.

8. The method according to claim 1, characterized in that The path command information representing the planned path includes the angular velocity and the vehicle speed of the vehicle.

9. The method according to claim 4, characterized in that The input occupancy grid map is generated according to at least one of the following contents: height values ​​of a grid or polygon surrounding a group of points in the 3D point cloud, average height values ​​of one or more columns of points in the 3D point cloud, intensity values ​​of one or more points in the 3D point cloud, value ranges based on one or more height values ​​of one or more points in the 3D point cloud, and point density values ​​of one or more points in the 3D point cloud.

10. A method for training a map generation module to generate an importance occupancy grid map based on a point cloud, a binary map and path command information, characterized in that: The method comprises: Receive n consecutive time {t –n ,t –n+1 ,t –n+2 ,...,t 2, t1, t0}, wherein the set of labeled training data includes each time t in the n consecutive times i Corresponding labeled training data, each labeled training data includes a 3D point cloud, a binary map, a path command information representing the planned path of the vehicle, and a ground truth importance OGM; For each time t in the n consecutive times i , where i = {–n,–n+1,...,2,1,0}, will be related to time t i The corresponding labeled training data is provided to the map generation module to train the map generation module in the following manner: According to the time t i The corresponding 3D point cloud included in the labeled training data generates an input occupancy grid map; By combining the i correspondingly labeling the binary map included in the training data and the input occupancy grid map to generate a binary occupancy grid map; Generates the map feature extractor neural network with time t i an associated set of map feature vectors representing one or more regions of interest; According to the time t i The path command information representing the planned path of the vehicle included in the corresponding labeled training data is generated using the path feature extractor neural network with respect to time t i an associated set of path feature vectors, wherein the set of path feature vectors represent vehicle trajectories, and the set of map feature vectors and the set of path feature vectors have the same dimension; The time t i A set of map feature vectors associated with time t i A set of path feature vectors associated with time t are merged to obtain i an associated set of merged feature vectors; According to the time t i The associated merged feature vector, generated using a recurrent neural network with time t i an associated set of predicted features, wherein the predicted features represent the one or more regions of interest and the vehicle trajectory; Use the decoder neural network to extract the i The prediction features associated with t i The associated importance occupies the grid map; According to the time t i The corresponding labeled training data includes the ground truth occupancy grid map and the i The associated importance occupies the grid map and the error signal is calculated; The error signal is back-propagated to the map feature extractor neural network, the path feature extractor neural network, the recursive neural network and the decoder neural network to adjust the parameters of the map feature extractor neural network, the path feature extractor neural network, the recursive neural network and the decoder neural network.

11. The method according to claim 10, characterized in that The path command information representing the planned path includes the angular velocity and the vehicle speed of the vehicle.

12. The method according to claim 10 or 11, characterized in that: The input occupancy grid map is also generated based on the time t i The path command information included in the corresponding labeled training data includes at least one of the following information: i Information about the height values ​​of a grid or polygon of a set of points in a 3D point cloud, expressed as a function of time t i The information of the average height value of one or more columns of points in the associated 3D point cloud, indicating the relationship between the average height value and the time t i The information of the intensity value of one or more points in the associated 3D point cloud, based on the time t i The information, representation and time t of the value range of one or more height values ​​of one or more points in the associated 3D point cloud i Information about the point density value of one or more points in the associated 3D point cloud.

13. A method for generating a ground truth importance occupancy grid map, characterized in that The method comprises: Receiving a plurality of 3D point clouds, wherein the plurality of 3D point clouds are respectively associated with corresponding times in a set of continuous times, and the plurality of 3D point clouds each include one or more objects; Processing the plurality of 3D point clouds to obtain a unique target ID and a target type for each of the one or more targets; receiving a plurality of GPS coordinates of the vehicle, wherein the plurality of GPS coordinates are generated at corresponding times in the set of consecutive times; receiving a plurality of binary maps, wherein the plurality of binary maps are respectively associated with corresponding times in the set of consecutive times and are based on GPS coordinates of the vehicle at the corresponding times; For each corresponding time in the set of consecutive times, generating a target map based on the 3D point cloud associated with the corresponding time, wherein the target map includes a representation of each of the one or more targets; For each of the one or more targets, generating a track map associated with the unique target ID for the target at all times in the set of continuous times based on a target map associated with each time in the set of continuous times, wherein the track map includes each instance of the target at all times in the set of continuous times, each instance of the target being generated based on a GPS position of the target at a corresponding time in the set of continuous times; Determining trajectory paths for the one or more targets respectively according to the trajectory map of the target; generating a list of related targets for the vehicle based on the trajectory paths of the one or more targets; For each corresponding time in the set of consecutive times, generating a trajectory map for the vehicle including a trajectory path of the vehicle, wherein the trajectory path of the vehicle is determined based on the binary map associated with the corresponding time and the GPS coordinates of the vehicle associated with the corresponding time, and the trajectory path of the vehicle is represented using a road mask; For each corresponding time in the set of continuous times, a ground truth importance occupancy grid map is generated based on the trajectory path of the vehicle associated with the corresponding time and the trajectory path of each target in the relevant target list, wherein the ground truth importance occupancy grid map includes: a road mask representing the trajectory path of the vehicle associated with the corresponding time, a target mask of each target in the relevant target list, and a distance between the vehicle in the trajectory path of the vehicle and each target mask of each target in the relevant target list.

14. The method according to claim 13, characterized in that The related target list includes at least one target among the one or more targets, the at least one target having a minimum Euclidean distance with the vehicle within the set of continuous times, the minimum Euclidean distance being less than a predetermined threshold.

15. The method according to claim 13 or 14, characterized in that The road mask for representing the trajectory path of the vehicle has a width determined according to one or more of: a road type of the trajectory path, a speed of the vehicle, and a type of the vehicle.

16. The method according to claim 15, characterized in that The target mask of each target in the associated target list has a shape, size, radius, width or length determined according to the kinematic information of the target.

17. A vehicle control system, characterized in that: The vehicle control system comprises: processor; A memory storing instructions, wherein when the instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 16.

18. A computer-readable medium comprising instructions, characterized in that: When the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 16.

19. A computer program product comprising instructions, characterized in that When the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 16.

Images