Cooperative v2x sensor sharing

Abstract

A system for sharing sensor data between a vehicle and a plurality of remote systems generally includes a vehicle communication system, a three-dimensional (3D) sensor, and a controller. The controller is programmed to generate an initial three-dimensional point cloud using the 3D sensor, generate an occupancy grid map based on the initial 3D point cloud, and select a producer remote system to provide data for cells of the occupancy grid map classified as first blind spots. The controller is further programmed to send a data request to the producer remote system using the vehicle communication system, receive data from the producer remote system including information related to the cells of the occupancy grid map classified as first blind spots, generate a merged 3D point cloud by merging the data received from the producer remote system and the initial 3D point cloud, and identify an object using the merged 3D point cloud.

Description

Technical Field

[0001] This disclosure relates to a system and method for sharing sensor data between a vehicle and one or more remote systems. Background Technology

[0002] For autonomous driving and / or driver assistance systems to operate, vehicles must be able to detect and / or measure objects in their surrounding environment. Systems and methods have been developed to enable vehicles to collect information about their environment. For example, ranging sensors, such as infrared ranging systems, can be used to measure the distance between the vehicle and objects in the environment. The effectiveness of ranging sensor systems can be limited by the geometry of the environment. For example, certain areas of the environment may be obscured by objects that are opaque to the ranging sensors, resulting in blind spots in the vehicle's perception of the environment. To mitigate blind spots, systems and methods have been developed to share sensor data between systems (e.g., between vehicles). However, systems and methods for sharing sensor data often consume significant bandwidth on communication networks, leading to reduced efficiency.

[0003] Therefore, while current systems and methods for sharing sensor data between a vehicle and one or more remote systems have achieved their intended purpose, there is a need for new and improved systems and methods for sharing sensor data between a vehicle and one or more remote systems. Summary of the Invention

[0004] According to several aspects, a system for sharing sensor data between a vehicle and multiple remote systems is provided. The system includes a vehicle communication system, three-dimensional (3D) sensors, and a controller that electrically communicates with the vehicle communication system and the 3D sensors. The controller is programmed to generate an initial 3D point cloud of the environment surrounding the vehicle using the 3D sensors, wherein each point in the initial 3D point cloud represents a position on a surface in the environment and has x, y, and z coordinates relative to the vehicle. The controller is further programmed to generate an occupancy mesh map based on the initial 3D point cloud, which includes cells classified as first blind spots in the occupancy mesh map. The controller is further programmed to select a producer remote system to provide data for the cells classified as first blind spots in the occupancy mesh map. The controller is further programmed to send a data request to the producer remote system using the vehicle communication system, the data request being based at least on the cells classified as first blind spots in the occupancy mesh map. The controller is further programmed to receive data from the producer remote system, which includes information about the cells classified as first blind spots in the occupancy mesh map. The controller is further programmed to generate a merged 3D point cloud by merging the data received from the producer remote system and the initial 3D point cloud. The controller is further programmed to identify objects in the environment surrounding the vehicle using merged 3D point clouds.

[0005] In another aspect of this disclosure, to generate an occupancy mesh map, the controller is further programmed to classify each point in the initial 3D point cloud as at least one of ground points or object points, based at least in part on the x, y, and z coordinates of each point. To generate the occupancy mesh map, the controller is further programmed to divide the vehicle's surrounding environment into a plurality of 2D meshes, each of which is defined by a set of x and y coordinates relative to the vehicle. To generate the occupancy mesh map, the controller is further programmed to determine the number of ground points having x and y coordinates within that set of x and y coordinates for each of the plurality of meshes in the 2D mesh, and to determine the number of object points having x and y coordinates within that set of x and y coordinates for each of the plurality of meshes in the 2D mesh. To generate the occupancy mesh map, the controller is further programmed to classify each of the plurality of meshes in the 2D mesh as a drivable unit, a non-drivable unit, or a blind spot unit, based at least in part on the number of ground points having x and y coordinates within that set of x and y coordinates for each of the plurality of meshes in the 2D mesh. To generate the occupancy mesh map, the controller is further programmed to define the occupancy mesh map as a plurality of 2D meshes.

[0006] In another aspect of this disclosure, for selecting a producer remote system, the controller is further programmed to receive, using a vehicle communication system, a plurality of remote occupancy grid maps from a plurality of remote systems, each of the plurality of remote occupancy grid maps comprising a 2D grid of a plurality of remote units, each of the plurality of remote units being classified as at least one of a remote drivable unit, a remote non-drivable unit, and a remote blind spot unit. For selecting a producer remote system, the controller is further programmed to receive, using the vehicle communication system, a plurality of geographic locations corresponding to each of the plurality of remote systems. For selecting a producer remote system, the controller is further programmed to, for a first blind spot unit of the occupancy grid map, identify a plurality of potential producer remote systems using information about the first blind spot unit of the occupancy grid map, wherein the plurality of potential producer remote systems is a subset of the plurality of remote systems. For selecting a producer remote system, the controller is further programmed to select a producer remote system from the plurality of potential producer remote systems to provide data for the first blind spot unit of the occupancy grid map.

[0007] In another aspect of this disclosure, the system further includes a Global Navigation Satellite System (GNSS). To identify multiple potential producer remote systems, the controller is further programmed to determine the vehicle's geographic location using GPS. To identify multiple potential producer remote systems, the controller is further programmed to determine the cell location of the first blind spot cell in the occupancy grid using the vehicle's geographic location and the set of x and y coordinates defining the first blind spot cell in the occupancy grid. To identify multiple potential producer remote systems, the controller is further programmed to determine multiple remote cell locations for each remote drivable cell in each remote occupancy grid and each remote non-drivable cell in each remote occupancy grid based on the geographic location of each of the multiple remote systems and the set of x and y coordinates defining each of the multiple remote cells. To identify multiple potential producer remote systems, the controller is further programmed to identify multiple potential producer remote systems, wherein the multiple potential producer remote systems comprise a subset of multiple remote systems, wherein the remote occupancy grid map of each of the multiple potential producer remote systems includes at least one remotely drivable unit or remotely non-drivable unit, and wherein the remote unit position of at least one remotely drivable unit or remotely non-drivable unit is the same as the unit position of a first blind spot unit in the occupancy grid map.

[0008] In another aspect of this disclosure, in order to send a data request to the producer's remote system, the controller is further programmed to request data from the producer's remote system for at least one remotely drivable or remotely non-drivable unit of a remotely occupied grid.

[0009] In another aspect of this disclosure, in order to receive data from the producer's remote system, the controller is further programmed to receive a remote 3D point cloud of at least one remotely drivable or remotely non-drivable unit from the producer's remote system's remote occupancy grid.

[0010] In another aspect of this disclosure, to generate the merged 3D point cloud, the controller is further programmed to calculate the time difference between the timestamp of the remote 3D point cloud and the timestamp of the initial 3D point cloud, and adjust the x, y, and z coordinates of the remote 3D point cloud based on the velocity of the producer's remote system to compensate for the velocity of the producer's remote system in response to determining that the time difference is less than a predetermined staleness threshold. To generate the merged 3D point cloud, the controller is further programmed to transform the remote 3D point cloud such that the origin of the remote 3D point cloud is the same as the origin of the initial 3D point cloud. To generate the merged 3D point cloud, the controller is further programmed to identify multiple overlapping points in the remote 3D point cloud based on multiple remote cell positions from the producer's remote system's remote occupancy grid map and multiple cell positions from the vehicle's occupancy grid map, these overlapping points being expected to have the same x, y, and z coordinates as multiple reference points in the initial 3D point cloud. To generate the merged 3D point cloud, the controller is further programmed to determine the percentage of multiple overlapping points having the same x, y, and z coordinates as multiple reference points in the initial 3D point cloud. To generate the merged 3D point cloud, the controller is further programmed to combine the transformed remote 3D point cloud with the initial 3D point cloud in response to determining that the percentage of multiple overlapping points having the same x, y, and z coordinates as the reference point in the initial 3D point cloud is greater than or equal to a predetermined validity threshold.

[0011] In another aspect of this disclosure, in order to select a producer remote system, the controller is further programmed to select a producer remote system from the plurality of potential producer remote systems based at least in part on the distance between the vehicle and each of the plurality of potential producer remote systems.

[0012] In another aspect of this disclosure, the controller is further programmed to continuously broadcast an occupancy grid map using a vehicle communication system.

[0013] In another aspect of this disclosure, the controller is further programmed to transmit a 3D point cloud to a remote system using a vehicle communication system in response to a remote data request from one of a plurality of remote systems.

[0014] According to several aspects, a method for sharing sensor data between a vehicle and multiple remote systems is provided. The method includes generating an initial 3D point cloud of the vehicle's surrounding environment using 3D sensors, wherein each point in the initial 3D point cloud has x, y, and z coordinates relative to the vehicle. The method also includes generating an occupancy grid map based on the initial 3D point cloud, including a first blind spot. The method further includes selecting a producer remote system to provide data for the first blind spot in the occupancy grid map. The method also includes sending a data request to the producer remote system using a vehicle communication system, at least in part, based on the first blind spot in the occupancy grid map. The method also includes receiving data from the producer remote system, including information about the first blind spot in the occupancy grid map. The method further includes generating a merged 3D point cloud by merging the data received from the producer remote system and the initial 3D point cloud. The method also includes identifying objects in the vehicle's surrounding environment using the merged 3D point cloud.

[0015] In another aspect of this disclosure, generating the occupancy mesh map may further include classifying each point in the initial 3D point cloud as a ground point or an object point, at least in part based on the x, y, and z coordinates of each point. Generating the occupancy mesh map may further include dividing the vehicle's surrounding environment into a plurality of 2D meshes, each of which is defined by a set of x and y coordinates relative to the vehicle. Generating the occupancy mesh map may further include determining the number of ground points having x and y coordinates within that set of x and y coordinates for each of the plurality of meshes in the 2D mesh, and determining the number of object points having x and y coordinates within that set of x and y coordinates for each of the plurality of meshes in the 2D mesh. Generating the occupancy mesh map may further include classifying each of the plurality of meshes in the 2D mesh as a drivable unit, a non-drivable unit, or a blind spot unit, at least in part based on the number of ground points and the number of object points having x and y coordinates within that set of x and y coordinates for each of the plurality of meshes in the 2D mesh. Generating the occupancy mesh map may further include defining the occupancy mesh map as a plurality of 2D meshes.

[0016] In another aspect of this disclosure, the process of classifying each of a plurality of cells in a 2D mesh as a drivable cell, a non-drivable cell, or a blind spot cell may further include comparing the number of ground points in each of the plurality of cells with the number of object points in each of the plurality of cells. Classifying each of the plurality of cells in the 2D mesh may further include classifying a first cell in the plurality of cells in the 2D mesh as a drivable cell in response to determining that the number of ground points in a first cell is greater than zero and that the number of ground points in the first cell is greater than the number of object points in the first cell. Classifying each of the plurality of cells in the 2D mesh may further include classifying a second cell in the plurality of cells in the 2D mesh as a non-drivable cell in response to determining that the sum of the number of ground points and object points having x and y coordinates within the set of x and y coordinates defining a third cell is less than or equal to a predetermined blind spot threshold.

[0017] In another aspect of this disclosure, selecting a producer remote system may further include receiving multiple remote occupancy grid maps from multiple remote systems using a vehicle communication system, each of the multiple remote occupancy grid maps comprising a 2D grid of multiple remote units, each of the multiple remote units being classified as a remote drivable unit, a remote non-drivable unit, or a remote blind spot unit. Selecting a producer remote system may further include receiving multiple geographic locations corresponding to each of the multiple remote systems using the vehicle communication system. Selecting a producer remote system may further include identifying multiple potential producer remote systems using information about a first blind spot unit of the occupancy grid map, wherein the multiple potential producer remote systems are a subset of the multiple remote systems. Selecting a producer remote system may further include selecting a producer remote system from the multiple potential producer remote systems to provide data for the first blind spot unit of the occupancy grid map based at least in part on the distance between the vehicle and each of the multiple potential producer remote systems.

[0018] In another aspect of this disclosure, identifying multiple potential producer remote systems may further include determining the geographic location of the vehicle using a Global Navigation Satellite System (GNSS). Identifying multiple potential producer remote systems may further include determining the cell location of the first blind spot cell in the occupancy grid using the vehicle's geographic location and the set of x and y coordinates defining the first blind spot cell of the occupancy grid. Identifying multiple potential producer remote systems may further include determining multiple remote cell locations for each remote drivable cell in each remote occupancy grid and each remote non-drivable cell in each remote occupancy grid in the multiple remote occupancy grid, based on the geographic location of each of the multiple remote systems and the set of x and y coordinates defining each of the multiple remote cells. Identifying multiple potential producer remote systems may further include identifying multiple potential producer remote systems, wherein the multiple potential producer remote systems comprise a subset of the multiple remote systems, wherein the remote occupancy grid of each of the multiple potential producer remote systems includes at least one remote drivable cell or remote non-drivable cell, wherein the remote cell location of at least one remote drivable cell or remote non-drivable cell is the same as the cell location of the first blind spot cell in the occupancy grid.

[0019] In another aspect of this disclosure, receiving data from the producer's remote system may further include receiving a remote 3D point cloud of at least one remotely drivable or remotely non-drivable unit from the producer's remote system, representing a remote occupancy grid.

[0020] In another aspect of this disclosure, generating the merged 3D point cloud may further include calculating the time difference between the timestamp of the remote 3D point cloud and the timestamp of the initial 3D point cloud. Generating the merged 3D point cloud may further include adjusting the x, y, and z coordinates of the remote 3D point cloud based on the velocity of the producer's remote system to compensate for the velocity of the producer's remote system in response to determining that the time difference is less than a predetermined slack threshold. Generating the merged 3D point cloud may further include transforming the remote 3D point cloud such that the origin of the remote 3D point cloud is the same as the origin of the initial 3D point cloud. Generating the merged 3D point cloud may further include identifying multiple overlapping points in the remote 3D point cloud based on multiple remote cell locations from the producer's remote system's remote occupancy mesh map and multiple cell locations from the vehicle's occupancy mesh map, these overlapping points being expected to have the same x, y, and z coordinates as multiple reference points in the initial 3D point cloud. Generating the merged 3D point cloud may further include determining the percentage of multiple overlapping points having the same x, y, and z coordinates as multiple reference points in the initial 3D point cloud. Generating a merged 3D point cloud may further include combining the transformed remote 3D point cloud with the initial 3D point cloud to generate a merged 3D point cloud in response to determining that the percentage of multiple overlapping points having the same x, y, and z coordinates as reference points in the initial 3D point cloud is greater than or equal to a predetermined validity threshold.

[0021] According to several aspects, a system for sharing sensor data between a vehicle and multiple remote systems is provided. The system also includes a vehicle communication system, a Global Navigation Satellite System (GNSS), and a three-dimensional (3D) sensor, wherein the 3D sensor includes at least one of: a lidar sensor and a stereo camera system. The system further includes a controller electrically communicating with the vehicle communication system, GPS, and the 3D sensor, wherein the controller is programmed to generate an initial 3D point cloud of the vehicle's surrounding environment using the 3D sensor, wherein each point in the initial 3D point cloud has x, y, and z coordinates relative to the vehicle. The controller is further programmed to generate an occupancy grid map based on the initial 3D point cloud, wherein the occupancy grid map represents the vehicle's surrounding environment as a 2D grid of multiple cells, each of which is defined by a set of x and y coordinates relative to the vehicle. The controller is further programmed to classify each of the multiple cells in the occupancy grid map into a drivable cell, a non-drivable cell, or a blind spot cell, at least in part based on the x, y, and z coordinates of the multiple points having x and y coordinates within each of the multiple cells in the initial 3D point cloud. The controller is further programmed to select a producer remote system to provide data for a first blind spot cell in the occupancy grid. The controller is further programmed to send a data request to the producer remote system using a vehicle communication system, based at least in part on the first blind spot cell in the occupancy grid. The controller is further programmed to receive data from the producer remote system, including information about the first blind spot in the occupancy grid. The controller is further programmed to generate a merged 3D point cloud by combining the data received from the producer remote system with an initial 3D point cloud. The controller is further programmed to use the merged 3D point cloud to identify objects in the environment surrounding the vehicle.

[0022] In another aspect of this disclosure, for selecting a producer remote system, the controller is further programmed to receive multiple remote occupancy grid maps from multiple remote systems using a vehicle communication system. Each of the multiple remote occupancy grid maps includes a 2D grid of multiple remote units, each of the multiple remote units being classified as a remote drivable unit, a remote non-drivable unit, or a remote blind spot unit. For selecting a producer remote system, the controller is further programmed to receive multiple geographic locations corresponding to each of the multiple remote systems using the vehicle communication system. For selecting a producer remote system, the controller is further programmed to determine the vehicle's geographic location using GPS and to determine the unit location of the first blind spot unit of the occupancy grid map using the vehicle's geographic location and the set of x and y coordinates defining the first blind spot unit of the occupancy grid map. For selecting a producer remote system, the controller is further programmed to determine the multiple remote unit locations of each remote drivable unit and each remote non-drivable unit of each of the multiple remote occupancy grid maps based on the geographic location of each of the multiple remote systems and the set of x and y coordinates defining each of the multiple remote units. To select a producer remote system, the controller is further programmed to identify a plurality of potential producer remote systems, wherein the plurality of potential producer remote systems comprises a subset of multiple remote systems, wherein the remote occupancy grid map of each of the plurality of potential producer remote systems includes at least one remotely drivable unit or a remotely non-drivable unit, wherein the remote unit location of at least one remotely drivable unit or remotely non-drivable unit is the same as the unit location of a first blind spot unit in the occupancy grid map. To select a producer remote system, the controller is further programmed to select a producer remote system from the plurality of potential producer remote systems based at least in part on the distance between the vehicle and each of the plurality of potential producer remote systems.

[0023] In another aspect of this disclosure, the system further includes a human-machine interface (HMI), and the controller is further programmed to use the HMI to provide a visual representation of the merged 3D point cloud to the occupants of the vehicle.

[0024] The present invention has the following solutions.

[0025] Option 1. A system for sharing sensor data between a vehicle and multiple remote systems, the system comprising: Vehicle communication system; Three-dimensional (3D) sensors; and A controller that communicates electrically with the vehicle communication system and the 3D sensor, wherein the controller is programmed to: The 3D sensor is used to generate an initial 3D point cloud of the environment surrounding the vehicle, wherein each point in the initial 3D point cloud represents the position of a surface in the environment and has x, y, and z coordinates relative to the vehicle. An occupancy grid map is generated based on the initial 3D point cloud, which includes the cells of the occupancy grid map that are classified as first blind spots; Select a producer remote system to provide data for the cells in the occupancy grid map that are classified as the first blind spot; The vehicle communication system is used to send a data request to the producer remote system, the data request being based at least on the cells in the occupancy grid map that are classified as the first blind spot; Data is received from the producer's remote system, including information about the cells in the occupancy grid that are classified as the first blind spot; A merged 3D point cloud is generated by combining the data received from the producer's remote system with the initial 3D point cloud; and The merged 3D point cloud is used to identify objects in the environment surrounding the vehicle.

[0026] Option 2. The system according to Option 1, wherein in order to generate the occupancy grid map, the controller is further programmed to: Each point in the initial 3D point cloud is classified, at least in part, as a ground point or an object point based on the x, y, and z coordinates of each point. The environment surrounding the vehicle is divided into a plurality of 2D meshes, each of which is defined by a set of x and y coordinates relative to the vehicle; Determine the number of ground points with x and y coordinates within the set of x and y coordinates for each of the plurality of cells in the 2D mesh; Determine the number of object points with x and y coordinates within the set of x and y coordinates in each of the plurality of cells in the 2D mesh; Each cell in the 2D mesh is classified as a drivable cell, a non-drivable cell, or a blind spot cell, based at least in part on the number of ground points and object points having x and y coordinates within the set of x and y coordinates in each of the plurality of cells in the 2D mesh; and The occupancy grid is defined as the 2D grid of the plurality of cells.

[0027] Option 3. The system according to Option 2, wherein, in order to select the producer remote system, the controller is further programmed to: The vehicle communication system receives multiple remote occupancy grid maps from the multiple remote systems, each of the multiple remote occupancy grid maps comprising a 2D grid of multiple remote units, each of the multiple remote units being classified as at least one of a remote drivable unit, a remote non-drivable unit, and a remote blind spot unit; The vehicle communication system is used to receive multiple geographical locations corresponding to each of the plurality of remote systems; For a first blind spot cell in the occupancy grid, multiple potential producer remote systems are identified using information about the first blind spot cell in the occupancy grid, wherein the multiple potential producer remote systems are a subset of the multiple remote systems; and The producer remote system is selected from the plurality of potential producer remote systems to provide data for the first blind spot cell of the occupied grid graph.

[0028] Option 4. The system according to Option 3 further includes a Global Navigation Satellite System (GNSS) in electrical communication with the controller, wherein, in order to identify the plurality of potential producer remote systems, the controller is further programmed to: The vehicle's geographical location was determined using GPS; The cell position of the first blind spot cell in the occupancy grid is determined using the vehicle's geographic location and the set of x and y coordinates that define the first blind spot cell in the occupancy grid. Based on the geographic location of each of the plurality of remote systems and defining the set of x and y coordinates for each of the plurality of remote units, determine the locations of multiple remote units, including each remotely drivable unit in each remotely occupied grid map and each remotely non-drivable unit in each remotely occupied grid map; and Identify the plurality of potential producer remote systems, wherein the plurality of potential producer remote systems comprises a subset of the plurality of remote systems, wherein the remote occupancy grid map of each of the plurality of potential producer remote systems includes at least one remotely drivable unit or a remotely non-drivable unit, and wherein the remote unit position of the at least one remotely drivable unit or remotely non-drivable unit is the same as the unit position of the first blind spot unit of the occupancy grid map.

[0029] Option 5. The system according to Option 3, wherein in order to send the data request to the producer remote system, the controller is further programmed to: The producer remote system requests data from at least one remotely drivable or remotely non-drivable unit of the remote occupancy grid.

[0030] Option 6. The system according to Option 3, wherein in order to receive data from the producer's remote system, the controller is further programmed to: Receive remote 3D point cloud of at least one remotely drivable or remotely non-drivable unit from the remote occupancy grid map from the producer remote system.

[0031] Option 7. The system according to Option 6, wherein, in order to generate the merged 3D point cloud, the controller is further programmed to: Calculate the time difference between the timestamp of the remote 3D point cloud and the timestamp of the initial 3D point cloud; The x, y, and z coordinates of the remote 3D point cloud are adjusted based on the speed of the producer remote system in response to determining that the time difference is less than a predetermined timeout threshold to compensate for the speed of the producer remote system. Transform the remote 3D point cloud so that the origin of the coordinates of the remote 3D point cloud is the same as the origin of the coordinates of the initial 3D point cloud. Based on the multiple remote cell positions from the remote occupancy grid map of the producer's remote system and the multiple cell positions from the vehicle's occupancy grid map, multiple overlapping points in the remote 3D point cloud are identified, the multiple overlapping points being expected to have the same x, y, and z coordinates as the multiple reference points in the initial 3D point cloud; Determine the percentage of the plurality of overlapping points, which have the same x, y, and z coordinates as the plurality of reference points in the initial 3D point cloud; and In response to determining that the percentage of the plurality of overlapping points having the same x, y, and z coordinates as the reference point in the initial 3D point cloud is greater than or equal to a predetermined validity threshold, the transformed remote 3D point cloud is combined with the initial 3D point cloud to generate the merged 3D point cloud.

[0032] Option 8. The system according to Option 3, wherein, in order to select the producer remote system, the controller is further programmed to: The producer remote system is selected from the plurality of potential producer remote systems based at least in part on the distance between the vehicle and each of the plurality of potential producer remote systems.

[0033] Option 9. The system according to Option 1, wherein the controller is further programmed to: The vehicle communication system is used to continuously broadcast the occupancy grid map.

[0034] Option 10. The system according to Option 1, wherein the controller is further programmed to: In response to a remote data request from one of the plurality of remote systems, the vehicle communication system is used to transmit a 3D point cloud to the remote system.

[0035] Option 11. A method for sharing sensor data between a vehicle and multiple remote systems, the method comprising: An initial 3D point cloud of the environment surrounding the vehicle is generated using 3D sensors, wherein each point in the initial 3D point cloud has x, y, and z coordinates relative to the vehicle. An occupancy grid map is generated based on the initial 3D point cloud, including the first blind spot. Select a producer remote system to provide data for the first blind spot in the occupancy grid; At least in part based on the first blind spot in the occupancy grid, a data request is sent to the producer remote system using the vehicle communication system; Data is received from the producer's remote system, including information about the first blind spot in the occupancy grid diagram; A merged 3D point cloud is generated by combining the data received from the producer's remote system with the initial 3D point cloud; and The merged 3D point cloud is used to identify objects in the environment surrounding the vehicle.

[0036] Option 12. The method according to Option 11, wherein generating the occupancy grid map further comprises: Each point in the initial 3D point cloud is classified as a ground point or an object point, at least in part, based on the x, y, and z coordinates of each point. The environment surrounding the vehicle is divided into a plurality of 2D meshes, each of which is defined by a set of x and y coordinates relative to the vehicle; Determine the number of ground points with x and y coordinates within the set of x and y coordinates for each of the plurality of cells in the 2D mesh; Determine the number of object points with x and y coordinates within the set of x and y coordinates in each of the plurality of cells in the 2D mesh; Each cell in the 2D mesh is classified as a drivable cell, a non-drivable cell, or a blind spot cell, based at least in part on the number of ground points and object points having x and y coordinates within the set of x and y coordinates in each of the plurality of cells in the 2D mesh; and The occupancy grid is defined as the 2D grid of the plurality of cells.

[0037] Option 13. The method according to Option 12, wherein classifying each of the plurality of cells in the 2D mesh into a drivable cell, a non-drivable cell, or a blind spot cell further comprises: The number of ground points in each of the plurality of units is compared with the number of object points in each of the plurality of units; In response to determining that the number of ground points in the first cell is greater than zero, and that the number of ground points in the first cell is greater than the number of object points in the first cell, the first cell among the plurality of cells in the 2D mesh is classified as a drivable cell; In response to determining that the number of object points in the second cell is greater than zero, and that the number of object points in the second cell is greater than or equal to the number of ground points in the second cell, the second cell among the plurality of cells in the 2D mesh is classified as a non-drivable cell; and In response to determining that the sum of the number of ground points and object points with x and y coordinates within the group of x and y coordinates that define the third unit is less than or equal to a predetermined blind spot threshold, the third unit among the plurality of units in the 2D mesh is classified as a blind spot unit.

[0038] Option 14. The method according to Option 11, wherein selecting the producer remote system further includes: The vehicle communication system is used to receive multiple remote occupancy grid maps from the multiple remote systems. Each of the multiple remote occupancy grid maps includes a 2D grid of multiple remote units, each of which is classified as a remote drivable unit, a remote non-drivable unit, or a remote blind spot unit. The vehicle communication system is used to receive multiple geographical locations corresponding to each of the plurality of remote systems; Identify multiple potential producer remote systems using information about the first blind spot cells of the occupancy grid map, wherein the multiple potential producer remote systems are a subset of the multiple remote systems; and The producer remote system is selected from the plurality of potential producer remote systems to provide data for the first blind spot cell of the occupancy grid map based at least in part on the distance between the vehicle and each of the plurality of potential producer remote systems.

[0039] Option 15. The method according to Option 14, wherein identifying the plurality of potential producer remote systems further comprises: The geographical location of the vehicle was determined using a Global Navigation Satellite System (GNSS); The cell position of the first blind spot cell in the occupancy grid is determined using the vehicle's geographic location and the set of x and y coordinates that define the first blind spot cell in the occupancy grid. Based on the geographical location of each of the plurality of remote systems and the defined set of x and y coordinates of each of the plurality of remote units, the locations of multiple remote units are determined for each remotely drivable unit in each remotely occupied grid map and for each remotely non-drivable unit in each remotely occupied grid map; and Identify the plurality of potential producer remote systems, wherein the plurality of potential producer remote systems comprises a subset of the plurality of remote systems, wherein the remote occupancy grid map of each of the plurality of potential producer remote systems includes at least one remotely drivable unit or a remotely non-drivable unit, wherein the remote unit position of the at least one remotely drivable unit or the remotely non-drivable unit is the same as the unit position of the first blind spot unit of the occupancy grid map.

[0040] Option 16. The method according to Option 11, wherein receiving data from the producer remote system further comprises: Receive remote 3D point cloud of at least one remotely drivable or remotely non-drivable unit from the remote occupancy grid map from the producer remote system.

[0041] Option 17. The method according to Option 16, wherein generating the merged 3D point cloud further comprises: Calculate the time difference between the timestamp of the remote 3D point cloud and the timestamp of the initial 3D point cloud; The x, y, and z coordinates of the remote 3D point cloud are adjusted based on the speed of the producer remote system in response to determining that the time difference is less than a predetermined timeout threshold to compensate for the speed of the producer remote system. Transform the remote 3D point cloud so that the origin of the coordinates of the remote 3D point cloud is the same as the origin of the coordinates of the initial 3D point cloud. Based on the multiple remote cell positions from the remote occupancy grid map of the producer's remote system and the multiple cell positions from the vehicle's occupancy grid map, multiple overlapping points in the remote 3D point cloud are identified, the multiple overlapping points being expected to have the same x, y, and z coordinates as multiple reference points in the initial 3D point cloud; Determine the percentage of the plurality of overlapping points, which have the same x, y, and z coordinates as the plurality of reference points in the initial 3D point cloud; and In response to determining that the percentage of the plurality of overlapping points having the same x, y, and z coordinates as the reference point in the initial 3D point cloud is greater than or equal to a predetermined validity threshold, the transformed remote 3D point cloud is combined with the initial 3D point cloud to generate a merged 3D point cloud.

[0042] Option 18. A system for sharing sensor data between a vehicle and multiple remote systems, the system comprising: Vehicle communication system; Global Navigation Satellite System (GNSS); A three-dimensional (3D) sensor, wherein the 3D sensor comprises at least one of: a lidar sensor and a stereo camera system; and A controller that communicates electrically with the vehicle communication system, GPS, and the 3D sensor, wherein the controller is programmed to: The 3D sensor is used to generate an initial 3D point cloud of the environment surrounding the vehicle, wherein each point in the initial 3D point cloud has x, y, and z coordinates relative to the vehicle. An occupancy mesh map is generated based on the initial 3D point cloud, wherein the occupancy mesh map represents the environment around the vehicle as a plurality of 2D meshes, each of the plurality of cells being defined by a set of x and y coordinates relative to the vehicle; Each of the multiple cells in the occupied grid is classified as a drivable cell, a non-drivable cell, or a blind spot cell, based at least in part on the x, y, and z coordinates of multiple points having x and y coordinates within each of the multiple cells in the initial 3D point cloud. Select a producer remote system to provide data for the first blind spot cell in the occupied grid diagram; At least in part, based on the first blind spot unit in the occupancy grid, the vehicle communication system sends a data request to the producer remote system. Data is received from the producer's remote system, including information about the first blind spot in the occupancy grid diagram; A merged 3D point cloud is generated by combining the data received from the producer's remote system with the initial 3D point cloud; and The merged 3D point cloud is used to identify objects in the environment surrounding the vehicle.

[0043] Option 19. The system according to Option 18, wherein, in order to select the producer remote system, the controller is further programmed to: The vehicle communication system is used to receive multiple remote occupancy grid maps from the multiple remote systems. Each of the multiple remote occupancy grid maps includes a 2D grid of multiple remote units, each of which is classified as a remote drivable unit, a remote non-drivable unit, or a remote blind spot unit. The vehicle communication system is used to receive multiple geographical locations corresponding to each of the plurality of remote systems; The vehicle's geographical location was determined using GPS; The cell position of the first blind spot cell in the occupancy grid is determined using the vehicle's geographic location and the set of x and y coordinates that define the first blind spot cell in the occupancy grid. Based on the geographical location of each of the multiple remote systems and the defined set of x and y coordinates of each of the multiple remote units, determine the locations of multiple remote units, including each remote drivable unit in each remote occupancy grid and each remote non-drivable unit in each remote occupancy grid. Identify multiple potential producer remote systems, wherein the multiple potential producer remote systems comprise a subset of the multiple remote systems, wherein the remote occupancy grid map of each of the multiple potential producer remote systems includes at least one remotely drivable unit or a remotely non-drivable unit, wherein the remote unit location of the at least one remotely drivable unit or remotely non-drivable unit is the same as the unit location of the first blind spot unit in the occupancy grid map; and The producer remote system is selected from the plurality of potential producer remote systems based at least in part on the distance between the vehicle and each of the plurality of potential producer remote systems.

[0044] Option 20. The system according to Option 18, the system further includes a human-machine interface (HMI), and wherein the controller is further programmed to: The HMI is used to provide the occupants of the vehicle with a visual representation of the merged 3D point cloud.

[0045] Further applicability will become apparent from the description provided herein. It should be understood that the descriptions and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0046] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0047] Figure 1This is a system diagram of a system for sharing sensor data between a vehicle and multiple remote systems, according to an exemplary embodiment.

[0048] Figure 2A This is a flowchart of a method for sharing sensor data between a vehicle and multiple remote systems, according to an exemplary embodiment.

[0049] Figure 2B This is a continuation of the flowchart of a method for sharing sensor data between a vehicle and multiple remote systems according to an exemplary embodiment.

[0050] Figure 3A This is an example of an occupancy grid map generated by a vehicle according to an exemplary embodiment; and Figure 3B This is an example of an occupancy grid map generated by a remote vehicle according to an exemplary embodiment. Detailed Implementation

[0051] The following description is merely exemplary and is not intended to limit this disclosure, its application, or its uses.

[0052] refer to Figure 1 The diagram illustrates a system for sharing sensor data between a vehicle and multiple remote systems, and is generally indicated by reference numeral 10. System 10 is shown together with an exemplary vehicle 12. Although a passenger vehicle is shown, it should be understood that vehicle 12 can be any type of vehicle without departing from the scope of this disclosure. System 10 typically includes a controller 14, multiple vehicle sensors 16, and a human-machine interface (HMI) 18.

[0053] The controller 14 is programmed to implement a method 100 for sharing sensor data between a vehicle and multiple remote systems, as will be described below. The controller 14 includes at least one processor 20 and a non-transitory computer-readable storage device or medium 22. The processor 20 may be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a means for executing instructions. The computer-readable storage device or medium 22 may include volatile and non-volatile storage, such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables when the processor 20 is powered off. The computer-readable storage device or medium 22 may be implemented using a variety of memory devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or another electrical, magnetic, optical, or combined memory device capable of storing data, some of which represents executable instructions used by controller 14 to control various systems of vehicle 12. Controller 14 may also consist of multiple controllers that communicate electrically with each other.

[0054] The controller 14 communicates electrically with multiple vehicle sensors 16 and the HMI 18. This electrical communication is established using, for example, a CAN bus, a WiFi network, a cellular data network, etc. Those skilled in the art will understand that various other wired and wireless technologies and communication protocols can be used to communicate with the controller 14.

[0055] Multiple vehicle sensors 16 are used by controller 14 to collect information about vehicle 12 and its surrounding environment 24. The multiple vehicle sensors 16 are also used to communicate with systems external to vehicle 12. In a non-limiting example, the multiple vehicle sensors 16 include at least one of a lidar system 26 and a stereo camera system 28. The multiple vehicle sensors 16 further include a Global Navigation Satellite System (GNSS) 30, a vehicle system 32, and a vehicle communication system 34. The vehicle system 32 provides information such as vehicle steering angle, vehicle speed, vehicle braking, vehicle heading, vehicle attitude (i.e., pitch, yaw, roll), and vehicle acceleration.

[0056] The lidar system 26 and the stereo camera system 28 are three-dimensional (3D) sensors used to determine the distance between vehicle 12 and objects in the surrounding environment 24. Those skilled in the art will understand that various other sensors and / or systems can be used to determine the distance between vehicle 12 and objects in the surrounding environment 24.

[0057] The vehicle communication system 34 is used by the controller 14 to communicate with other systems. For example, the vehicle communication system 34 includes the ability to communicate with vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems at remote call centers (e.g., on-star systems at General Motors), and / or personal devices. In some embodiments, the vehicle communication system 34 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication. However, other or alternative communication methods, such as Dedicated Short Range Communication (DSRC) channels and / or mobile communication protocols based on 3GPP standards, are also considered within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short-to-medium range wireless communication channel specifically designed for automotive use, along with a set of corresponding protocols and standards. 3GPP refers to a collaborative relationship between several standards organizations that develop protocols and standards for mobile communications. The structure of 3GPP standards is a “version”. Therefore, communication methods based on 3GPP versions 14, 15, 16 and / or future 3GPP versions are considered to be within the scope of this disclosure. Thus, the vehicle communication system 34 may include one or more antennas and / or communication transceivers for receiving and / or transmitting signals, such as cooperative sensing messages (CSM). The vehicle communication system 34 is configured to wirelessly communicate information between vehicle 12 and another vehicle. Furthermore, the vehicle communication system 34 is configured to wirelessly transmit information between vehicle 12 and infrastructure or other vehicles. As described above, a plurality of vehicle sensors 16 communicate electrically with the controller 14.

[0058] refer to Figure 2A and Figure 2B The diagram illustrates a flowchart of a method 100 for sharing sensor data between a vehicle and multiple remote systems. Method 100 begins at block 102.

[0059] At block 102, controller 14 uses 3D sensors to scan the environment 24 surrounding vehicle 12. Figure 1 In the exemplary embodiment depicted, the 3D sensor includes a LiDAR system 26 and / or a stereo camera system 28. The LiDAR system 26 and / or the stereo camera system 28 are used to generate an initial 3D point cloud of the environment 24 surrounding the vehicle 12. Each point in the initial 3D point cloud represents an object or surface detected by the 3D sensor. Each point in the initial 3D point cloud has corresponding x, y, and z coordinates, which define the position of that point relative to the vehicle 12 in the environment 24. Therefore, the initial 3D point cloud provides information about the distance between the vehicle 12 and objects in the environment 24 surrounding the vehicle 12. After block 102, method 100 proceeds to blocks 104 and 106.

[0060] At block 104, each point in the initial 3D point cloud is classified as either a ground point or an object point. Ground points are determined to correspond to ground surfaces, such as roads or other surfaces drivable by vehicle 12. Object points are determined to correspond to objects in environment 24, such as another vehicle. In a non-limiting example, a machine learning algorithm is used to classify these points. The machine learning algorithm is trained by feeding it multiple 3D point clouds containing pre-classified points. For example, multiple 3D point clouds can include scans from 3D sensors in various driving environments. After sufficient training, the machine learning algorithm can classify points in 3D scans with high accuracy and precision. In another non-limiting example, points in the initial 3D point cloud are classified based on their z-coordinate, i.e., the height of each point relative to vehicle 12. For example, the ground position relative to vehicle 12 can be determined based on the size of the vehicle and external inputs, such as information about the terrain in the environment 24 surrounding vehicle 12 provided by GPS 30 and / or vehicle communication system 34. Points located at or around a surface location are classified as ground points, and points above a surface location are classified as object points. Those skilled in the art will understand that various other methods can be used to classify points in an initial 3D point cloud.

[0061] At block 106, the surrounding environment 24 of vehicle 12 is divided into an occupancy grid. (Refer to...) Figure 3A The best way to understand block 106 is... Figure 3A An exemplary occupancy grid map 36 generated by controller 14 at block 106 is shown. Occupancy grid map 36 is a 2D grid of multiple cells 38, each of which includes a set of x and y coordinates. Therefore, each cell 38 in occupancy grid map 36 corresponds to one or more points in an initial 3D point cloud whose x and y coordinates lie within the set of x and y coordinates defining cell 38. Each of the multiple cells 38 has a length D1 and a width D2. In the exemplary embodiment, both D1 and D2 are one meter. The size of occupancy grid map 36 can be predetermined or can be dynamically determined based on various factors, such as, for example, vehicle speed, vehicle size, and vehicle parking distance. After block 106, method 100 proceeds to the first sub-process 108.

[0062] At the first sub-process 108, each cell 38 of the occupied grid 36 is classified as a drivable cell 38a, a non-drivable cell 38b, or a blind spot cell 38c. During the first sub-process 108, the following steps (blocks 110-122) are repeated for each of the plurality of cells 38 in the occupied grid 36.

[0063] At block 110, the number of ground points in cell 38 is determined. As discussed above, cell 38 in the occupying mesh map 36 corresponds to one or more points in the initial 3D point cloud, whose x and y coordinates are defined in the set of x and y coordinates of cell 38. Furthermore, each point in the initial 3D point cloud has already been classified as a ground point or an object point at block 104. Therefore, the number of ground points in cell 38 can be determined. At block 112, the number of object points in cell 38 is determined. Then, the first subprocess 108 proceeds to block 116.

[0064] At block 116, unit 38 is classified as a drivable unit 38a, a non-drivable unit 38b, or a blind spot unit 38c. A drivable unit 38a represents an area in environment 24 where vehicle 12 can drive without colliding with objects, such as an empty road. A non-drivable unit 38b represents an area in environment 24 where vehicle 12 may be unable to drive due to objects, such as a parked vehicle on the side of the road. A blind spot unit 38c represents an area in environment 24 for which controller 14 does not have sufficient data to determine whether vehicle 12 can safely drive. For example, blind spot unit 38c may occur because an area of ​​environment 24 is obscured by a structure that is opaque to the 3D sensors (LiDAR system 26 and / or stereo camera system 28) of vehicle 12. In another example, the 3D sensors of vehicle 12 may have a maximum range smaller than the size of the occupied mesh map 36. Therefore, units 38 located outside the maximum range of the 3D sensors are classified as blind spot units 38c. To classify unit 38, the number of ground points in unit 38 determined at block 110 is compared with the number of object points in unit 38 determined at block 112. If the number of ground points is greater than zero and the number of ground points is greater than the number of object points, then the first sub-process 108 proceeds to block 118. At block 118, unit 38 is classified as a drivable unit 38a. If the number of object points is greater than zero and the number of object points is greater than or equal to the number of ground points, then the first sub-process 108 proceeds to block 120. At block 120, unit 38 is classified as a non-drivable unit 38b. If the total number of ground points and object points contained in unit 38 is less than or equal to a predetermined blind spot threshold, then the first sub-process 108 proceeds to block 122. At block 122, unit 38 is classified as a blind spot unit 38c.

[0065] As discussed above, the first sub-process 108 is repeated for each of the multiple cells 38 occupying the mesh diagram 36. The result of the first sub-process 108 is... Figure 3AThe occupancy grid diagram 36 is shown, wherein each of the plurality of cells 38 in the occupancy grid diagram 36 is classified as a drivable cell 38a, a non-drivable cell 38b, or a blind spot cell 38c. After repeating the first sub-process 108 for each of the plurality of cells 38 in the occupancy grid diagram 36, method 100 proceeds to block 124.

[0066] At block 124, GPS 30 is used and the set of x and y coordinates for each cell 38 is defined relative to vehicle 12 to determine the geographic location in environment 24 for each cell 38. After block 124, method 100 proceeds to blocks 126, 128, and 130.

[0067] At block 126, a data request is received from a remote system. In an exemplary embodiment, the remote system is a remote vehicle near vehicle 12. Figure 3B An exemplary remote occupancy grid map 40 generated by an exemplary remote vehicle 42 is shown. The remote occupancy grid map 40 includes a plurality of remote units 44, which are classified as remote drivable units 44a, remote non-drivable units 44b, and remote blind spot units 44c. In another exemplary embodiment, the remote system also includes fixed infrastructure installed near the road, such as a sensing station mounted on a pole near a road intersection. The data request includes one or more units of the occupancy grid map 36 of the vehicle 12 requesting data from the remote system. In a non-limiting example, the remote system is the remote vehicle 42. The remote vehicle 42 generates the data request based on the geographic location of the remote blind spot unit 44c in the remote occupancy grid map 40 determined by the remote vehicle 42's GPS system. Figure 3A and Figure 3B In the exemplary embodiment shown, the remote occupancy grid map 40 of the remote vehicle 42 includes at least one remote blind spot unit 44c. The remote vehicle 42 can send a data request to obtain data about the corresponding unit 38a in the occupancy grid map 36 of the vehicle 12. After block 130, method 100 proceeds to block 132.

[0068] At block 132, based on the data request received at block 130, the initial 3D point cloud generated at block 102 is segmented. Points with x and y coordinates within the set of x and y coordinates defined by the cells in the data request received at block 130 are selected. Figure 3A and Figure 3B In the exemplary embodiment shown, all points having x and y coordinates within unit 38a are selected. After block 132, method 100 proceeds to block 134.

[0069] At block 134, the vehicle communication system 34 is used to transmit a segmented 3D point cloud containing only the points selected at block 132 to a remote system that sends a data request received at block 130. Figure 3A and Figure 3B In the exemplary embodiment shown, the remote system is a remote vehicle 42. Therefore, the segmented 3D point cloud is transmitted from the controller 14 of vehicle 12 to the remote vehicle 42.

[0070] At block 128, controller 14 uses vehicle communication system 34 to broadcast the occupied grid map 36, the location and timestamp of vehicle 12 to remote systems, including remote vehicles or fixed infrastructure as discussed above.

[0071] At block 130, the vehicle communication system 34 receives multiple remote occupancy grid maps 40, remote system locations, and timestamps from multiple remote systems. In an exemplary embodiment, the multiple remote systems may include remote vehicles 42 and / or fixed infrastructure installed near the road. In a non-limiting example, multiple remote systems transmit remote occupancy grid maps 40, which are received by the controller 14 of vehicle 12 using the vehicle communication system 34 and stored in medium 22 of the controller 14. After block 130, method 100 proceeds to block 136.

[0072] At block 136, a geographic location in environment 24 is determined for each of the multiple remote cells 44 of the multiple remote occupancy grid maps 40 received at block 130. The geographic location of each of the multiple remote cells 44 is determined based on the geographic locations of each of the multiple remote systems received at block 130, and based on the set of x and y coordinates that define the geographic locations of each of the multiple remote cells 44 of the multiple remote occupancy grid maps 40 relative to each of the multiple remote systems. After block 136, method 100 proceeds to the second sub-process 138.

[0073] At the second sub-process 138, a producer remote system is selected for each blind cell 38c in the occupied mesh diagram 36. During the second sub-process 138, the following steps (blocks 140-142) are repeated for each of the multiple blind cells 38c in the occupied mesh diagram 36.

[0074] At block 140, multiple potential producer remote systems are identified from multiple remote systems. Multiple potential producer remote systems are identified based on their geographic locations within environment 24 determined at block 124 for blind spot units 38c of occupancy grid map 36. The multiple potential producer remote systems are a subset of the multiple remote systems from which multiple remote occupancy grid maps and remote system locations are received at block 130. In a non-limiting example, the multiple remote systems may include remote vehicle 42 and / or fixed infrastructure installed near the road. More specifically, the multiple potential producer remote systems are those remote systems having a remote occupancy grid map 40, which includes remote drivable units 44a or remote non-drivable units 44b corresponding to the geographic locations of blind spot units 38c of occupancy grid map 36 for vehicle 12. Figure 3A and Figure 3B In an exemplary embodiment, blind spot unit 38d in the occupancy grid map 36 of vehicle 12 is geographically located in the same location as remote non-drivable unit 44d in the remote occupancy grid map 40 of remote vehicle 42. Therefore, remote vehicle 42 is selected as a potential producer remote system for blind spot unit 38d because remote vehicle 42 has data about blind spot unit 38d. Blind spot unit 38e in the occupancy grid map 36 of vehicle 12 is geographically located in the same location as remote blind spot unit 44e in the remote occupancy grid map 40 of remote vehicle 42. Therefore, remote vehicle 42 is not selected as a potential producer remote system for blind spot unit 38e because remote vehicle 42 does not have data about blind spot unit 38e. After block 140, the second subprocess 138 proceeds to block 142.

[0075] At block 142, a producer remote system for blind spot unit 38c is selected from the group of potential producer remote systems identified at block 140. In a non-limiting example, there may be multiple potential producer remote systems with data about blind spot unit 38c. Therefore, it is necessary to select a producer remote system from this group of potential producer remote systems to request data about blind spot unit 38c from that system. In an exemplary embodiment, the producer remote system is selected based on the distance between vehicle 12 and each of the multiple potential producer remote systems. The potential producer remote system closest to vehicle 12 is selected to maximize data quality. Those skilled in the art will understand that various other methods can be used to select a producer remote system from this group of potential producer remote systems.

[0076] As discussed above, the second sub-process 138 is repeated for each blind cell 38c occupying the mesh diagram 36. In an exemplary embodiment, the result of the second sub-process 138 is one of a plurality of producer remote systems for each blind cell 38c occupying the mesh diagram 36. In another exemplary embodiment, the plurality of potential producer remote systems may not include the effective producer remote system occupying each blind cell 38c of the mesh diagram 36. After the second sub-process 138, method 100 proceeds to block 144.

[0077] At block 144, controller 14 uses vehicle communication system 34 to request data from a plurality of producer remote systems selected at second subprocess 138. In an exemplary embodiment, the plurality of producer remote systems includes remote vehicle 42. The data request includes the identification of one or more cells 44 of occupancy grid map 40 of remote vehicle 42, for which controller 14 of vehicle 12 requests data. Remote vehicle 42 responds to the data request with a segmented 3D point cloud comprising only points having x and y coordinates that fall within the set of x and y coordinates defining cell 44 in the data request, as discussed above with reference to blocks 132 and 134. The segmented 3D point cloud is transmitted along with a timestamp, allowing controller 14 of vehicle 12 to determine when the points in the segmented 3D point cloud were measured. In a non-limiting example, data is requested only for cells 38 in occupancy grid map 36 that are classified as blind spot cells 38c, thereby reducing data traffic. The data received at block 144 is referred to as the remote 3D point cloud. After block 144, method 100 proceeds to block 146.

[0078] At block 146, the time difference between the timestamp received at block 144 and the current time determined by controller 14 is calculated. If the time difference is greater than or equal to a predetermined aging threshold, the remote 3D point cloud received at block 144 is determined to be too old, and method 100 returns to the second sub-process 138 for selecting another producer remote system. If the time difference is less than the predetermined aging threshold, method 100 proceeds to... Figure 2B Block 148 in the middle.

[0079] At block 148, the remote 3D point cloud received at block 144 is transformed into the coordinate system of vehicle 12. This transformation process may consist of translation and / or rotation operations, such that the remote 3D point cloud is transformed from the perspective of the producer remote system to the perspective of vehicle 12. In an exemplary embodiment, the remote 3D point cloud is transformed such that the origin of the remote 3D point cloud is the same as the origin of the initial 3D point cloud generated at block 102. In another exemplary embodiment, the remote 3D point cloud may be further adjusted to compensate for the speed of the producer remote system supplying the remote 3D point cloud. The transformation result at block 148 is the transformed remote 3D point cloud. After block 148, method 100 proceeds to block 150.

[0080] At block 150, multiple overlapping points are identified in the transformed remote 3D point cloud. Overlapping points are points expected to have the same x, y, and z coordinates as reference points in the initial 3D point cloud generated at block 102. For example, a set of reference points contained within a 1m x 1m x 1m cube centered at x=2, y=3, z=1 in the initial 3D point cloud is expected to match a set of overlapping points at the same location in the transformed remote 3D point cloud, assuming data for that location exists in both the initial and transformed remote 3D point clouds. After identifying overlapping points at block 150, method 100 proceeds to block 152.

[0081] At block 152, overlapping points in the transformed remote 3D point cloud are compared with reference points in the initial 3D point cloud. If the percentage of overlapping points with the same x, y, and z coordinates as the reference points is less than a predetermined validity threshold, method 100 returns to the second sub-procedure 138. If the percentage of overlapping points with the same x, y, and z coordinates as the reference points is greater than or equal to the predetermined validity threshold, method 100 proceeds to block 154.

[0082] Blocks 150 and 152 perform validity checks on the received remote 3D point cloud. The validity check prevents data from faulty or malicious remote systems from being used in method 100. If invalid data is repeatedly received from a particular remote system, controller 14 can configure vehicle communication system 34 to prevent future connections to that particular remote system.

[0083] At block 154, the transformed remote 3D point cloud is combined with the initial 3D point cloud to generate a merged 3D point cloud. To combine the 3D point clouds, the transformed remote 3D point cloud is stacked on top of the initial 3D point cloud, such that the merged 3D point cloud contains all points from both the initial 3D point cloud and the transformed remote 3D point cloud. After block 154, method 100 proceeds to blocks 156 and 158.

[0084] At block 156, the merged 3D point cloud is analyzed, and actions are taken based on the analysis results. In an exemplary embodiment, object detection is performed using the merged 3D point cloud to identify objects in environment 24. For example, vehicle 12 can use data provided by remote vehicle 42 from remote non-drivable unit 44d to identify obstacles located in blind spot unit 38d. Various actions can be taken based on object detection, such as changing the route of vehicle 12, notifying the driver of vehicle 12, and / or performing accident avoidance operations (i.e., automatic emergency braking).

[0085] At block 158, HMI 18 is used to visualize the merged 3D point cloud to the occupants of vehicle 12. In an exemplary embodiment, vehicle 12 is a fully autonomous vehicle, and the visualization enhances occupant comfort by visualizing the vehicle's awareness of environment 24. In another exemplary embodiment, vehicle 12 is driven by a driver, and the visualization assists the driver by increasing the driver's awareness of environment 24. After blocks 156 and 158, method 100 enters a standby state at block 160.

[0086] The systems and methods disclosed herein offer several advantages. These include allowing cooperative data sharing while reducing network traffic by sending only the necessary data. Network bandwidth is conserved by sharing only the data needed to fill blind spots, rather than entire frames. A more complete and accurate perception of the environment is achieved by merging the received shared data with data collected by the vehicle, improving occupant safety.

[0087] The description in this disclosure is merely exemplary in nature, and variations thereof that do not depart from the spirit and scope of this disclosure are intended to be within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A system for sharing sensor data between a vehicle and multiple remote systems, the system comprising: Vehicle communication system; 3D sensor; as well as A controller that communicates electrically with the vehicle communication system and the three-dimensional sensor, wherein the controller is programmed to: The three-dimensional sensor is used to generate an initial three-dimensional point cloud of the environment surrounding the vehicle, wherein each point in the initial three-dimensional point cloud represents the position of a surface in the environment and has x, y, and z coordinates relative to the vehicle. An occupancy mesh map is generated based on the initial 3D point cloud, which includes the cells of the occupancy mesh map that are classified as first blind spots; Select a producer remote system to provide data for the cells in the occupancy grid map that are classified as the first blind spot; The vehicle communication system is used to send a data request to the producer remote system, the data request being based at least on the cells in the occupancy grid map that are classified as the first blind spot; Data is received from the producer's remote system, including information about the cells in the occupancy grid that are classified as the first blind spot; A merged 3D point cloud is generated by merging the data received from the producer's remote system and the initial 3D point cloud. as well as The merged 3D point cloud is used to identify objects in the environment surrounding the vehicle. In order to generate the occupancy grid map, the controller is further programmed to: Each point in the initial 3D point cloud is classified, at least in part, as a ground point or an object point based on the x, y, and z coordinates of each point. The environment surrounding the vehicle is divided into a plurality of 2D meshes, each of which is defined by a set of x and y coordinates relative to the vehicle; Determine the number of ground points with x and y coordinates within the set of x and y coordinates for each of the plurality of cells in the 2D mesh; Determine the number of object points in each of the plurality of cells in the 2D mesh, having x and y coordinates within the set of x and y coordinates; Each of the plurality of cells in the 2D mesh is classified as a drivable cell, a non-drivable cell, or a blind spot cell, based at least in part on the number of ground points and object points in each cell having x and y coordinates within the set of x and y coordinates. as well as The occupancy grid is defined as the 2D grid of the plurality of cells.

2. The system of claim 1, wherein, in order to select the producer remote system, the controller is further programmed to: The vehicle communication system receives multiple remote occupancy grid maps from the multiple remote systems, each of the multiple remote occupancy grid maps comprising a 2D grid of multiple remote units, each of the multiple remote units being classified as at least one of a remote drivable unit, a remote non-drivable unit, and a remote blind spot unit; The vehicle communication system is used to receive multiple geographical locations corresponding to each of the plurality of remote systems; For the first blind spot cell of the occupancy grid, multiple potential producer remote systems are identified using information about the first blind spot cell of the occupancy grid, wherein the multiple potential producer remote systems are a subset of the multiple remote systems; as well as The producer remote system is selected from the plurality of potential producer remote systems to provide data for the first blind spot cell of the occupied grid graph.

3. The system of claim 2, further comprising a Global Navigation Satellite System (GNSS) in electrical communication with the controller, wherein, in order to identify the plurality of potential producer remote systems, the controller is further programmed to: The geographical location of the vehicle is determined using a global navigation satellite system; The location of the first blind spot cell in the occupancy grid is determined using the vehicle's geographic location and the set of x and y coordinates that define the first blind spot cell in the occupancy grid. Based on the geographic location of each of the plurality of remote systems and the set of x and y coordinates defined for each of the plurality of remote units, the locations of a plurality of remote units are determined for each remote drivable unit in each remote occupancy grid and each remote non-drivable unit in each remote occupancy grid. as well as Identify the plurality of potential producer remote systems, wherein the plurality of potential producer remote systems comprises a subset of the plurality of remote systems, wherein the remote occupancy grid map of each of the plurality of potential producer remote systems includes at least one remotely drivable unit or a remotely non-drivable unit, and wherein the remote unit position of the at least one remotely drivable unit or remotely non-drivable unit is the same as the unit position of the first blind spot unit of the occupancy grid map.

4. The system of claim 2, wherein, in order to send the data request to the producer remote system, the controller is further programmed to: The producer remote system requests data from at least one remotely drivable or remotely non-drivable unit of the remote occupation grid map.

5. The system of claim 2, wherein, in order to receive data from the producer remote system, the controller is further programmed to: Receive remote 3D point cloud of at least one remotely drivable or remotely non-drivable unit from the producer remote system remotely occupied grid map.

6. The system of claim 5, wherein, in order to generate the merged 3D point cloud, the controller is further programmed to: Calculate the time difference between the timestamp of the remote 3D point cloud and the timestamp of the initial 3D point cloud; The x, y, and z coordinates of the remote 3D point cloud are adjusted based on the speed of the producer remote system in response to determining that the time difference is less than a predetermined timeout threshold to compensate for the speed of the producer remote system. Transform the remote 3D point cloud so that the origin of the coordinate system of the remote 3D point cloud is the same as the origin of the coordinate system of the initial 3D point cloud. Based on multiple remote cell locations from the remote occupancy grid map of the producer's remote system and multiple cell locations from the vehicle's occupancy grid map, multiple overlapping points in the remote 3D point cloud are identified, and these multiple overlapping points are expected to have the same x, y, and z coordinates as multiple reference points in the initial 3D point cloud; Determine the percentage of the plurality of overlapping points, which have the same x, y, and z coordinates as the plurality of reference points in the initial 3D point cloud; as well as In response to determining that the percentage of the plurality of overlapping points having the same x, y, and z coordinates as the reference point in the initial 3D point cloud is greater than or equal to a predetermined validity threshold, the transformed remote 3D point cloud is combined with the initial 3D point cloud to generate the merged 3D point cloud.

7. The system of claim 2, wherein, in order to select the producer remote system, the controller is further programmed to: The producer remote system is selected from the plurality of potential producer remote systems based at least in part on the distance between the vehicle and each of the plurality of potential producer remote systems.

8. The system of claim 1, wherein the controller is further programmed to: The vehicle communication system is used to continuously broadcast the occupancy grid map.

9. The system of claim 1, wherein the controller is further programmed to: In response to a remote data request from one of the plurality of remote systems, the vehicle communication system is used to transmit a 3D point cloud to the remote system.

10. The system of claim 1, further comprising a human-machine interface (HMI), wherein the controller is further programmed to: The human-machine interface is used to provide the occupants of the vehicle with a visual representation of the merged 3D point cloud.

11. A method for sharing sensor data between a vehicle and multiple remote systems, the method comprising: An initial 3D point cloud of the vehicle’s surrounding environment is generated using a 3D sensor, wherein each point in the initial 3D point cloud has x, y, and z coordinates relative to the vehicle. An occupied mesh map is generated based on the initial 3D point cloud, including the first blind spot. Select a producer remote system to provide data for the first blind spot in the occupancy grid; At least in part based on the first blind spot in the occupancy grid, a data request is sent to the producer remote system using the vehicle communication system; Data is received from the producer's remote system, including information about the first blind spot in the occupancy grid diagram; A merged 3D point cloud is generated by merging the data received from the producer's remote system and the initial 3D point cloud. as well as The merged 3D point cloud is used to identify objects in the environment surrounding the vehicle. Generating the occupancy grid map further includes: Each point in the initial 3D point cloud is classified as a ground point or an object point, at least in part, based on the x, y, and z coordinates of each point. The environment surrounding the vehicle is divided into a plurality of 2D meshes, each of which is defined by a set of x and y coordinates relative to the vehicle; Determine the number of ground points with x and y coordinates within the set of x and y coordinates for each of the plurality of cells in the 2D mesh; Determine the number of object points in each of the plurality of cells in the 2D mesh, having x and y coordinates within the set of x and y coordinates; Each cell in the 2D mesh is classified as a drivable cell, a non-drivable cell, or a blind spot cell, based at least in part on the number of ground points and object points having x and y coordinates within the set of x and y coordinates in each of the plurality of cells in the 2D mesh; and The occupancy grid is defined as the 2D grid of the plurality of cells.

12. The method of claim 11, wherein classifying each of the plurality of cells in the 2D mesh into a drivable cell, a non-drivable cell, or a blind spot cell further comprises: The number of ground points in each of the plurality of units is compared with the number of object points in each of the plurality of units; In response to determining that the number of ground points in the first cell is greater than zero, and that the number of ground points in the first cell is greater than the number of object points in the first cell, the first cell among the plurality of cells in the 2D mesh is classified as a drivable cell; In response to determining that the number of object points in the second cell is greater than zero, and that the number of object points in the second cell is greater than or equal to the number of ground points in the second cell, the second cell among the plurality of cells in the 2D mesh is classified as a non-driveable cell; as well as In response to determining that the sum of the number of ground points and object points having x and y coordinates within the set of x and y coordinates defining the third unit is less than or equal to a predetermined blind spot threshold, the third unit among the plurality of units in the 2D mesh is classified as a blind spot unit.

13. The method of claim 11, wherein selecting the producer remote system further comprises: The vehicle communication system is used to receive multiple remote occupancy grid maps from the multiple remote systems. Each of the multiple remote occupancy grid maps includes a 2D grid of multiple remote units, each of which is classified as a remote drivable unit, a remote non-drivable unit, or a remote blind spot unit. The vehicle communication system is used to receive multiple geographical locations corresponding to each of the plurality of remote systems; Identify multiple potential producer remote systems using information about the first blind spot cell of the occupancy grid, wherein the multiple potential producer remote systems are a subset of the multiple remote systems; as well as The producer remote system is selected from the plurality of potential producer remote systems to provide data for the first blind spot cell of the occupancy grid map based at least in part on the distance between the vehicle and each of the plurality of potential producer remote systems.

14. The method of claim 13, wherein identifying the plurality of potential producer remote systems further comprises: The geographical location of the vehicle was determined using the Global Navigation Satellite System (GNSS). The location of the first blind spot cell in the occupancy grid is determined using the vehicle's geographic location and a set of x and y coordinates that define the first blind spot cell in the occupancy grid. Based on the geographical location of each of the multiple remote systems and a set of x and y coordinates for each of the multiple remote units, the locations of multiple remote units are determined for each remote drivable unit and each remote non-drivable unit in each of the multiple remote occupancy grids. as well as Identify the plurality of potential producer remote systems, wherein the plurality of potential producer remote systems comprises a subset of the plurality of remote systems, wherein the remote occupancy grid map of each of the plurality of potential producer remote systems includes at least one remotely drivable unit or a remotely non-drivable unit, wherein the remote unit position of the at least one remotely drivable unit or the remotely non-drivable unit is the same as the unit position of the first blind spot unit of the occupancy grid map.

15. The method of claim 11, wherein receiving data from the producer remote system further comprises: Receive remote 3D point cloud of at least one remotely drivable or remotely non-drivable unit from the producer remote system remotely occupied grid map.

16. The method of claim 15, wherein generating the merged 3D point cloud further comprises: Calculate the time difference between the timestamp of the remote 3D point cloud and the timestamp of the initial 3D point cloud; The x, y, and z coordinates of the remote 3D point cloud are adjusted based on the speed of the producer remote system in response to determining that the time difference is less than a predetermined timeout threshold to compensate for the speed of the producer remote system. Transform the remote 3D point cloud so that the origin of the coordinate system of the remote 3D point cloud is the same as the origin of the coordinate system of the initial 3D point cloud. Based on multiple remote cell locations from the remote occupancy grid map of the producer's remote system and multiple cell locations from the vehicle's occupancy grid map, multiple overlapping points in the remote 3D point cloud are identified, and these multiple overlapping points are expected to have the same x, y, and z coordinates as multiple reference points in the initial 3D point cloud; Determine the percentage of the plurality of overlapping points, which have the same x, y, and z coordinates as the plurality of reference points in the initial 3D point cloud; as well as In response to determining that the percentage of the plurality of overlapping points having the same x, y, and z coordinates as the reference point in the initial 3D point cloud is greater than or equal to a predetermined validity threshold, the transformed remote 3D point cloud is combined with the initial 3D point cloud to generate the merged 3D point cloud.

Images