Fusing sensor data from multiple vehicles

Abstract

A method for fusing data from one or more vehicles includes capturing sensor data using one or more vehicle sensors. The method can also include transmitting the sensor data to a remote system. The sensor data is transmitted as tokenized data based at least in part on one or more features identified in the sensor data. The method can also include receiving, using the remote system, the tokenized data from the one or more vehicles. The method can also include fusing, using the remote system, the tokenized data from the one or more vehicles. Fusing the tokenized data can also include identifying at least one of: important sensor types and important features, and transmitting at least one of: important sensor types and important features of the one or more vehicles.

Description

Technical Field

[0001] This disclosure relates to systems and methods for collecting, transmitting, and fusing sensor data from multiple vehicle sources. Background Technology

[0002] To enhance situational awareness and coordination capabilities, connected vehicles can be equipped with advanced sensor systems that collect and transmit data from various sources to assist in monitoring the environment and improving driving decisions. These sensor systems can include cameras, radar, LiDAR, ultrasonic sensors, and more. Connected vehicles can share data with nearby vehicles, infrastructure, and / or remote backend servers using wireless communication networks. Data from multiple vehicle sources can be combined or “fused” to provide a more complete view of the environment. For example, data fusion can allow a vehicle to use data from other accessible vehicles to perceive objects that might otherwise be obscured by environmental features. However, current data aggregation and fusion systems and methods require significant network bandwidth for transmitting sensor data between vehicles, infrastructure, and / or remote backend systems.

[0003] Therefore, while current data collection systems and methods have achieved their intended purpose, new and improved systems and methods are needed for fusing data from one or more vehicles. Summary of the Invention

[0004] According to several aspects, a method for fusing data from one or more vehicles is provided. The method may include capturing sensor data using one or more vehicle sensors. The method may also include transmitting the sensor data to a remote system. The sensor data is transmitted as tagged data, at least in part based on one or more features identified in the sensor data. The method may also include receiving tagged data from one or more vehicles using the remote system. The method may further include fusing the tagged data from one or more vehicles using the remote system.

[0005] In another aspect of this disclosure, transmitting sensor data may further include using a feature extraction algorithm to identify one or more features in the sensor data. Transmitting sensor data may also include labeling the sensor data at least in part based on one or more features.

[0006] In another aspect of this disclosure, labeling sensor data may also include using a data importance classification machine learning model to determine the importance level of the sensor data. Labeling sensor data may also include labeling the sensor data at least in part based on its importance level.

[0007] In another aspect of this disclosure, determining the importance level may also include using a data importance classification machine learning model to determine the importance level of sensor data based at least in part on at least one of the following: one or more features in the sensor data, the sensor type of the sensor data, and the road configuration in the sensor data.

[0008] In another aspect of this disclosure, labeling sensor data at least in part based on the importance level of the sensor data may also include labeling sensor data at least in part based on the importance level of the sensor data. The size of the labeled data varies directly with the importance level of the sensor data.

[0009] In another aspect of this disclosure, receiving tokenized data may further include generating de-tagged data by de-tagged the tokenized data using a database of known features.

[0010] In another aspect of this disclosure, the fused tokenized data may further include combining de-tagged data received from multiple vehicles to aggregate the combined de-tagged data. The fused tokenized data may also include identifying at least one of the following based at least in part on the combined de-tagged data: important sensor types and important features. The fused tokenized data may further include transmitting at least one of the following to one or more vehicles: important sensor types and important features.

[0011] In another aspect of this disclosure, combining de-tagged data may also include combining de-tagged data received from multiple vehicles using a trained data fusion machine learning model to fuse data captured from multiple vehicles under various environmental conditions.

[0012] In another aspect of this disclosure, identifying at least one of the following: important sensor types and important features based at least in part on combined delabeled data may further include: using a trained sensor and feature importance classification machine learning model to identify at least one of the following: important sensor types and important features based at least in part on combined delabeled data, thereby identifying important sensor types and important features based at least in part on combined delabeled data.

[0013] In another aspect of this disclosure, determining the importance level may also include using a data importance classification machine learning model, based at least in part on at least one of the following: important sensor types and important features.

[0014] According to several aspects, a system for fusing data from one or more vehicles is provided. The system may include one or more vehicle systems, each of which includes one or more vehicle sensors, a vehicle communication system, and a vehicle controller in electrical communication with the one or more vehicle sensors and the vehicle communication system. The vehicle controller is programmed to capture sensor data using the one or more vehicle sensors. The vehicle controller is also programmed to transmit the sensor data to a remote system using the vehicle communication system. The sensor data is transmitted as tagged data, at least in part based on one or more features identified in the sensor data.

[0015] In another aspect of this disclosure, the system also includes a remote system comprising a remote system communication system and a remote system controller in electrical communication with the remote system communication system. The remote system controller is programmed to receive tagged data from one or more vehicles using the remote system communication system. The remote system controller is also programmed to fuse tagged data from one or more vehicles.

[0016] In another aspect of this disclosure, for transmitting sensor data, the vehicle controller is also programmed to use a feature extraction algorithm to identify one or more features in the sensor data. For transmitting sensor data, the vehicle controller is also programmed to use a data importance classification machine learning model to determine the importance level of the sensor data based at least in part on at least one of the following: one or more features in the sensor data, the sensor type of the sensor data, and the road configuration in the sensor data. For transmitting sensor data, the vehicle controller is also programmed to label the sensor data at least in part based on the importance level of the sensor data. The size of the labeled data varies directly with the importance level of the sensor data.

[0017] In another aspect of this disclosure, to fuse tokenized data, the remote system controller is also programmed to generate de-tagged data by de-tagged the tokenized data using a database of known features. To fuse the tokenized data, the remote system controller is also programmed to combine de-tagged data received from multiple vehicles to aggregate the combined de-tagged data. To fuse the tokenized data, the remote system controller is also programmed to identify at least one of the following based at least in part on the combined de-tagged data: important sensor types and important features. To fuse the tokenized data, the remote system controller is also programmed to transmit at least one of the following to one or more vehicles: important sensor types and important features.

[0018] In another aspect of this disclosure, in order to combine de-tagged data, the remote system controller is also programmed to use a trained data fusion machine learning model to combine de-tagged data received from multiple vehicles to fuse data captured from multiple vehicles under various environmental conditions.

[0019] In another aspect of this disclosure, in order to identify at least one of the following: important sensor types and important features based at least in part on combined delabeled data, the remote system controller is also programmed to use a trained sensor and feature importance classification machine learning model to identify at least one of the following: important sensor types and important features based at least in part on combined delabeled data, so as to identify important sensor types and important features based at least in part on combined delabeled data.

[0020] In another aspect of this disclosure, to determine the importance level, the vehicle controller is also programmed to receive at least one of the following from a remote system using a vehicle communication system: important sensor types and important features. To determine the importance level, the vehicle controller is also programmed to use a data importance classification machine learning model to determine the importance level of the sensor data based at least in part on at least one of the following: important sensor types and important features.

[0021] According to several aspects, a method for fusing data from one or more vehicles is provided. The method may include capturing sensor data using one or more vehicle sensors. The method may also include identifying one or more features in the sensor data using a feature extraction algorithm. The method may further include determining the importance level of the sensor data using a data importance classification machine learning model, based at least in part on at least one of the following: one or more features in the sensor data, the sensor type of the sensor data, and road configuration in the sensor data, and at least in part on at least one of the following: important sensor types and important features received from a remote system. The method may further include generating tokenized data by tokenizing the sensor data at least in part based on the importance level of the sensor data. The method may further include transmitting the tokenized data to a remote system. The method may further include using the tokenized data, based at least in part on the remote system, to identify at least one of the following: important sensor types and important features. The method may further include using the remote system to transmit at least one of the following to one or more vehicles: important sensor types and important features.

[0022] In another aspect of this disclosure, generating tokenized data may further include tokenizing the sensor data at least in part based on the importance level of the sensor data. The size of the tokenized data varies directly with the importance level of the sensor data.

[0023] In another aspect of this disclosure, identifying at least one of the following: important sensor types and important features may further include generating de-labeled data by de-labeling the labeled data using a database of known features. Identifying at least one of the following: important sensor types and important features may further include combining de-labeled data received from multiple vehicles to aggregate the combined de-labeled data. Identifying at least one of the following: important sensor types and important features may further include identifying at least one of the following: important sensor types and important features using a trained sensor and feature importance classification machine learning model, to identify the important sensor types and important features at least in part based on the combined de-labeled data.

[0024] Further areas of application will become apparent from the description provided herein. It should be understood that these descriptions and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0025] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0026] Figure 1 This is a schematic diagram of a system for fusing data from one or more vehicles according to an exemplary embodiment; and

[0027] Figure 2 This is a flowchart of a method for fusing data from one or more vehicles according to an exemplary embodiment. Detailed Implementation

[0028] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses.

[0029] In various aspects of this disclosure, it is advantageous to aggregate and fuse sensor data from multiple nearby vehicles in an environment to gain a more complete understanding of the situation. However, the transmission of sensor data can lead to intensive use of resources (e.g., transmission bandwidth and / or processing power). Therefore, this disclosure provides a new and improved system and method for efficiently fusing data from one or more vehicles.

[0030] refer to Figure 1 The diagram illustrates a system 10 for fusing data from one or more vehicles. System 10 includes one or more vehicles 12, each of which includes a vehicle system 14. System 10 also includes a remote system 16.

[0031] The vehicle system 14 includes a vehicle controller 18, one or more vehicle sensors 20, and a vehicle communication system 22.

[0032] Vehicle controller 18 is used to implement method 100 for fusing data from one or more vehicles, as described below. Vehicle controller 18 includes at least one processor and a non-transitory computer-readable storage device or medium. The processor may be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among a plurality of processors associated with vehicle controller 18, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions.

[0033] Computer-readable storage devices or media may include volatile and non-volatile storage such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory used to store various operational variables when the processor is powered off. Computer-readable storage devices or media may be implemented using a variety of storage devices, such as programmable read-only memory (PROM), electrical PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, or other electrical, magnetic, optical, or combined storage devices capable of storing data, some of which represents executable instructions used by vehicle controller 18 to control various systems of one or more vehicles 12.

[0034] The vehicle controller 18 may also include multiple controllers that are electrically communicating with each other. The vehicle controller 18 may interconnect with additional systems and / or controllers of one or more vehicles 12, allowing the vehicle controller 18 to access data such as the speed, acceleration, braking, and steering angle of one or more vehicles 12.

[0035] The vehicle controller 18 communicates electrically with one or more vehicle sensors 20 and a vehicle communication system 22. In one exemplary embodiment, electrical communication is established using, for example, a CAN network, a FLEXRAY network, a local area network (e.g., WiFi, Ethernet, etc.), or a Serial Peripheral Interface (SPI) network. It should be understood that various additional wired and wireless technologies and communication protocols used for communicating with the vehicle controller 18 are within the scope of this disclosure.

[0036] One or more vehicle sensors 20 are used to acquire data about the environment surrounding one or more vehicles 12. Within the scope of this disclosure, telemetry data includes, for example, engine RPM, vehicle speed, fuel level, engine temperature, odometer reading, battery voltage, braking system status, transmission data, tire pressure, GNSS location, acceleration and deceleration, steering angle, suspension system data, exhaust emission levels, diagnostic fault codes (DTCs), airbag status, windshield wiper status, lights and indicator lights, and cruise control status. In one exemplary embodiment, one or more vehicle sensors 20 include sensors for determining performance data about one or more vehicles 12. In a non-limiting example, one or more vehicle sensors 20 further include at least one of the following: an electric motor speed sensor, an electric motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a brake position sensor, a coolant temperature sensor, a cooling fan speed sensor, wheel speed sensors, and a transmission fluid temperature sensor.

[0037] In another exemplary embodiment, the one or more vehicle sensors 20 may further include sensors for determining information about the environment within the one or more vehicles 12. In a non-limiting example, the one or more vehicle sensors 20 may also include at least one of a seat occupancy sensor, a passenger compartment air temperature sensor, a passenger compartment motion detection sensor, a passenger compartment camera, a passenger compartment microphone, etc.

[0038] In another exemplary embodiment, the one or more vehicle sensors 20 may further include sensors for determining information about the environment surrounding the one or more vehicles 12. In a non-limiting example, the one or more vehicle sensors 20 may further include at least one of the following: an ambient air temperature sensor, a barometric pressure sensor, a global navigation satellite system (GNSS), and / or a photographic and / or video camera positioned to observe the environment in front of the one or more vehicles 12.

[0039] GNSS is used to determine the geographic location of one or more vehicles 12. In one exemplary embodiment, the GNSS is a Global Positioning System (GPS). In a non-limiting example, the GPS includes a GPS receiver antenna (not shown) and a GPS controller (not shown) in electrical communication with the GPS receiver antenna. The GPS receiver antenna receives signals from multiple satellites, and the GPS controller calculates the geographic location of one or more vehicles 12 based on the signals received by the GPS receiver antenna. In one exemplary embodiment, the GNSS also includes a map. The map includes information about infrastructure, such as municipal boundaries, roads, railways, sidewalks, buildings, etc. Therefore, the map information is used to contextualize the geographic location of one or more vehicles 12. In one non-limiting example, the map is retrieved from a remote source using a wireless connection. In another non-limiting example, the map is stored in a GNSS database. It should be understood that various additional types of satellite-based radio navigation systems, such as Global Positioning System (GPS), Galileo, GLONASS, and BeiDou Navigation Satellite System (BDS), are within the scope of this disclosure. It should be understood that, without departing from the scope of this disclosure, the GNSS may be integrated with the vehicle controller 18 (e.g., on the same circuit board as the vehicle controller 18 or otherwise as part of the vehicle controller 18).

[0040] In another exemplary embodiment, at least one of the one or more vehicle sensors 20 is a sensing sensor capable of sensing objects and / or measuring distances in the environment surrounding one or more vehicles 12. In a non-limiting example, the one or more vehicle sensors 20 include a stereo camera with distance measurement capability. In one example, at least one of the one or more vehicle sensors 20 is fixed inside one or more vehicles 12, for example, fixed to the inner roof of one or more vehicles 12, having a view through the windshield of one or more vehicles 12. In another example, at least one of the one or more vehicle sensors 20 is fixed outside one or more vehicles 12, for example, fixed to the roof of one or more vehicles 12, having a view of the environment surrounding one or more vehicles 12. It should be understood that various additional types of sensing sensors, such as LiDAR sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors, are all within the scope of this disclosure. As described above, the one or more vehicle sensors 20 are in electrical communication with the vehicle controller 18.

[0041] Vehicle controller 18 uses vehicle communication system 22 to communicate with other systems outside one or more vehicles 12. For example, vehicle communication system 22 includes the ability to communicate with vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems at remote call centers (e.g., General Motors’ ON-STAR), and / or personal communication devices. Generally, the term vehicle-to-everything communication (“V2X” communication) refers to communication between one or more vehicles 12 and any remote system (e.g., vehicles, infrastructure, and / or remote systems). In some embodiments, vehicle communication system 22 is a wireless communication system configured to communicate using the IEEE 802.11 standard or via a wireless local area network (WLAN) using cellular data communication (e.g., using GSMA standards such as SGP.02, SGP.22, SGP.32, etc.). Therefore, vehicle communication system 22 may also include an embedded universal integrated circuit card (eUICC) configured to store at least one cellular connectivity configuration profile, such as an embedded subscriber identity module (eSIM) profile.

[0042] The vehicle communication system 22 is also configured to communicate via a personal area network (e.g., Bluetooth) and / or near field communication (NFC). However, additional or alternative communication methods, such as Dedicated Short Range Communication (DSRC) channels and / or mobile telecommunications protocols based on 3GPP standards, are also considered within the scope of this disclosure. A DSRC channel refers to a unidirectional or bidirectional short- to medium-range wireless communication channel designed specifically for automotive use, along with a set of corresponding protocols and standards. 3GPP refers to a partnership among multiple standards organizations that develop mobile telecommunications protocols and standards. 3GPP standards are structured as “releases.” Therefore, communication methods based on 3GPP releases 14, 15, 16, and / or future 3GPP releases are considered within the scope of this disclosure. Consequently, the vehicle communication system 22 may include one or more antennas and / or communication transceivers for receiving and / or transmitting signals, such as Cooperative Sensing Messages (CSM).

[0043] The vehicle communication system 22 is configured to wirelessly transmit information between one or more vehicles 12 and another vehicle. Furthermore, the vehicle communication system 22 is configured to wirelessly transmit information between one or more vehicles 12 and infrastructure or other vehicles. It should be understood that, without departing from the scope of this disclosure, the vehicle communication system 22 may be integrated with the vehicle controller 18 (e.g., on the same circuit board as the vehicle controller 18 or otherwise as part of the vehicle controller 18).

[0044] Continue to refer to Figure 1The remote system 16 includes a remote system controller 26a that is electrically in communication with a remote system database 28 and a remote system communication system 30. In a non-limiting example, the remote system 16 is located in a server farm, data center, etc., and is connected to the Internet using the remote system communication system 30. The remote system controller 26a includes at least one remote system processor 26b and a remote system non-transitory computer-readable storage device or remote system medium 26c. The description of the type and configuration given above for the vehicle controller 18 also applies to the remote system controller 26a. In some examples, the remote system controller 26a differs from the vehicle controller 18 in that it can have higher processing speeds, include more memory, include more inputs / outputs, etc. In a non-limiting example, the remote system processor 26b and remote system medium 26c of the remote system controller 26a are structurally and / or functionally similar to the processor and medium of the vehicle controller 18, as described above.

[0045] The remote system database 28 is used to store data received from one or more vehicles 12, as will be discussed in more detail below. The remote system communication system 30 is used to communicate with external systems (e.g., vehicle controller 18) via the vehicle communication system 22. In a non-limiting example, the remote system communication system 30 is structurally and / or functionally similar to the vehicle communication system 22 of vehicle system 14, as described above. In some examples, the remote system communication system 30 differs from the vehicle communication system 22 in that it is capable of higher power signal transmission, more sensitive signal reception, higher bandwidth transmission, additional transmission / reception protocols, etc.

[0046] refer to Figure 2 A flowchart of a method 100 for fusing data from one or more vehicles is shown. Method 100 begins at block 102 and proceeds to block 104. At block 104, a vehicle controller 18 of a vehicle system 14 of each of the one or more vehicles 12 uses one or more vehicle sensors 20 to capture sensor data of the environment surrounding each of the one or more vehicles 12. In a non-limiting example, the sensor data includes camera data of the environment. In another non-limiting example, the sensor data includes distance data, such as LiDAR data, radar data, etc. In an exemplary embodiment, after capturing sensor data using one or more vehicle sensors 20, the vehicle controller 18 fuses the sensor data captured by the multiple sensors to form a detailed view of the environment. In a non-limiting example, the detailed view may include camera image data including one or more objects (e.g., other vehicles) and distance data aligned with the camera image data providing distances to the one or more objects. After block 104, method 100 proceeds to block 106.

[0047] At box 106, the vehicle controller 18 uses a feature extraction algorithm to identify one or more features in the sensor data. Within the scope of this disclosure, features include vehicles, pedestrians, traffic control devices (e.g., street signs, road markings, traffic lights, etc.), road edges, structures, environmental terrain, etc. Within the scope of this disclosure, the feature extraction algorithm is a software algorithm configured to identify features in the sensor data.

[0048] In a non-limiting example, the feature extraction algorithm first preprocesses the sensor data to enhance relevant patterns while reducing noise. For example, if the sensor data comes from a LiDAR sensor, the feature extraction algorithm might filter out irrelevant points, such as those from sensor noise. In a non-limiting example, the feature extraction algorithm includes a data segmentation module, a feature descriptor module, and a feature classification module. The data segmentation module divides the sensor data into regions of interest, often called "clusters," based on features such as spatial proximity or intensity. For example, data points corresponding to nearby vehicles can be grouped together based on their spatial alignment and surface contours. The feature descriptor module then analyzes each cluster to generate unique features such as object orientation, velocity, position, size, shape, or texture.

[0049] Once segmented and described, the feature classification module is used to extract features. For example, the feature classification module uses rule-based algorithms and / or machine learning algorithms to distinguish pedestrians, vehicles, and other objects based on patterns in the sensor data cluster. After box 106, method 100 proceeds to box 108.

[0050] At box 108, the vehicle controller 18 uses a data importance classification machine learning model to determine the importance level of the sensor data. Within the scope of this disclosure, the importance level indicates the criticality of the sensor data to the driving task. For example, sensor data about objects relatively far from one or more vehicles 12 has a relatively low importance level. Sensor data about objects relatively close to one or more vehicles 12 has a relatively high importance level. In another example, sensor data about stationary objects has a relatively low importance level. Sensor data about moving objects has a relatively high importance level. In another example, sensor data about objects moving away from one or more vehicles 12 has a relatively low importance level. Sensor data about objects moving towards one or more vehicles 12 has a relatively high importance level.

[0051] In one exemplary embodiment, the importance level is determined at least in part based on at least one of the following importance indicators: one or more features identified in the sensor data at block 106, the sensor type of the sensor data (i.e., camera, LiDAR, radar, etc.), the road configuration in the sensor data (i.e., road type / location, intersection type, number of lanes, etc.), events occurring in the environment (e.g., emergency braking events, road construction, etc.), and / or the driving style of surrounding vehicles (i.e., based on speed, following distance, etc.). In one exemplary embodiment, road configuration, events, and driving style are identified using a rule-based algorithm or a machine learning-based algorithm based on one or more features identified in the sensor data at block 106. In one exemplary embodiment, the importance indicator also includes important sensor types and / or important features determined by the remote system 16, as will be discussed in more detail below.

[0052] In a non-limiting example, the data importance classification machine learning model comprises multiple layers, including an input layer and an output layer, as well as one or more hidden layers. The input layer receives sensor data captured at box 104 and importance indicators as input. The input is then passed to the hidden layers. Each hidden layer applies a transformation (e.g., a non-linear transformation) to the data and passes the result to the next hidden layer, until the last hidden layer. The output layer produces the importance levels of the sensor data.

[0053] To train a data importance classification machine learning model, an input dataset and the importance levels of its corresponding sensor data are used. The algorithm is trained by adjusting the internal weights between nodes in each hidden layer to minimize prediction error. During training, optimization techniques (such as gradient descent) are used to adjust the internal weights to reduce prediction error. The training process is repeated on the entire dataset until the prediction error is minimized, and then the resulting trained model is used to classify new input data.

[0054] After fully training the machine learning model for data importance classification, the algorithm is able to determine the importance level of the sensor data based on the sensor data and importance indicators captured at box 104. By adjusting the weights between nodes in each hidden layer during training, the algorithm "learns" to recognize patterns in the data that indicate importance levels. After box 108, method 100 proceeds to box 110.

[0055] At block 110, vehicle controller 18 generates tokenized data by tokenizing sensor data based on one or more features identified at block 106 and the importance level of the sensor data determined at block 108. Within the scope of this disclosure, tokenization is a process by which sensor data and one or more features are transformed into a simplified coded representation to ensure privacy and efficiency. In one exemplary embodiment, the tokenization process includes representing one or more features as one or more tags. Each of the one or more tags has multiple attributes, including, for example, a unique identifier (i.e., a unique code used to distinguish the tag), a tag type identifier (i.e., a code representing the feature identified by the tag, such as a vehicle, pedestrian, etc.), and a location (i.e., the coordinate location of the feature identified by the tag in the environment).

[0056] It should be understood that the tag may have additional attributes, such as the orientation of the tag-recognized feature, the speed of the tag-recognized feature, the boundary or edge of the tag-recognized feature, etc. In an exemplary embodiment, the tag attributes do not include identification information, such as vehicle color or license plate number.

[0057] Therefore, at block 110, the vehicle controller 18 generates tokenized data based on the sensor data captured at block 104 and one or more features identified at block 106. In one exemplary embodiment, the tokenized data is generated at least in part based on the importance level of the sensor data. In a non-limiting example, the size of the tokenized data is adjusted based on the importance level of the sensor data. For example, the size of the tokenized data can be increased by including more tags and / or more tag attributes for each tag. The size of the tokenized data can be reduced by including fewer tags and / or fewer tag attributes for each tag.

[0058] In one exemplary embodiment, the size of the tagged data varies directly with the importance level of the sensor data. In other words, sensor data with a higher importance level will be tagged such that the resulting tagged data is larger (i.e., includes more information) than the tagged data produced by tagging sensor data with a lower importance level. In any case, the size of the tagged data is smaller than the original size of the original sensor data captured at block 104. After block 110, method 100 proceeds to block 112.

[0059] At block 112, vehicle controller 18 uses vehicle communication system 22 to transmit tokenized data to remote system 16. In an exemplary embodiment, the tokenized data is further compressed before transmission (e.g., using an entropy-based compression algorithm). Furthermore, at block 112, remote system controller 26a uses remote system communication system 30 to receive the tokenized data. After reception, the tokenized data is decompressed and stored in remote system database 28. Following block 112, method 100 proceeds to block 114.

[0060] It should be understood that the above disclosure applies to each of the one or more vehicles 12, and the vehicle system 14 of each of the one or more vehicles 12 can perform the method steps of blocks 104, 106, 108, 110, and 112. Therefore, each of the one or more vehicles 12 transmits the tokenized data to the remote system 16.

[0061] At block 114, remote system controller 26a de-tagged the tokenized data received at block 112 to generate de-tagged data. In an exemplary embodiment, to de-tagged the tokenized data, remote system controller 26a uses a database of known features stored in remote system database 28. In a non-limiting example, remote system database 28 includes tag type identifier codes and a list of corresponding tag types (e.g., vehicles, pedestrians, etc.). Furthermore, based on tag attributes received with each tag (e.g., the location of the tag-identified feature, the orientation of the tag-identified feature, the speed of the tag-identified feature, the boundary or edge of the tag-identified feature, etc.), remote system controller 26a is able to create a reconstructed detailed view of the environment similar to the detailed view of the environment discussed above with reference to block 104.

[0062] In a non-limiting example, the reconstructed detailed view does not include personal or identifying information about features in the environment (e.g., vehicles, pedestrians, etc.). Furthermore, as mentioned above, the amount of detail in the reconstructed detailed view depends on the importance level of the sensor data upon which the reconstructed detailed view is based. Following box 114, method 100 proceeds to box 116.

[0063] At box 116, the remote system controller 26a combines de-tagged data received from multiple vehicles among one or more vehicles 12 to aggregate the combined de-tagged data. In one exemplary embodiment, the remote system controller 26a stitches together (i.e., fuses) the de-tagged data from multiple vehicles to provide a more complete view of the environment. In a non-limiting example, the remote system controller 26a fuses the de-tagged data based on common features in the tagged data from multiple vehicles. In one exemplary embodiment, the remote system controller 26a uses a data fusion machine learning model trained to fuse data captured from multiple vehicles under various environmental conditions. In a non-limiting example, the data fusion machine learning model is trained to fuse data including various road configurations (i.e., road type / location, intersection type, number of lanes, etc.).

[0064] In a non-limiting example, the data fusion machine learning model includes multiple layers, including an input layer and an output layer, and one or more hidden layers. The input layer receives de-labeled data received from multiple vehicles among one or more vehicles 12 as input. The input is then passed to the hidden layers. Each hidden layer applies a transformation (e.g., a non-linear transformation) to the data and passes the result to the next hidden layer, until the last hidden layer. The output layer produces the combined de-labeled data.

[0065] To train the data fusion machine learning model, a dataset of delabeled data containing the inputs and their corresponding combinations is used. The algorithm is trained by adjusting the internal weights between nodes in each hidden layer to minimize the prediction error. During training, optimization techniques (such as gradient descent) are used to adjust the internal weights to reduce the prediction error. The training process is repeated on the entire dataset until the prediction error is minimized, and then the resulting trained model is used to process new input data.

[0066] After sufficient training of the data fusion machine learning model, the algorithm is able to generate combined de-labeled data based on de-labeled data received from multiple vehicles among one or more vehicles 12, including various road configurations (i.e., road type / location, intersection type, number of lanes, etc.). In another exemplary embodiment, the data fusion machine learning model is generated by fine-tuning a pre-trained general or "backbone" machine learning model. Following box 116, method 100 proceeds to box 118.

[0067] At box 118, the remote system controller 26a identifies important sensor types and important feature types based on the combined de-labeled data. Within the scope of this disclosure, an important sensor type is one of one or more vehicle sensors 20 considered most important for collecting relevant information about the environment. For example, in an environment including many other vehicles, a ranging sensor (e.g., a LiDAR sensor) might be an important sensor type because it allows for determining clearances to other vehicles. In another example, in an environment with complex road configurations (e.g., numerous traffic signals, numerous driving lanes, complex intersections, etc.), a perception sensor (e.g., a camera) might be an important sensor type because it allows for the identification of lane markings, traffic light phase states, etc.

[0068] Within the scope of this disclosure, important feature types are those considered most important for gathering relevant information about the environment (e.g., vehicles, pedestrians, structures, traffic control infrastructure, etc.). For example, in an environment that includes many other vehicles, the vehicle feature type might be an important feature type. In another example, in an environment with complex road configurations (e.g., many traffic signals, many driving lanes, complex intersections, etc.), the traffic control infrastructure feature type might be an important feature type.

[0069] In an exemplary embodiment, in order to determine important sensor types and important feature types, the remote system controller 26a uses a trained sensor and feature importance classification machine learning model to identify important sensor types and important features based at least in part on combined de-labeled data.

[0070] In a non-limiting example, the sensor and feature importance classification machine learning model comprises multiple layers, including an input layer and an output layer, and one or more hidden layers. The input layer receives combined de-labeled data as input. The input is then passed to the hidden layers. Each hidden layer applies a transformation (e.g., a non-linear transformation) to the data and passes the result to the next hidden layer, until the last hidden layer. The output layer produces the important sensor types and important features.

[0071] To train a machine learning model for sensor and feature importance classification, an input dataset and its corresponding important sensor types and important features are used. The algorithm is trained by adjusting the internal weights between nodes in each hidden layer to minimize prediction error. During training, optimization techniques (such as gradient descent) are used to adjust the internal weights to reduce prediction error. The training process is repeated on the entire dataset until the prediction error is minimized, and then the resulting trained model is used to process new input data.

[0072] After sufficient training of the sensor and feature importance classification machine learning model, the algorithm is able to determine important sensor types and important features based on combined delabeled data. By adjusting the weights between nodes in each hidden layer during training, the algorithm "learns" to recognize patterns in the combined delabeled data that indicate important sensor types and important features. In another exemplary embodiment, the sensor and feature importance classification machine learning model is generated by fine-tuning a pre-trained general or "backbone" machine learning model.

[0073] In one non-limiting example, a single important sensor type and important feature are determined for a specific geographic area (e.g., an area of ​​one square kilometer). In another non-limiting example, important sensor types and important features are determined individually for each of one or more vehicles 12. In one non-limiting example, important sensor types and important features are applied to a limited predetermined time range (e.g., one minute) before being re-evaluated based on changed environmental conditions.

[0074] Furthermore, at box 118, the remote system communication system 30 transmits important sensor types and important features to each of the one or more vehicles 12. As discussed above with reference to box 108, important sensor types and important features are used as importance indicators to determine the importance level of the sensor data collected by the one or more vehicles 12. In another example, the vehicle controller 18 of the one or more vehicles 12 uses important sensor types and / or important features to adjust the path planning algorithm of the autonomous driving system or advanced driver assistance system (ADAS) of the one or more vehicles 12. In another example, one or more tags from each of the one or more vehicles 12 are assigned to each of the one or more vehicles 12 (directly point-to-point or relayed via remote system 16) so that each of the one or more vehicles 12 can de-tag and create a reconstructed detailed view of the environment. After box 118, method 100 proceeds to the standby state at box 120.

[0075] In one exemplary embodiment, method 100 repeatedly exits standby state 120 and restarts method 100 at block 102. In a non-limiting example, the method exits standby state 120 and restarts on a timer, for example, every three hundred milliseconds.

[0076] The system 10 and method 100 of this disclosure offer several advantages. The tokenization process using system 10 and method 100 reduces the bandwidth required for data transmission between one or more vehicles 12 and the remote system 16, and protects the privacy of other road users. Furthermore, by identifying important sensor types and key features and transmitting that information to one or more vehicles 12, the data acquisition, tokenization, and transmission processes are further optimized to focus on the most relevant data.

[0077] The descriptions in this disclosure are merely exemplary in nature, and variations that do not depart from the spirit of this disclosure are intended to fall within its scope. These variations should not be considered as departing from the scheme and scope of this disclosure.

Claims

1. A method for fusing data from one or more vehicles, the method comprising: Use one or more vehicle sensors to capture sensor data; The sensor data is transmitted to a remote system, wherein the sensor data is transmitted as a tagged data based at least in part on one or more features identified in the sensor data; The remote system is used to receive the tagged data from the one or more vehicles; as well as The remote system is used to fuse the tokenized data from the one or more vehicles.

2. The method according to claim 1, wherein, Transmitting the sensor data also includes: Use a feature extraction algorithm to identify one or more features in the sensor data; and The sensor data is labeled at least in part based on one or more of the aforementioned features.

3. The method according to claim 2, wherein, The labeling of the sensor data also includes: The importance level of the sensor data is determined using a data importance classification machine learning model; and The sensor data is labeled at least in part based on the importance level of the sensor data.

4. The method according to claim 3, wherein, Determining the importance level also includes: The importance level of the sensor data is determined using the data importance classification machine learning model, based at least in part on at least one of the following: one or more features in the sensor data, the sensor type of the sensor data, and the road configuration in the sensor data.

5. The method according to claim 3, wherein, Labeling the sensor data based at least in part on the importance level of the sensor data also includes: The sensor data is labeled at least in part based on the importance level of the sensor data, wherein the size of the labeled data varies directly with the importance level of the sensor data.

6. The method according to claim 3, wherein, Receiving the tokenized data further includes: De-tagged data is generated by de-tagged data using a database with known features.

7. The method according to claim 6, wherein, The fusion of the tokenized data also includes: The de-tagged data received from multiple vehicles is combined to aggregate the combined de-tagged data; Identify at least one of the following, at least in part, based on the de-labeled data of the combination: important sensor types and important features; and Transmit at least one of the following to the one or more vehicles: the important sensor type and the important feature.

8. The method according to claim 7, wherein, The combination of the de-tagged data also includes: The trained data fusion machine learning model combines the de-labeled data received from multiple vehicles to fuse data captured from multiple vehicles under various environmental conditions.

9. The method according to claim 7, wherein, Identifying at least one of the following, at least in part, based on the de-labeled data of the combination: the important sensor type and the important feature further include: Using a trained sensor and feature importance classification machine learning model, at least one of the following is identified, at least in part, based on the combined delabeled data: the important sensor type and the important feature, to identify the important sensor type and the important feature, at least in part, based on the combined delabeled data.

10. The method according to claim 7, wherein, Determining the importance level also includes: The importance level of the sensor data is determined using the data importance classification machine learning model, based at least in part on at least one of the following: the important sensor type and the important feature.

Images