Attention driven streaming system

Abstract

An attention-driven streaming system includes an adaptive streaming module and a transceiver. The adaptive streaming module includes a filter, a compression module, an attention-driven policy module, and a fusion module. The filter filters sensor data received from sensors of a vehicle. The compression module compresses the filtered sensor data to generate compressed data. The attention-driven policy module generates feedforward information to adjust regions of interest based on a state of the vehicle and a state of an environment of the vehicle. The fusion module generates an adaptive streaming policy to adaptively adjust operation of each of the filters. The transceiver streams the compressed data to at least one of an edge computing device or a cloud-based network device and receives feedback information and pipeline monitoring information in response. The fusion module generates the adaptive streaming policy based on the feedforward information, the feedback information, and the pipeline monitoring information.

Description

[0001] introduction

[0002] The information provided in this section is for the purpose of presenting the overall context of this disclosure. The work of the currently attributed inventors, to the extent described in this section, and in all respects of that description which at the time of filing may not be regarded as prior art, is neither expressly nor implicitly considered prior art to this disclosure. Technical Field

[0003] This disclosure relates to edge computing and cloud-based processing of vehicle sensor data. Background Technology

[0004] An ego-vehicle can include various sensors used to detect objects and environmental conditions, such as cameras, radar sensors, lidar sensors, speed sensors, yaw rate sensors, etc. An ego-vehicle refers to a vehicle on which sensors are located and where processing of at least some sensor data occurs. Significant processing power is required to receive, process, and analyze the sensor data. To reduce the processing power and processing time required at the vehicle, real-time mission-critical sensor data can be offloaded to edge computing devices or cloud-based network devices. This is done to utilize the abundant resources (computing and storage resources) at edge computing devices and / or cloud-based network devices. The results of the processed and analyzed data can then be sent back to the vehicle. Real-time mission-critical sensor data can refer to data associated with, for example, the detection of an oncoming object (or vehicle) and requiring rapid processing to avoid a collision.

[0005] Cloud-based networks can provide computing and storage services from centralized locations. Edge computing, using, for example, fifth-generation (5G) broadband cellular networks, allows sensor data processing to be pushed to the edge of the vehicle's network. Sensor data can be processed at micro data centers deployed at cellular towers and / or regional stations, which are much closer to the vehicle than cloud-based network equipment.

[0006] Some vehicle functions can also be offloaded from the vehicle to cloud-based network devices and / or edge computing devices (referred to as teleprocessing devices). This can include streaming sensor data to the teleprocessing device and processing the sensor data according to the vehicle function. Vehicle functions may include object detection, object tracking, and position and mapping of the vehicle and surrounding objects. The teleprocessing device can perform these functions based on the received sensor data and can provide the results of the analysis back to the vehicle. The results can then be used to perform onboard vehicle functions to enhance onboard operation and improve vehicle performance and occupant experience. As a few examples, onboard functions may include collision avoidance, autonomous driving, driver assistance, navigation, situation reporting, etc. Summary of the Invention

[0007] An attention-driven streaming system is provided, comprising an adaptive streaming module and a transceiver. The adaptive streaming module includes a filter, a compression module, an attention-driven policy module, and a fusion module. The filter is configured to filter sensor data received from sensors of a vehicle. The compression module is configured to compress the filtered sensor data to generate compressed data. The attention-driven policy module is configured to generate feedforward information based on the vehicle's state and the state of its environment to adjust the region of interest. The fusion module is configured to generate an adaptive streaming policy to adaptively adjust the operation of each of the filters. The transceiver is configured to stream the compressed data to at least one of an edge computing device or a cloud-based network device and to receive feedback information and pipeline monitoring information in response. The fusion module is configured to generate the adaptive streaming policy based on the feedforward information, feedback information, and pipeline monitoring information.

[0008] Among other features, the filters include: a time-domain filter configured to resample sensor data at a set frequency; a spatial-domain filter configured to select one or more geographic areas outside the vehicle; and a lossy compression filter configured to select at least one of a lossy compression method or a lossy compression rate of the compression module.

[0009] Among other features, the spatial domain filter is configured to: select one or more image resolutions for one or more selected geographic regions; apply one or more temporal domain methods to the one or more selected geographic regions; and adjust one or more different lossy compression rates for the one or more regions.

[0010] Among the other features, feedforward information is a first streaming strategy for streaming data to at least one of the edge computing devices or cloud-based network devices; and feedback information is a second streaming strategy for streaming data to at least one of the edge computing devices or cloud-based network devices.

[0011] Among other features, feedforward information is generated within the vehicle and includes requests for higher resolution data for the region of interest.

[0012] Among other features, the feedforward information includes a streaming strategy generated based on prediction information generated within the vehicle, and indicates the geographic area to be focused on. Among other features, filters are constructed to adjust the sampling rate of one or more sensors used for geographic area monitoring based on the streaming strategy in the feedforward information.

[0013] Among other features, the feedback information is generated within at least one of the edge computing device or cloud-based network device, and includes a request for higher resolution data for the indicated geographic area.

[0014] Among other features, the feedback information includes a streaming strategy generated based on prediction information generated in at least one of the edge computing devices or cloud-based network devices, and indicates the geographic area to be monitored. The filter is configured to adjust the sampling rate of one or more sensors for the geographic area based on the streaming strategy included in the feedback information.

[0015] Among other features, pipeline monitoring information indicates how congestion at at least one of the edge computing devices or cloud-based network devices affects the streaming rate of compressed data.

[0016] Among other features, the attention-driven policy module is constructed to generate feedforward information based on: a probabilistic representation of the object's state; the confidence level of the object's state; and a list of objects expected to be observed in the future and events expected to occur in the future.

[0017] Among other features, the attention-driven policy module is configured to generate feedforward information based on at least one of the following: differences between different sensors; current environmental conditions currently experienced by the vehicle or upcoming environmental conditions to be experienced; state information from neighboring vehicles; or map tracking and trajectory planning information.

[0018] Among other features, the attention-driven policy module is configured to generate feedforward information to include one or more attention regions for the task being performed.

[0019] Among other features, a vehicle system is provided, which includes: an attention-driven streaming system; and sensors.

[0020] Among other features, an attention-driven strategy approach is provided, comprising: filtering sensor data received from sensors at a vehicle via filters; compressing the filtered sensor data to generate compressed data; generating feedforward information based on the state of the vehicle and the state of the vehicle's environment; generating an adaptive streaming strategy to adaptively adjust the operation of each of the filters; and streaming the compressed data from the vehicle to at least one of an edge computing device or a cloud-based network device, and receiving feedback information and pipeline monitoring information from the edge computing device or the cloud-based network device in response. The adaptive streaming strategy is generated based on the feedforward information, feedback information, and pipeline monitoring information.

[0021] Among other features, filtering of sensor data includes: resampling sensor data at a set frequency; selecting one or more geographic areas outside the vehicle; and selecting at least one of a lossy compression method or a lossy compression rate for compressing the sensor data.

[0022] Among the other features, feedforward information is a first streaming strategy for streaming data to at least one of the edge computing devices or cloud-based network devices; and feedback information is a second streaming strategy for streaming data to at least one of the edge computing devices or cloud-based network devices.

[0023] Among the other features, at least one of the feedforward or feedback information includes a request for higher resolution data for the indicated geographic area.

[0024] Among other features, the feedforward information includes a streaming strategy generated based on prediction information generated within the vehicle, and indicates the geographic area to be monitored. Filtering includes adjusting the sampling rate of one or more sensors used for geographic area monitoring based on the streaming strategy in the feedforward information.

[0025] Among other features, the feedback information includes a streaming strategy generated based on prediction information produced within the vehicle, and indicates the geographic area to be monitored. Filtering includes adjusting the sampling rate of one or more sensors used for geographic area monitoring based on the streaming strategy in the feedback information.

[0026] Among other features, the adaptive streaming strategy includes increasing the resolution for the first region of interest and decreasing the resolution for the second region of interest.

[0027] Further applicable areas of this disclosure will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0028] This disclosure will be more fully understood from the detailed description and accompanying drawings, in which:

[0029] Figure 1 These are functional block diagrams and views of an example attention-driven streaming system according to the present disclosure, which includes a vehicle with an adaptive streaming module;

[0030] Figure 2 This is a functional block diagram of an example vehicle according to the present disclosure, the vehicle including a vehicle system including an adaptive streaming module;

[0031] Figure 3 yes Figure 1 A functional block diagram of a part of an attention-driven streaming system;

[0032] Figure 4 Based on this disclosure Figure 3 A functional block diagram of a portion of an attention-driven streaming system, which includes an example of an attention-driven policy module;

[0033] Figure 5 The illustration shows the first part of the adaptive streaming method implemented by a vehicle according to this disclosure; and

[0034] Figure 6 The illustration shows a second part of an adaptive streaming method implemented by an edge computing device or a cloud-based network device according to this disclosure.

[0035] In the accompanying drawings, reference numerals may be reused to identify similar and / or identical elements. Detailed Implementation

[0036] Significant amounts of sensor data can be streamed from the vehicle to edge computing devices or cloud-based network devices. This requires significant amounts of bandwidth, time, and cost. Several factors can be adjusted when transmitting sensor data. These factors include: the frequency of the transmitted data (called the time domain); the region of interest and resolution; and the lossy compression ratio of the sensor data (or video compression ratio). There are trade-offs to consider when adjusting these factors. For example, there is a trade-off between (i) bandwidth usage and (ii) edge computing / cloud-side performance and autonomous driving performance. Generally, the higher the resolution of the collected sensor data and the more sensor data is offloaded for processing, the more bandwidth is required, the higher the transmission latency, the higher the processing latency, and the better the processing results. The better the processing results, the better the performance of autonomous vehicle-related functions such as object / collision avoidance and navigation functions.

[0037] Examples illustrated in this paper include an attention-driven streaming system comprising an adaptive streaming module that supports adaptive sensor data streaming to edge computing devices and cloud-based network devices. Adaptive data streaming is based on perceptual attention. Perceptual attention refers to focusing on certain areas (or geographic regions) of interest in the environment. Vehicle functions can be performed on off-vehicle edge computing devices and cloud-based network devices to meet the growing demand for computing resources for autonomous vehicles without increasing the cost of onboard hardware. Edge computing and cloud-based resources can be shared by multiple vehicles. This reduces the per-vehicle hourly operating cost associated with using edge computing and cloud-based resources. Examples include performing operations to address the trade-off between (i) perceptual performance (which typically improves with increasing data resolution) and (ii) the increased bandwidth requirements with increasing resolution. Examples include focusing area monitoring on one or more reduced-size regions of interest. The reduction in the overall size and / or number of monitored areas allows for increased resolution of (one or more) monitored areas without collecting additional amounts of data, which would require additional bandwidth for off-board streaming. This can reduce the total amount of data collected, thereby providing bandwidth savings while maintaining or improving vehicle perception performance.

[0038] Figure 1 An attention-driven streaming system 100 is illustrated, comprising a vehicle 102, an edge computing device 104, a cloud-based network 106, an infrastructure device 108, and a personal mobile network device 110. The vehicle 102 may each include a transceiver 120, a control module 122, and sensors 124. The control module 122 includes an adaptive streaming module 126 that adaptively adjusts the sampling rate, streaming frequency, region of interest (region of focus), resolution, lossy compression ratio, lossy compression method, and / or other streaming parameters and / or aspects of the sensor data transmitted from the vehicle 102 to the edge computing device 104 and / or the cloud-based network device 128 of the cloud-based network 106.

[0039] Edge computing device 104 and cloud-based network device 128 may include corresponding attention-driven policy modules. Attention-driven policy modules 130 and 132 are shown. Although in Figure 1 Not shown in the diagram, but the control module 122 of vehicle 102 may also include an attention-driven strategy module, an example of which is shown in [reference needed]. Figure 3As shown in the diagram. Each of the attention-driven policy modules performs confidence, anticipation, and task relevance operations to provide feedback information to the adaptive streaming module 126. The vehicle system's attention can be focused on one or more geographic regions corresponding to the vehicle's exterior. A region can refer to a multidimensional space for which the autonomous vehicle perception module implementing the perception algorithm determines that it is of interest. As an example, the perception module (examples of which are shown in the diagram)... Figure 4 The perception algorithm (shown in the diagram) can determine that a specific object in a specific region is of interest. As a result, one of the attention-driven policy modules can then request increased resolution for the specific region of interest to, for example, better monitor the object. This may include monitoring the object's position, trajectory (or path), velocity, etc. The attention-driven policy module performs confidence, anticipation, and task-related operations to create a streaming policy profile. The streaming policy profile is sent as feedback to the adaptive streaming module 126. The adaptive streaming module 126 then adjusts the streaming parameters based on the feedback.

[0040] A streaming strategy overview can include temporal, spatial, and lossy compression information. The temporal domain refers to the sampling frequency of the sensor data and its adjustment. The sampling frequency can be increased or decreased depending on the corresponding sensor data and / or the level of interest in the region of interest. Sensor data can be collected for multiple regions, each with a different level of interest and a different corresponding sampling rate. The sampling rate can be the resampling rate. The sampling rate can refer to the camera's frames per second, the GPS's points per second, the sampling rate of analog signals, etc.

[0041] The spatial domain relates to multidimensional sensor data and refers to the region of interest, which includes the dimension and location of the region. Each region may have an assigned (i) resampling resolution (e.g., a lower or higher image resolution relative to the default resolution), (ii) temporal method (which may vary for different regions), and (iii) lossy compression rate (which may vary for different regions).

[0042] The lossy compression domain refers to lossy compression ratios and a lossy compression method under which sensor data is compressed before being streamed to a remote device. Some example lossy compression methods are: (i) video (or image) compression methods H.264 (referred to as Advanced Video Coding) and H.265 (referred to as Efficient Video Coding); and (ii) audio compression methods Advanced Audio Coding (AAC) and Moving Picture Experts Group (MPEG) Layer III (MP3). In some embodiments, the stated temporal, spatial, and lossy compression methods are combined to filter and compress streaming sensor data transmitted from vehicle 102 to edge computing device 104 and / or cloud-based network device 128.

[0043] Edge computing device 104 may include regional cellular tower equipment, base station equipment, city-level network equipment, microcomputing center equipment, etc. Edge computing device 104 does not refer to equipment within vehicle 102. Cloud-based network device 128 may be part of a distributed network, which includes servers, storage devices with databases, etc.

[0044] Vehicle 102 can communicate with infrastructure equipment 108 via a vehicle-to-infrastructure (V2I) communication link (such as a Long Term Evolution (LTE) or 5th Generation (5G) link). Vehicle 102 can communicate with edge computing device 104 via a vehicle-to-network (V2N) communication link (such as LTE and 5G links). Vehicle 102 can communicate with personal mobile network device 110 via a vehicle-to-person (V2P) communication link (such as LTE and 5G links).

[0045] Figure 2 It shows Figure 1 An example of one of the vehicles 102. Vehicle 102 includes a vehicle system 200, which includes a control module 122 and sensors 124. Vehicle 102 may be a partially or fully autonomous vehicle or other types of vehicles. Control module 122 may include an adaptive streaming module 126, a vehicle-to-infrastructure (V2I) module 202, a perception module 205, a trajectory module 206, and other modules 208. V2I module 202 may collect data from road infrastructure equipment located outside the vehicle. Infrastructure equipment may include traffic lights, traffic signs, devices mounted on buildings and / or road structures, etc. Control module 122 and / or adaptive streaming module 126 may implement neural networks and adaptive learning to improve the adjustment of streaming parameters. Perception module 205 may identify upcoming regions of interest and / or events based on the currently observed state of vehicle 102 and the corresponding environment. Perception module 205 may execute perception algorithms to predict what will happen in the next predetermined time period. For example, the perception module 205 can predict the position of an object relative to vehicle 102, the position of vehicle 102, changes in environmental conditions, changes in road conditions, changes in traffic flow, etc. The trajectory module 206 can determine the trajectory of vehicle 102 and / or other nearby vehicles. Other modules 208 may include vehicle-mounted functional modules, such as... Figure 3 The vehicle's onboard functional modules are shown in the diagram.

[0046] Memory 204 may store streaming strategy 210, parameters 212, data 214, and algorithm 216 (e.g., attention-driven strategy algorithm, perception algorithm, etc.). Sensor 124 may be distributed throughout vehicle 102 positioning and includes camera 220, infrared (IR) sensor 222, radar sensor 224, lidar sensor 226, and / or other sensors 228. Other sensors 228 may include yaw rate sensor, accelerometer, global positioning system (GPS) sensor, etc. Control module 122 and sensor 124 may communicate directly with each other, or via controller area network (CAN) bus 230 and / or via Ethernet switch 232. In the example shown, sensor 124 is connected to control module 122 via Ethernet switch 232, but may also or alternatively be directly connected to control module 122 and / or CAN bus 230.

[0047] Vehicle 102 may further include a chassis control module 240 and a torque source (such as one or more electric motors 242 and one or more engines (an engine 244 is shown)). The chassis control module 240 can control the distribution of output torque to the axles of vehicle 102 via the torque source. The chassis control module 240 can control the operation of a propulsion system 246, which includes one or more electric motors 242 and one or more engines 244. Engine 244 may include a starter motor 250, a fuel system 252, an ignition system 254, and a throttle system 256.

[0048] Vehicle 102 may further include a Body Control Module (BCM) 260, a Telematics Processing Module 262, a Braking System 263, a Navigation System 264, an Infotainment System 266, an Air Conditioning System 270, other actuators 272, other devices 274, and other vehicle systems and modules 276. The Navigation System 264 may include a GPS 278. Other actuators 272 may include steering actuators and / or other actuators. Control modules, systems, and modules 122, 240, 260, 262, 264, 266, 270, and 276 may communicate with each other via a CAN bus 230. A power supply 280 may be included, which may power the BCM 260 and other systems, modules, controllers, memories, devices, and / or components. The power supply 280 may include one or more batteries and / or other power sources. Control module 122 and / or BCM 260 may perform countermeasures and / or autonomous operation based on detected objects, the position of the detected objects, and / or other relevant parameters. This may include controlling the stated torque source and actuator, and providing images, indications and / or commands via the infotainment system 266.

[0049] The telematics module 262 may include a transceiver 282 and a telematics control module 284. This telematics module can be used to communicate with other vehicles, networks, edge computing devices, and / or cloud-based devices. The transceiver 282 may include... Figure 1 The transceiver 120. The BCM 260 can control modules and systems 262, 263, 264, 266, 270, 276 and other actuators, devices and systems (e.g., actuator 272 and device 274). This control can be based on data from sensor 124.

[0050] Figure 3 It shows Figure 1 A portion 300 of the attention-driven streaming system 100. Portion 300 includes a vehicle portion 302 shown to the left of dashed line 304 and an edge computing / cloud-based portion 306 shown to the right of dashed line 304. Although in Figure 3 Although not shown in the diagram, portions 302 and 306 may include corresponding transceivers. For example, portion 302 may include... Figure 2 The transceiver 282. The vehicle section 302 includes sensors 124, an adaptive streaming module 126, a memory 204, and a vehicle onboard function module 310. The edge computing / cloud-based section 306 includes a sensor streaming module 312 and a vehicle monitoring module 314. A communication interface 316 may exist between sections 302 and 306.

[0051] The adaptive streaming module 126 may include a filter 320, a compression module 322, a streaming strategy module 324, and an attention-driven strategy module 326, which operates similarly to other attention-driven strategy modules disclosed herein. The filter 320 includes a temporal filter 330, a spatial filter 332, and a lossy rate filter 334. The temporal filter 330 samples sensor data at a set frequency, as described above. The spatial filter 332 selects a region of interest, a sampling resolution for each monitored region, applies a temporal method to the corresponding monitored region, and / or applies different lossy compression rates to the monitored region. The spatial filter 332 may increase or decrease the resolution level of each region of interest. In one embodiment, the spatial filter 332 transitions between three-dimensional (3D) collection and filtering of sensor data and two-dimensional (2D) collection and filtering. The lossy rate filter 334 adjusts the lossy compression rate of the collected sensor data for the monitored region and / or selects a lossy compression method for the monitored region. The parameters stated by filters 330, 332, and 334 are set by the streaming strategy module 324. The compression module 322 compresses the filtered data output by filter 320, and then the filtered data is transmitted to the sensor streaming module 312.

[0052] The streaming strategy module 324 includes a feedforward channel interface 340, a feedback channel interface 342, a pipeline channel interface 344, and a strategy fusion module 346. The feedforward channel interface 340 receives a first streaming strategy from the attention-driven strategy module 326. The attention-driven strategy module 130 calculates confidence and anticipated data and predicts the demand for quality sensor data (sensor data with a high and / or predetermined minimum resolution level) for one or more regions of interest. For example, an autonomous vehicle may turn onto a new (or different) road. The attention-driven strategy module 130 then predicts that a new (or approaching) vehicle will be observed from a certain direction. The attention-driven strategy module 130 then calculates the attention region for which it will increase resolution. When data from sensors (e.g., lidar sensors and / or cameras) leads to inconsistent perception results, the attention-driven strategy module 130 may increase the resolution associated with one or more of the sensors to attempt to provide consistent results. As an example, a lidar sensor may detect reflections indicating the presence of an object, while a computer vision sensor may not detect the object. Therefore, there is a conflict between the lidar sensor and the computer vision sensor. The attention-driven strategy module 130 can calculate a new sampling rate by providing higher resolution sensor data to one or more sensors in an attempt to correct inconsistencies. If the inconsistencies are not corrected, conservative actions can be taken, such as reducing the speed of the autonomous vehicle, stopping the autonomous vehicle, and / or performing other operations.

[0053] Attention-driven policy module 326 operates locally and in a similar manner to attention-driven policy module 130. As an example, attention-driven policy module 326 may direct attention to an area in front of its own vehicle based on local detection of driving on a highway and request magnification and higher resolution for that area. Sensor data collected at higher resolution for that area can then be sent to an edge computing device or a cloud-based network device for processing. Attention-driven policy module 130 can then adjust the attention of adaptive streaming module 126 based on the results of the processed data.

[0054] As another example, the ego vehicle can transmit low-resolution data to an edge computing device or a cloud-based network device. The vehicle function module at the edge computing device or cloud-based network device determines that another vehicle is within 100 meters of the ego vehicle. The vehicle function module provides a confidence level regarding the detection of the other vehicle, and this confidence level is low due to the low resolution of the data for the other vehicle and the size of the image. The attention-driven policy module 130 then requests higher-resolution data for the region including the other vehicle. The streaming policy module 324 then adjusts the resolution of the spatial filter 332 to provide the higher-resolution data back to the edge computing device or cloud-based network device to better detect the other vehicle and / or determine that the other vehicle (or object) is not of interest.

[0055] Feedback channel interface 342 receives a second streaming policy from attention-driven policy module 130. Edge computing devices and / or cloud-based network devices can run perception algorithms, create attention-driven policies, and send feedback to the ego vehicle and more specifically to feedforward channel interface 340. Edge computing devices and / or cloud-based network devices can detect objects and / or events in advance and provide feedback to allow preparation for events and / or adjustment of the resolution of data collected for one or more areas. This may include reducing resolution, increasing resolution, and / or a combination of both for one or more areas.

[0056] As an example, an edge computing device and / or a cloud-based network device can detect a small object at a distance of 100 meters from its own vehicle and determine that the detection confidence level of the object is low. The edge computing device and / or the cloud-based network device can send feedback information to the own vehicle to request higher-resolution data. As another example, the edge computing device and / or the cloud-based network device can crowdsource environmental data from multiple vehicles and detect roads that are at least partially covered by snow. The edge computing device and / or the cloud-based network device can then send feedback information to the own vehicle to request higher-resolution data before the own vehicle enters the snow-covered area.

[0057] As another example, edge computing devices and / or cloud-based network devices can crowdsource road traffic data from multiple vehicles and detect specific traffic shock waves at curvature sections of the highway (visual occlusions for the vehicle itself). The edge computing devices and / or cloud-based network devices can then send feedback to the vehicle to request higher-resolution data before the vehicle itself enters the curvature section of the highway.

[0058] Pipeline channel interface 344 receives a congestion signal from communication interface 316. This congestion signal may indicate: congestion at sensor streaming module 312; an alternative streaming strategy; and / or streaming parameters. Congestion may be caused by: (i) the amount of data received is greater than the amount forwarded from sensor streaming module 312 to vehicle monitoring module 314, and / or (ii) the data transmission rate is higher than the data transmission rate from sensor streaming module 312 to vehicle monitoring module 314. Congestion at vehicle monitoring module 314 can cause congestion at sensor streaming module 312.

[0059] The policy fusion module 346 determines a collective (or adaptive) streaming policy based on a first streaming policy, a second streaming policy, and a congestion signal. As a result, the policy fusion module 346 utilizes three resources to drive the attention of the filter 320 to certain regions of interest. For example, congestion can occur when the vehicle moves into an LTE coverage area with weak signal and / or slow communication links between the vehicle and edge computing devices or cloud-based network devices. On the edge computing device or cloud-based network device side, congestion can also occur, or alternatively, when the processing throughput of vehicle functions slows down.

[0060] The policy fusion module 346 can arbitrate streaming policies and parameters received from channel interfaces 340, 342, and 344. In one embodiment, the policy fusion module 346 operates on a hierarchical basis. The policy fusion module 346 can prioritize which streaming policies and / or parameters to use for certain scenarios. When a streaming policy provided by an edge computing device or a cloud-based network device conflicts with a streaming policy locally determined by the attention-driven policy module 326, the policy fusion module 346 can select the most conservative policy or combine two or more policies to provide a resulting policy to be used at filter 320. The policy fusion module 346 can select policies to reduce and / or eliminate congestion at the edge computing device and / or cloud-based network device.

[0061] The pipeline associated with signal transfer between the vehicle and edge computing devices and / or cloud-based network devices includes serial components (or elements). Some examples of components (or elements) are 4G and / or 5G channels, video encoding and decoding devices, and applications stored on edge computing devices and / or cloud-based network devices. The pipeline channel interface 344 monitors the congestion level at the stated components (or elements) and elsewhere, and adjusts the transmission to... Figure 3 The sensor streaming rate of the compressed sensor data in the sensor streaming module 312. The sensor streaming rate is adjusted to avoid congestion and accumulated end-to-end latency.

[0062] The memory 204 stores the states 348 of the following: (i) the self-vehicle associated with the sensor 124 and modules 126, 310, and (ii) the environment of the self-vehicle (referred to as the "world" state). This state information is provided to the attention-driven policy module 326.

[0063] The vehicle onboard function module 310 may include a V2I module 202, a sensing module 205, a trajectory module 206, an actuator control module 350, and / or other vehicle function modules. The vehicle onboard function module 310 may include any vehicle function module that invokes sensor data processing. The vehicle onboard function module 310 may perform operations based on a status signal 352, which indicates the state of the ego vehicle and the world state, as determined by the vehicle monitoring module 314. The actuator control module 350 can control... Figure 2 The braking system 263 and / or other actuators 272 are included. The actuator control module 350 can, for example, control the vehicle's motors, steering, braking, acceleration, and deceleration based on the results of analyzing monitored data from sensors, the vehicle's own state, the world state, and / or other collected information and / or data. Figure 2 The control module 122 can control other driver assistance operations, such as displaying warning indicators and / or suggested operating instructions, based on the results of analyzing monitored data from sensors, the status of the vehicle itself, the state of the world, and / or other collected information and / or data.

[0064] Available Figure 1 Implemented at either the edge computing device 104 and / or the cloud-based network device 128 Figure 3 The system comprises a sensor streaming module 312 and a vehicle monitoring module 314. The sensor streaming module 312 may include a decompression module 360 ​​and a pipeline monitoring module 362. The decompression module 360 ​​decompresses compressed data received from the compression module 322. The pipeline monitoring module 362 monitors for congestion at the sensor streaming module 312 and indicates the presence of congestion to another pipeline monitoring module 364 of the vehicle monitoring module 314. The pipeline monitoring module 364 determines whether congestion exists at the vehicle monitoring module 314 and indicates the presence of congestion at the sensor streaming module 312 and / or at the vehicle monitoring module 314 to another pipeline monitoring module 370 and / or directly to the pipeline channel interface 344. The pipeline monitoring module 370 may be part of a communication interface 316 and may be a station, node, or other network device located between the pipeline monitoring module 364 and the pipeline channel interface 344. The pipeline monitoring module 370 may forward congestion information received from the pipeline monitoring module 364 to the pipeline channel interface 344.

[0065] The vehicle monitoring module 314 may further include a vehicle function module 372 and a memory 374. The vehicle function module 372 may include a sensing module 376, an object detection module 378, a tracking module 380, a localization and mapping module 382, ​​a trajectory planning module 384, a vehicle-to-network (V2N) module 386, and / or other vehicle function modules. The vehicle function module 372 may include any vehicle function modules that invoke sensor data processing. Figure 2 Similar to the perception module 205, the perception module 376 can identify upcoming regions of interest and / or events based on the current observed state of the autonomous vehicle and the corresponding environment. The object detection module 378 detects objects within a set distance of the autonomous vehicle. The tracking module 380 tracks environmental conditions and / or the position of detected objects relative to the autonomous vehicle. The localization and mapping module 382 maps the positions of the autonomous vehicle and objects relative to a geographic map of the area where the autonomous vehicle is located. The trajectory planning module 384 monitors, updates, and predicts the trajectories of the autonomous vehicle and objects. The V2N module 386 collects data from other devices near and / or close to the autonomous vehicle. The collected data may include state information of objects and / or other devices.

[0066] Memory 374 stores the state 388 of the self-vehicle, objects, and environment (also known as the "world" state), as determined by vehicle function module 372. State 388 is provided to attention-driven policy module 130, which generates a second streaming policy based on state 388.

[0067] exist Figure 3 In the diagram, dashed lines between modules represent control flow signals. Solid lines between modules represent data flow signals.

[0068] As an example, Figure 3 The adaptive streaming module 126 can receive camera frames from one or more cameras. The compression module 322 can compress the filtered data received from the adaptive streaming module 126 and send the data to the sensor streaming module 312. The decompression module 360 ​​can decode the data. Then, the object detection module 378 can detect objects within a predetermined distance of the adaptive streaming module 126's own vehicle, generate a corresponding confidence level associated with the detected object, and provide this information as feedback to the adaptive streaming module 126. The adaptive streaming module 126 can then adjust the resolution for a concentrated area of ​​the detected object. The resolution of other monitored areas can be reduced. This allows for increasing the resolution for a selected region of interest without increasing the bandwidth required to transmit sensor data. Because the resolution of the area where the object is located is increased, the confidence level of the detected object can remain the same or increase.

[0069] Figure 4It shows Figure 3 This is a portion 400 of an attention-driven streaming system. Portion 400 includes examples of an attention-driven policy module 402, a first memory 404, a second memory 406, a V2I module 408, and a perception module 410. The attention-driven policy module 402 may operate similarly to and / or replace any of the attention-driven policy modules disclosed herein. The attention-driven policy module 402 may include a confidence module 420, a prediction module 422, and a task relevance module 424. The confidence module 420 may include a state confidence module 430 and a sensor disparity module 432. The prediction module 422 includes a map tracking module 434, an environment tracking module 436, a contextual awareness module 438, and a trajectory planning module 440. The task relevance module 424 includes a scene analysis module 442.

[0070] Attention-driven strategy module 402 generates streaming strategy signal 460 based on the outputs of modules 420, 422, and 424. This streaming strategy signal may include a streaming strategy overview (or simply "streaming strategy"). Streaming strategy signal 460 may be provided as (i) feedforward information if attention-driven strategy module 402 is implemented as part of a vehicle, or (ii) feedback information if attention-driven strategy module 402 is implemented as part of an edge computing device or a cloud-based network device.

[0071] The first memory 404 may include a map database 450 and an external database 452. The map database 450 may include road topology information, intersection information, traffic rules, etc. The external database 452 may include weather information (e.g., whether it is snowing, raining, foggy, sunny, cloudy, etc.), road conditions, traffic congestion information, etc. The second memory 406 stores the state 470 of the vehicle and its environment (or "world" state). The V2I module 408 and the perception module 410 may indicate the state of the vehicle and / or the environment.

[0072] Modules 420, 422, and 424 can operate based on information stored in memories 404 and 406. The confidence module 420 can determine a probabilistic representation of the object's state (x, y, z, velocity, direction of travel, etc.). This includes indicating whether an object has been detected and / or whether an event has been observed. The state confidence module 430 can determine the confidence level of detected and / or predicted objects and events. The variance and covariance values ​​of the observed and / or predicted states can be calculated. The sensor difference module 432 can determine differences and / or conflicts between data collected from different sensors. For example, when the first sensor indicates an object state different from that indicated by the second sensor.

[0073] The anticipation module 422 lists objects and events expected to exist and / or occur in multidimensional space. These events can be events expected to occur and / or be observed in the near future. The map tracking module 434 tracks the location of the ego vehicle on a map. For example, if the ego vehicle is at an intersection, it pays attention to the direction of travel of oncoming vehicles. The environment tracking module 436 tracks environmental conditions from an external database, such as weather, lighting conditions, traffic congestion, etc. For example, if it is dark (e.g., nighttime) and / or snowing or raining, the basic attention level can be increased to transmit more data within a predetermined time period.

[0074] The context awareness module 438 can receive vehicle context awareness information from neighboring vehicles via vehicle-to-vehicle (V2V), V2I, and V2N communication. For example, based on the context awareness information, the self-vehicle can predict its attention area. The trajectory planning module 440 plans short-range trajectories for the self-vehicle's control module.

[0075] The scene analysis module 442 of the task relevance module 424 performs task-driven scene analysis, including analyzing the scene and identifying attention regions relevant to the current task. The scene analysis module 442 may, for example, determine a region of interest including the traffic lights of the intersection when the vehicle approaches an intersection. Another region of interest may include the area where the vehicle approaches the intersection from a direction perpendicular to its direction of travel. As another example, the scene analysis module 442 may detect when the vehicle changes lanes. In contrast to the vehicle's current lane, one or more areas including at least a portion of adjacent lanes then become regions of interest (or attention).

[0076] Figure 5 It shows the vehicle (e.g., Figure 1 and Figure 2 The first part of the adaptive streaming method implemented in one of the vehicles 102. Although mainly about Figure 1-4 Examples to describe Figure 5-6 The following operations are applicable to other embodiments of this disclosure. The operations of the method can be performed iteratively. The method may begin at 500. At 502, sensor 124 may generate sensor data. At 504, filter 320 may filter the sensor data according to a default or the latest multi-channel based filtering strategy to provide filtered data. At 506, compression module 322 compresses the filtered data.

[0077] At point 508, Figure 2 The transceiver 282 transmits compressed data to including Figure 3 The decompression module of 360 edge computing devices or cloud-based network devices.

[0078] At 510, in response to transmitting compressed data, feedback channel interface 342 receives a remotely determined state of the ego vehicle and receives the environmental (or world) state via signal 352 generated by memory 374. State information may include the position and trajectory of the ego vehicle and nearby objects, as well as road and weather conditions. This information can be received from the attention-driven policy module.

[0079] At 512, the vehicle onboard function module 310 performs vehicle functions to provide the vehicle's state and the state of its own environment to the memory 204. These functions are performed based on the state information received from the memory 374 at 510. At 514, the memory 204 stores the state of the own vehicle and its environment and provides it to the attention-driven policy module 326.

[0080] The following operations can be performed simultaneously: 516, 518, and 520. At 516, the attention-driven policy module 326 performs confidence, anticipation, and task relevance operations based on the received states of the ego vehicle and the environment. At 516A, the first confidence module determines a probabilistic representation of the states of one or more objects. At 516B, the first confidence module determines the confidence level of the one or more observed states.

[0081] At 518, the first confidence module determines the differences between sensors indicating different states of the same one or more objects, different road conditions on the same road, and / or different weather conditions of the same type of weather. The first anticipation module generates a list of objects and / or events expected to be detected and / or observed in the future within a multi-dimensional space. At 518A, the first anticipation module tracks the vehicle's position on a map. At 518B, the first anticipation module tracks environmental conditions. At 518C, the first anticipation module collects state information from other vehicles to provide contextual awareness information. Contextual awareness information may include object information, road conditions, and weather conditions. Contextual awareness information may also include the distance between the object and the self-vehicle, the rate of change of distance, and indications of whether the distance is increasing or decreasing. At 518D, the first anticipation module predicts the area of ​​attention to be monitored based on the contextual awareness information. At 518E, the first anticipation module plans a short-range trajectory for the self-vehicle.

[0082] At point 520, the first task relevance module analyzes the current scene of the one or more monitored regions and identifies one or more attention regions. An attention region can be one or more currently monitored regions or a portion of other regions.

[0083] At point 522, the first attention-driven policy module determines a local attention-driven policy for the self-vehicle based on confidence level, sensor differences, contextual awareness information, short-range trajectory of the self-vehicle, and one or more attention regions. The local attention-driven policy includes temporal, spatial, and / or lossy rate values ​​of filter 320.

[0084] At 524, the pipeline channel interface 344 can receive streaming strategy adjustment values ​​and / or other time, space, and / or loss rate values ​​for filter 320. Streaming strategy adjustment values ​​can be provided to mitigate congestion occurring at one of modules 312 and 314.

[0085] At point 526, the first attention-driven policy module can receive a remote attention-driven policy from an edge computing device or a cloud-based network device as feedback information at the feedback channel interface. At the edge computing device or cloud-based network device located remotely from the vehicle, the remote attention-driven policy is determined based on data streamed from the vehicle to the edge computing device or cloud-based network device.

[0086] At point 528, the policy fusion module 346 determines a multi-channel-based filtering strategy, which is provided to filter 320. This includes temporal, spatial, and lossy filter parameters to adaptively adjust the data stream to edge computing devices or cloud-based network devices.

[0087] At 530, the adaptive streaming module 126 determines whether there is additional sensor data to be filtered. If so, operation 504 can be performed; otherwise, the method can end at 532.

[0088] Figure 6 This illustrates edge computing devices or cloud-based network devices ( Figure 1 The second part of the adaptive streaming method implemented by one of the devices 104 and 128. This second part may begin at 600. At 602, the decompression module 360 ​​receives compressed data from the compression module 322. At 604, the decompression module 360 ​​decompresses the compressed data and forwards the decompressed sensor data to the vehicle monitoring module 314.

[0089] At 606, the vehicle function module 372 performs vehicle functions to provide the state of its own vehicle and the state of its own environment. At 608, these states are stored in memory 374. At 610, memory 374 and / or vehicle monitoring module 314 provide the states as described above to vehicle onboard function module 310.

[0090] The following operations can be performed simultaneously: 612, 614, and 616. At 612, the attention-driven policy module 130 performs reliability, anticipation, and task relevance operations based on the state of the self-vehicle and the environment. At 612A, the second reliability module determines a probabilistic representation of the state of one or more objects. At 612B, the second reliability module determines the confidence level of the one or more observed states.

[0091] At 614, the second confidence module determines the differences between sensors indicating different states of the same one or more objects, different road conditions on the same road, and / or different weather conditions of the same type of weather. The second anticipatory module generates a list of objects and / or events expected to be detected and / or observed in the future within a multi-dimensional space. At 614A, the second anticipatory module tracks the vehicle's position on a map. At 614B, the second anticipatory module tracks environmental conditions. At 614C, the second anticipatory module collects state information from other vehicles to provide contextual awareness information. Contextual awareness information may include object information, road conditions, and weather conditions. Contextual awareness information may also include the distance between the object and the self-vehicle, the rate of change of distance, and indications of whether the distance is increasing or decreasing. At 614D, the second anticipatory module predicts the area of ​​attention to be monitored based on the contextual awareness information. At 614E, the second anticipatory module plans a short-range trajectory for the self-vehicle.

[0092] At point 616, the second task relevance module analyzes the current scene of the one or more monitored regions and identifies one or more attention regions. An attention region can be one or more currently monitored regions or a portion of other regions.

[0093] At 618, the attention-driven strategy module 130 determines a remote attention-driven strategy based on the confidence level determined at 614, 616, and 618, sensor differences, contextual awareness information, the short-range trajectory of the ego vehicle, and the one or more attention regions. The local attention-driven strategy includes the temporal, spatial, and / or loss rate values ​​of the filter 320.

[0094] At 620, the attention-driven strategy module 130 can transmit remote attention-driven strategies to the feedback channel interface 342 of the ego vehicle.

[0095] At 622, pipeline monitoring module 362 monitors the congestion level at sensor streaming module 312 and generates a first streaming rate adjustment value. At 624, pipeline monitoring module 364 monitors the congestion level at vehicle monitoring module 314 and / or vehicle function module 372 and generates a second streaming rate adjustment value, which may be based on the first streaming rate adjustment value. At 626, pipeline monitoring module 364 may transmit the second streaming rate adjustment value to pipeline channel interface 344 via communication interface 316 and / or pipeline monitoring module 370.

[0096] At 630, the sensor streaming module 312 can determine whether additional streaming data has been received. If so, operation 604 can be performed; otherwise, the method can end at 632.

[0097] This paper discloses an architecture that uses an attention-driven strategy to prioritize sensor data based on scene understanding and task-driven strategies while streaming sensor data in real time for edge computing devices and cloud-based network devices. Figure 3 The adaptive streaming module 126 can determine a loss function based on (i) the Quality of Service (QoS) of the streaming data from onboard sensors to off-board network devices, and (ii) perception performance. The adaptive streaming module 126 can adjust the operation of filter 320 based on the loss function as described above. A set of policies can be generated and implemented based on perception attention operations to adjust the sensor streaming data rate. A trade-off between bandwidth usage and edge computing / cloud-side functional performance is balanced without sacrificing perception performance (e.g., object detection rate). This improves cloud-based processing of sensor data for mission-critical real-time applications and overcomes the drawbacks associated with traditional cloud-based processing. The processing power of edge computing devices and cloud-based network devices is utilized in real-time to enhance local vehicle operations.

[0098] Examples include selective attention as input to a real-time adaptive sensor data streaming strategy. The attention strategy is derived using three independent sources: confidence, assumption, and task-relevant sources. In one embodiment, a task-oriented strategy is used, and certain regions are selected for high-quality images, while data from other regions and / or sensors are partially ignored and / or disregarded. Partial disregard for other regions may include collecting images of other regions with reduced image quality and / or processing data from other regions using fewer resources (processing power, processing time, memory, etc.).

[0099] The foregoing description is merely illustrative in nature and is in no way intended to limit this disclosure, its application, or use. The broad teachings of this disclosure can be implemented in many forms. Therefore, although this disclosure includes specific examples, its true scope should not be so limited, as other modifications will become apparent upon examination of the drawings, specification, and the following claims. It should be understood that one or more steps within the method may be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, while each of the embodiments described above is described as having certain features, any one or more of those features described with respect to any embodiment of this disclosure can be implemented and / or combined with features of any other embodiment, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more embodiments with each other remains within the scope of this disclosure.

[0100] Spatial and functional relationships between components (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including “connection,” “joint,” “link,” “proximity,” “adjacent,” “on top of,” “above,” “below,” and “placement.” Unless explicitly described as “direct,” when describing the relationship between the first and second components in the above disclosure, the relationship can be a direct relationship in which no other intervening components exist between the first and second components, or an indirect relationship (spatially or functionally) between the first and second components. As used herein, the phrases A, B, and C at least one should be interpreted as referring to the logic (A OR B OR C) using non-exclusive logic OR, and should not be interpreted as referring to “at least one of A, at least one of B, and at least one of C.”

[0101] In the accompanying drawings, the direction of the arrows, as indicated by the arrows, typically illustrates the flow of information of interest (such as data or instructions). For example, when components A and B exchange various types of information, but the information transmitted from component A to component B is relevant to the illustration, the arrow may point from component A to component B. This unidirectional arrow does not imply that no other information is transmitted from component B to component A. Furthermore, for information sent from component A to component B, component B may send a request for or confirmation of receipt of the information to component A.

[0102] In this application (including the definitions below), the term "module" or "controller" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores code executed by the processor circuitry; other suitable hardware components that provide the described functionality; or combinations of some or all of the above, such as in a system-on-a-chip.

[0103] A module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of this disclosure may be distributed among multiple modules connected via the interface circuits. For example, multiple modules may allow for load balancing. In another example, a server (also referred to as a remote or cloud) module may perform a function on behalf of a client module.

[0104] The term "code" as used above can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" covers a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" covers a processor circuit that, in combination with additional processor circuits, executes some or all of the code from one or more modules. The reference to multiple processor circuits covers multiple processor circuits on a discrete die, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" covers a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" covers a memory circuit that, in combination with additional memory, stores some or all of the code from one or more modules.

[0105] The term memory circuit is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not cover transient electrical or electromagnetic signals propagating through a medium (such as a carrier wave); therefore, the term computer-readable medium can be considered tangible and non-transitory. Non-limiting examples of non-transitory, tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0106] The apparatus and methods described in this application can be implemented, in part or in whole, by a dedicated computer created by a general-purpose computing mechanism to perform one or more specific functions embodied in a computer program. The function blocks, flowchart components, and other elements described above serve as software specifications that can be converted into computer programs through the routine work of skilled technicians or programmers.

[0107] A computer program includes processor-executable instructions stored on at least one non-transitory, tangible, computer-readable medium. A computer program may also include or depend on stored data. A computer program may encompass a basic input / output system (BIOS) for interacting with the hardware of a special-purpose computer, device drivers for interacting with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0108] Computer programs may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code executed by an interpreter; and (v) source code compiled and executed by a just-in-time (JIT) compiler, etc. As an example only, source code may be written in languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language version 5), Ada, ASP (Dynamic Server Web Pages), PHP (PHP: Hypertext Preprocessing Language), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.

Claims

1. An attention-driven streaming system, comprising: An adaptive streaming module, comprising: Multiple filters are configured to filter sensor data received from multiple sensors of the vehicle. The compression module is configured to compress filtered sensor data to generate compressed data. An attention-driven strategy module is configured to generate feedforward information based on the state of the vehicle and the state of the vehicle's environment to adjust the region of interest. A fusion module, configured to generate an adaptive streaming strategy to adaptively adjust the operation of each of the plurality of filters; and A transceiver configured to stream the compressed data to at least one of an edge computing device or a cloud-based network device, and to receive feedback and pipeline monitoring information in response. The fusion module is configured to generate the adaptive streaming strategy based on the feedforward information, the feedback information, and the pipeline monitoring information. The plurality of filters include: A time-domain filter is configured to resample the sensor data at a set frequency; A spatial domain filter, configured to select one or more geographic regions outside the vehicle; and A lossy compression filter configured to select at least one of a lossy compression method or a lossy compression ratio from the compression module. The spatial domain filter is constructed as follows: Select one or more image resolutions for the selected geographic regions; Apply one or more time-domain methods to one or more selected geographic regions; and Adjust one or more different lossy compression ratios for the one or more regions. The adaptive streaming strategy adaptively adjusts the sampling rate, streaming frequency, region of interest, resolution, lossy compression ratio, and / or lossy compression method of the sensor data. The fusion module is configured to prioritize streaming policies and / or parameters, and when a streaming policy provided by an edge computing device or a cloud-based network device conflicts with a streaming policy locally determined by the attention-driven policy module, the fusion module combines two or more policies to provide a resulting policy to be used at the plurality of filters.

2. The attention-driven streaming system according to claim 1, wherein: The feedforward information is a first streaming strategy for streaming data to at least one of the edge computing device or the cloud-based network device. and The feedback information is a second streaming strategy for streaming data to at least one of the edge computing device or the cloud-based network device.

3. The attention-driven streaming system according to claim 1, wherein, The feedforward information is generated within the vehicle and includes a request for higher resolution data for the region of interest.

4. The attention-driven streaming system according to claim 1, wherein: The feedforward information includes a streaming strategy generated based on prediction information generated within the vehicle, and indicates the geographic areas to be monitored. and The plurality of filters are configured to adjust the sampling rate of one or more of the plurality of sensors used for the geographic region based on the streaming strategy in the feedforward information.

5. The attention-driven streaming system according to claim 1, wherein, The feedback information is generated within at least one of the edge computing device or the cloud-based network device, and includes a request for higher resolution data for the indicated geographic area.

6. The attention-driven streaming system according to claim 1, wherein: The feedback information includes a streaming strategy generated based on prediction information generated in at least one of the edge computing device or the cloud-based network device, and indicates the geographical area to be centrally monitored; and The plurality of filters are configured to adjust the sampling rate of one or more of the plurality of sensors used for the geographic area based on the streaming strategy included in the feedback information.

7. The attention-driven streaming system according to claim 1, wherein, The pipeline monitoring information indicates an adjustment of the streaming rate of the compressed data based on congestion at at least one of the edge computing device or the cloud-based network device.

8. The attention-driven streaming system according to claim 1, wherein, The attention-driven policy module is configured to generate the feedforward information based on the following: The probabilistic representation of the state of an object; The confidence level of the state of the object; as well as A list of objects expected to be observed in the future and events expected to occur in the future.

9. The attention-driven streaming system according to claim 1, wherein, The attention-driven policy module is configured to generate the feedforward information based on at least one of the following: Differences between different sensors; The current environmental conditions that the vehicle is currently experiencing or the upcoming environmental conditions that it will experience; Status information from adjacent vehicles; or Map tracking and trajectory planning information.

Images