Method and system for performing dynamic bandwidth adjustment between vehicle sensors

The dynamic bandwidth adjustment method optimizes sensor data rates based on environmental conditions and sensor detection using machine learning, addressing inefficiencies in constant bandwidth systems by reallocating bandwidth to enhance processing efficiency.

DE102019102201B4Active Publication Date: 2024-12-05GM GLOBAL TECHNOLOGY OPERATIONS LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
DE102019102201
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-02-01
Filing Date
2019-01-29
Publication Date
2024-12-05
Estimated Expiration
2039-01-29

AI Technical Summary

Technical Problem

Existing vehicle sensors provide data with a constant bandwidth, which is inefficient when one sensor cannot detect an object due to environmental conditions, leading to unnecessary data volume and processing load, thereby limiting the bandwidth available for other sensors.

Method used

A method and system for dynamically adjusting the bandwidth of vehicle sensors based on environmental conditions, sensor detection, and vehicle dynamics using a controller that implements machine learning algorithms to optimize data rates.

Benefits of technology

Optimizes data bandwidth allocation among vehicle sensors, ensuring efficient data processing by reducing unnecessary data rates from non-detecting sensors and reallocating bandwidth to detecting sensors, thereby maintaining overall data volume and processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Method for performing a dynamic bandwidth adjustment between two or more vehicle sensors (110), wherein the method comprises: Receiving inputs at a processor, wherein the input includes data from each of the two or more vehicle sensors (110), wherein the two or more vehicle sensors (110) include a camera (111), an audio detector (112), a radar system (113) or a lidar system; Determining a bandwidth using the processor at which each of the two or more vehicle sensors (110) is to provide the data; Providing a control signal to each of the two or more vehicle sensors (110) by the processor to adjust the bandwidth based on the determination; and where determining the bandwidth involves reducing the bandwidth from a standard bandwidth for one or more of the two or more vehicle sensors (110) that cannot detect an object (230) that is detected by one or more of the two or more vehicle sensors (110).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The description relates to a method and a system for performing dynamic bandwidth adaptation between vehicle sensors.

[0002] Vehicles (e.g., automobiles, trucks, agricultural equipment, construction machinery, automated factory equipment) increasingly incorporate sensors that facilitate the expansion and automation of vehicle systems. One exemplary sensor type is a camera that receives images. Multiple cameras can be arranged to obtain, for example, a 360-degree view around the perimeter of the vehicle. Another exemplary sensor type is an audio detector that receives audio from outside the vehicle. Additional exemplary sensors include a radio detection and ranging system (radar) and a light detection and ranging system (lidar). Exemplary vehicle systems include collision avoidance, adaptive cruise control, and autonomous driving. In earlier vehicles, the various sensors provide data at a constant bandwidth.

[0003] US 2017 / 0 031 363 A1 describes a system for automated operation of a host vehicle, comprising an object detection device and a controller. The object detection device is capable of detecting an object in a field of view near a host vehicle. The object detection device is capable of varying a field of focus of the object detection device, which is used to observe a portion of the field of view. The controller is configured to determine a direction of travel of the object relative to a path of travel of the host vehicle based on information received from the object detection device. The controller is further configured to adjust the field of focus of the object detection device based on the direction of travel.

[0004] US 2016 / 0 231 746 A1 describes systems and methods for operating an automated vehicle, such as an autonomous vehicle, which may include an autonomous guidance system, a method for automatically controlling an autonomous vehicle based on electronic messages from road infrastructure or other vehicles, a method for automatically controlling an autonomous vehicle based on location information from mobile phones, a pulsed LED vehicle-to-vehicle communication system, a method and apparatus for controlling an autonomous vehicle, an autonomous vehicle with unobtrusive sensors and adaptive cruise control with integrated lane departure warning.The systems and methods can use information from radar, lidar, a camera or vision / imaging devices, ultrasonic sensors and digital map data to determine a route or road position and to control the steering, braking and acceleration of a host vehicle.

[0005] It can be considered a task to specify a dynamic bandwidth adaptation between vehicle sensors.

[0006] According to the invention, a method for performing dynamic bandwidth adjustment between two or more vehicle sensors includes receiving an input at a processor, the input including data from each of the two or more vehicle sensors. The two or more vehicle sensors include a camera, an audio detector, a radar system, or a lidar system. The method also includes determining, using the processor, a bandwidth at which each of the two or more vehicle sensors is to provide the data, and providing, by the processor, a control signal to each of the two or more vehicle sensors to adjust the bandwidth based on the determination.

[0007] In one embodiment, receiving the inputs also includes receiving information about environmental conditions.

[0008] In one embodiment, receiving the information about the environmental conditions also includes receiving weather information.

[0009] In one embodiment, receiving the inputs also includes receiving information about the vehicle.

[0010] In one embodiment, receiving information about the vehicle also includes receiving location information, heading information, or vehicle dynamics information.

[0011] According to the invention, determining the bandwidth includes reducing the bandwidth from a standard bandwidth for one or more of the two or more vehicle sensors that cannot detect an object detected by one or more of the two or more vehicle sensors.

[0012] In one embodiment, determining the bandwidth is based on a direction of travel of the vehicle and a position of the two or more sensors on the vehicle.

[0013] In one embodiment, determining the bandwidth is based on an event, wherein the event is a detection by at least one of the two or more vehicle sensors or a change in movement by the vehicle.

[0014] In one embodiment, determining the bandwidth includes implementing a machine learning algorithm.

[0015] According to the invention, a system for performing dynamic bandwidth adjustment includes two or more vehicle sensors, wherein the two or more vehicle sensors include a camera, an audio detector, a radar system, or a lidar system. The system also includes a controller configured to receive inputs, the input including data from each of the two or more vehicle sensors, determine a bandwidth at which each of the two or more vehicle sensors should provide the data, and provide a control signal to each of the two or more vehicle sensors to respectively adjust the bandwidth for each of the two or more vehicle sensors.

[0016] In one embodiment, the inputs include information about environmental conditions.

[0017] In one embodiment, the environmental information includes weather information.

[0018] In one embodiment, the inputs include information about the vehicle.

[0019] In one embodiment, the information about the vehicle also includes location information, direction of travel, or vehicle dynamics information.

[0020] According to the invention, the controller reduces the bandwidth from a standard bandwidth for one or more of the two or more vehicle sensors that cannot detect an object detected by one or more of the two or more vehicle sensors.

[0021] In one embodiment, the controller determines the bandwidth based on a direction of travel of the vehicle and a position of the two or more sensors on the vehicle.

[0022] In one embodiment, the controller determines the bandwidth based on an event, wherein the event is a detection by at least one of the two or more vehicle sensors or a change in motion by the vehicle.

[0023] In one embodiment, the controller determines the bandwidth by implementing a machine learning algorithm. Fig. 1 is a block diagram of a vehicle with dynamic bandwidth adjustment between vehicle sensors according to one or more embodiments; Fig. 2 illustrates a scenario in which dynamic bandwidth adaptation between vehicle sensors is performed according to one or more embodiments; Fig. 3 illustrates an exemplary bandwidth controller according to one or more embodiments; and Fig. 4 is a process flow of a method for performing dynamic bandwidth adaptation between vehicle sensors according to one or more embodiments.

[0024] As mentioned earlier, different types of sensors can provide data to facilitate the expansion or automation of vehicle systems. For example, the radar system can track an object moving into the vehicle's path. The information from the radar system can be used to provide a warning to the driver or, for example, to take evasive action with the collision avoidance system. Data from more than one type of sensor can be used together in a scheme called data fusion. In data fusion, data from one type of sensor can validate data from another type of sensor or, for example, facilitate the determination of a false alarm. As mentioned earlier, vehicle sensors currently provide data with a constant bandwidth.

[0025] The bandwidth at which each sensor provides data affects the bandwidth available for the other sensors to provide data. This is because the volume of data that can be stored and processed by the controller receiving data from the various sensors is not unlimited. Thus, if all sensors provide data at a constant bandwidth, the bandwidth set for each sensor must be selected taking into account the data volume and processing load it generates.

[0026] However, situations may arise where one type of sensor provides useful information while another type of sensor does not or cannot provide useful information. For example, in heavy fog, an audio detector may record audio data from a nearby motorcycle, but a camera may not be able to obtain images of the motorcycle because the camera's field of view is obstructed by the fog. In such a situation, maintaining the data rate from the camera that cannot provide useful information is not optimal. Furthermore, the volume of data still provided by the blocked camera prevents a higher data rate for information from the audio detector. Embodiments of the systems and methods described herein relate to dynamic bandwidth adaptation between vehicle sensors.

[0027] According to an exemplary embodiment, Fig. 1 is a block diagram of a vehicle 100 with dynamic bandwidth adjustment between vehicle sensors 110. The exemplary vehicle 100 in Fig. 1 is an automobile 101. The vehicle includes sensors 110 of various types. As illustrated, the vehicle 100 includes cameras 111 as one type of sensor 110a, audio detectors (e.g., microphones) 112 as another type of sensor 110b, and a radar system 113 as a third type of sensor 110c. The exemplary sensors 110 are shown for illustrative purposes and are not intended to limit the types, number, or locations of the sensors 110 according to one or more embodiments. Each of the sensors 110 provides data to a controller 120. The controller 120 may be the same electronic control unit (ECU) that provides control signals to various vehicle systems. The controller 120 may instead be in communication with the vehicle ECU along with the sensors 110.

[0028] The controller 120 includes processing circuitry, which may include an application-specific integrated circuit (ASIC), an electronic circuit, a hardware computer processor (shared or dedicated, or group), and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality. According to additional or alternative embodiments, the controller 120 may implement a machine learning algorithm. The controller 120 not only receives data from each of the sensors 110 but also provides control signals to the sensors 110 to adjust the bandwidth, as discussed herein. The data from the sensors 110 may be raw data for processing by the controller 120, may be processed data indicating an object location or other information, or a combination of the two.Based on the data from the sensors 110, the controller 120 may provide information (e.g., driver warnings) for display on a screen of an infotainment system 130 or provide information to other vehicle systems 140.

[0029] In addition to the sensor data, the controller 120 receives a variety of information about the vehicle 100. For example, the infotainment system 130 may include a mapping application that includes a global positioning system (GPS). The infotainment system 130 may provide both the location of the vehicle 100 and information about the location based on the mapping (e.g., the vehicle 100 is approaching a railroad crossing). The infotainment system 130 or other vehicle systems 140 may additionally provide environmental conditions (e.g., snow, fog, temperature) to the controller 120. Other vehicle systems 140 may provide information to the controller 120, which it uses to make decisions about vehicle operation. The information may include, for example, vehicle dynamics, direction (e.g., the vehicle is moving in reverse), and movement (e.g., the vehicle is changing lanes).

[0030] Fig. Figure 2 illustrates a scenario in which dynamic bandwidth adjustment between vehicle sensors 110 is performed according to one or more embodiments. The vehicle 100 is shown with a camera 111 and an audio detector 112 at the rear of the vehicle 100. While two sensors 110 are discussed for illustrative purposes, any number of sensors 110 may be used and their bandwidth adjusted according to the embodiments described herein. The camera field of view 210 and the audio detection range 220 are indicated. An object 230 is shown at three different positions 1, 2, 3 relative to the vehicle 100. As Fig. 2 shows, when the object 230 is at position 1, the object 230 is within the detection range 220 of the audio detector 112, but not within the field of view 210 of the camera 111. At position 2, the object 230 moves to a location that, in addition to the detection range 220 of the audio detector 112, is within the field of view 210 of the camera 111. At position 3, the object 230 is still within the field of view 210 of the camera 111 and within the detection range 220 of the audio detector 112.

[0031] How Fig. As shown in Figure 2, maintaining the same data rate for data from camera 111 at positions 1 and 3 of object 230 is not optimal. This is because when object 230 is in position 1, camera 111 does not detect object 230 at all, thus providing no information. Additionally, once camera 111 has object 230 in camera field of view 210 (at position 2), maintaining the same data rate for data from audio detector 112 may be unnecessary. By reducing the data rate of data from one sensor 110, the data rate of data from another sensor 110 can be increased while maintaining the same overall volume of data and processing.

[0032] According to one or more embodiments, the controller 120 dynamically adjusts the bandwidth for data received from each of the sensors 110. As described with reference to Fig. 3, is used for the Fig. 2, the bandwidth for data from camera 111 and audio detector 112 is adjusted as object 230 moves from position 1 to position 2 to position 3. Additional information used by controller 120 may include, for example, an indication that vehicle 100 is in reverse gear. As the vehicle moves forward and thus away from object 230, adjusting the data rates for sensors 110 at the rear of vehicle 100 may be deemed unnecessary or implemented differently (e.g., the data rates for camera 111 and audio detector 112 may both be reduced).

[0033] Fig. 3 illustrates an exemplary bandwidth controller according to one or more embodiments. Fig. The scenario shown in Figure 2 corresponds to the one in Fig. 3, and the three positions 1, 2, 3 are indicated along the time axis 301. The data rate is indicated along the axis 302. The bandwidth control is implemented by the controller 120. Fig. 3 shows the data rate 310 of the camera 111 and the data rate 320 of the audio detector 112. How Fig. 3, the data rate 320 of the audio detector 112 is maximum when the object 230 is at position 1 and is reduced to a minimum data rate when the object 230 moves to position 2 and then to position 3. After the object 230 has moved out of the detection area 220 of the audio detector 112, in a time period specified as T, a default data rate is used by the audio detector 112.

[0034] How Fig. 3 also shows, the data rate 310 of the camera 111 is minimal when the object 230 is at position 1 and is increased to a maximum data rate when the object 230 moves to position 2 and then to position 3. After the object 230 has moved out of the field of view 210 of the camera 111, in a time period specified as T, a standard data rate is used by the camera 111. As Fig. As shown in Figure 3, the default data rate used by audio detector 112 is lower than the default data rate used by camera 111 during time period T. The maximum data rate of each type of sensor 110 does not have to be the same. According to alternative embodiments, the minimum and maximum data rate may be different for each type of sensor 110.

[0035] Fig.4 is a process flow of a method for performing dynamic bandwidth adjustment between vehicle sensors 110 according to one or more embodiments. The dynamic bandwidth adjustment is performed by the sensors 110 based on control signals from the controller 120. The bandwidth adjustment may be performed based on events (e.g., a detection by one of the sensors 110), periodically, or according to a combination of both. For example, the controller 120 may initially set the bandwidth for each sensor 110 to a default value. The default data rate for two different types of sensors 110 (e.g., camera 111, radar system 113) may be different. The bandwidth of one of the types of sensors 110 (e.g., camera 111) may be set to the maximum value as a default. The default may vary based on environmental conditions or vehicle movement.For example, in rain or snow, when cameras 111 have limited visibility, their data rate can be reduced to the minimum data rate as a default.

[0036] At block 410, the controller 120 receives inputs. The inputs can come from various types of sources. One type of input source is the sensors 110 themselves. This input indicates which sensors 110 are detecting an object and which sensors 110 are not, for example. Other types of input sources include the infotainment system 130 and other vehicle systems 140. These sources provide environmental conditions, vehicle dynamics, and other information that indicate which sensors 110 are of interest, according to the particular scenario in which the vehicle 100 is currently operating.

[0037] Determining the bandwidth adjustments at block 420 may be rule-based, implemented by a machine learning algorithm, or performed via other known techniques. For example, each sensor 110 that detected an object may be instructed to increase its bandwidth to its maximum value, while each sensor 110 that did not detect the object at the same time may be instructed to decrease its bandwidth to its minimum value. This rule-based approach may, for example, be refined to different percentages of the maximum bandwidth based on different confidence levels for the detections. As previously mentioned, determining the bandwidth adjustments at block 420 may be triggered by an event. The event may be detected by one of the sensors 110 or a change in the motion of the vehicle 100 (e.g., a lane change, a change from reversing to driving forward).

[0038] The event may also be a malfunction of one of the sensors 110. A combination of events may also trigger a change in bandwidth. For example, if information from the GPS and mapping systems indicates that the vehicle 100 is approaching a railroad crossing, the controller 120 may also determine that the view of the camera 111 is impaired (e.g., the lens is dirty). In this case, the controller 120 may increase the bandwidth of the audio detectors 112 while decreasing the bandwidth of the camera 111. The audio detectors 112 may detect a ringing tone that accompanies the closing of a barrier or an approaching train. The controller 120 may issue a warning (e.g., via the infotainment system 130) or control vehicle systems 140 accordingly. While the blockage of a sensor 110 (e.g., the camera 111) results in a reduction in its bandwidth, the location of the vehicle 100 (e.g.,railroad crossing) or another event triggers a corresponding increase in the bandwidth of one or more specific sensors 110 (e.g., audio detectors 112).

[0039] The detection-based adjustment may be weighted according to the movement of the vehicle 100 and the environmental conditions. For example, if a sound is present in the environment that saturates the audio detectors 112 and renders other sounds undetectable, the detection by the audio detectors 112 may be weighted less as a factor to increase the data rate from the audio detectors 112. Likewise, if the vehicle 100 is moving forward, the detection by a rear-facing camera 111 may be weighted less as a factor to increase the data rate from that camera 111. At block 430, providing control signals to the sensors 110 from the controller 120 facilitates the bandwidth adjustment based on the determination at block 420.

Claims

[1] A method for performing dynamic bandwidth adjustment between two or more vehicle sensors (110), the method comprising: Receiving inputs at a processor, the input including data from each of the two or more vehicle sensors (110), the two or more vehicle sensors (110) including a camera (111), an audio detector (112), a radar system (113), or a lidar system; Determining, using the processor, a bandwidth at which each of the two or more vehicle sensors (110) is to provide the data; providing a control signal to each of the two or more vehicle sensors (110) by the processor to adjust the bandwidth based on the determination; and wherein determining the bandwidth includes decreasing the bandwidth from a default bandwidth for one or more of the two or more vehicle sensors (110) that cannot detect an object (230) detected by one or more of the two or more vehicle sensors (110). [2] The method of claim 1, wherein receiving the input includes receiving information about environmental conditions and receiving the information about the environmental conditions includes receiving weather information. [3] The method of claim 1, wherein receiving the input includes receiving information about the vehicle (100) and receiving the information about the vehicle (100) includes receiving location information, heading or vehicle dynamics information. [4] The method of claim 1, wherein determining the bandwidth is based on a direction of travel of the vehicle (100) and a position of the two or more sensors (110) on the vehicle (100), or determining the bandwidth is based on an event, wherein the event is a detection by at least one of the two or more vehicle sensors (110) or a change in movement by the vehicle (100). [5] A system for performing dynamic bandwidth adjustment, the system comprising: two or more vehicle sensors (110), wherein the two or more vehicle sensors (110) include a camera (111), an audio detector (112), a radar system (113) or a lidar system; a controller (120) configured to receive inputs, the input including data from each of the two or more vehicle sensors (110), to determine a bandwidth at which each of the two or more vehicle sensors (110) is to provide the data, and to provide a control signal to each of the two or more vehicle sensors (110) to respectively adjust the bandwidth for each of the two or more vehicle sensors (110); and wherein the controller (120) reduces the bandwidth from a standard bandwidth for one or more of the two or more vehicle sensors (110) that do not detect an object (230) sensed by one or more of the two or more vehicle sensors (110). [6] The system of claim 5, wherein the input includes information about environmental conditions and the information about the environmental conditions includes weather information. [7] The system of claim 5, wherein the input includes information about the vehicle (100) and the information about the vehicle (100) includes location information, heading or vehicle dynamics information. [8] The system of claim 5, wherein the controller (120) determines the bandwidth based on a direction of travel of the vehicle (100) and a position of the two or more sensors (110) on the vehicle (100) or determines the bandwidth based on an event, wherein the event is a detection by at least one of the two or more vehicle sensors (110) or a change in movement by the vehicle (100).

Citation Information

Patent Citations

  • System And Method To Operate An Automated Vehicle

    US20160231746A1

  • Variable Object Detection Field-Of-Focus For Automated Vehicle Control

    US20170031363A1