Detection of filtering motorcycles

Abstract

Control system for an autonomous vehicle, which includes the following: one or more processing units programmed to do the following: Receiving one or more image streams; Receiving one or more audio streams; Identifying multiple vehicle images in the one or more image streams; Identifying multiple audible features in the one or more audio streams and Identifying a potential filtering motorcycle by correlating multiple vehicle images and multiple audible features, where one or the several processing facilities are further programmed to do the following: Identifying one or more audible features in the one or more audio streams, wherein the one or more audible features exhibit a motorcycle-specific sound pattern; Identifying motorcycle-specific image patterns among the multiple vehicle images; Attempt to correlate the one or more audible features and the one or more visual patterns; If one or more audible features and one or more visual patterns are correlated, augmenting a confidence score that the potential filtering motorcycle is a potential obstacle.

Description

BACKGROUND OF THE INVENTION

[0001] This invention relates to performing obstacle avoidance in autonomous vehicles. BACKGROUND OF THE INVENTION

[0002] In many congested urban areas such as San Francisco, California, USA, or London, UK, motorcyclists frequently overtake slow or stopped vehicles in traffic by driving between lanes, a process known as filtering. This is a very dangerous driving activity because it is difficult for drivers in neighboring vehicles to detect such activity, especially when lanes converge. Even for future autonomous vehicles, which might have built-in 360-degree sensing systems, it would be challenging to identify fast-moving objects such as motorcycles that change lanes at much higher relative speeds. This difficulty would therefore present challenges for the overall sensing suites and algorithms. It is also problematic where objects or vehicles obscure a sensing system.

[0003] US 2013 / 0 141 576 A1 discloses a controller for an autonomous vehicle having at least one processing device programmed to receive one or more image streams, receive one or more audio streams, identify a plurality of vehicle images in the one or more image streams, and identify a plurality of audible features in the one or more audio streams.

[0004] DE 10 2008 062 735 discloses a motor vehicle safety unit. The motor vehicle safety unit comprises a status detection unit with at least one acoustic sensor and an evaluation unit for detecting dangerous situations using sounds detected by the at least one acoustic sensor. The motor vehicle safety unit also comprises a control unit. The control unit changes the magnitude and / or direction of the motor vehicle's speed when a dangerous situation is detected by the evaluation unit.

[0005] DE 10 2004 045 690 A1 describes a device for determining external sound sources in the environment of a motor vehicle.

[0006] DE 10 2008 061 301 A1 discloses a driver assistance system for a vehicle. The driver assistance system comprises a radar sensor, an optical sensor, and a control unit. The control unit is designed to validate the object data based on the optical data, and the driver assistance system performs a driving intervention depending on the validation of the object data.

[0007] US 2014 / 0 236 414 A1 discloses a computing device. The computing device receives sensor information indicating the respective characteristics of vehicles on a route traveled by a first vehicle. Based on the respective characteristics, the computing device identifies a second vehicle exhibiting aggressive driving behavior, manifesting as unsafe or illegal driving.

[0008] The systems and methods disclosed herein provide an improved approach to detecting motorcyclists who are filtering. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] So that the advantages of the invention will be readily understood, a more detailed description of the invention briefly described above will be made by reference to specific embodiments illustrated in the accompanying drawings. With the understanding that these drawings depict only typical embodiments of the invention and therefore should not be considered limiting its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings: Fig. 1 is a schematic block diagram of a system for implementing embodiments of the invention; Fig. 2 is a schematic block diagram of an exemplary computing device suitable for implementing methods according to embodiments of the invention; Fig. 3 is a schematic diagram illustrating an autonomous vehicle in traffic including a filtering motorcyclist; and Fig. 4 is a process flow diagram of a method for detecting filtering motorcyclists according to an embodiment of the invention. DETAILED DESCRIPTION

[0010] It should be readily understood that the components of the present invention, as generally described herein and illustrated in the figures, could be arranged and designed in a wide variety of different configurations. Thus, the following more particular description of the embodiments of the invention as represented in the figures is not intended to limit the scope of the invention as claimed, but is merely representative of certain examples of presently contemplated embodiments in accordance with the invention. The presently described embodiments will best be understood by reference to the drawings, wherein like parts are designated by like reference numerals throughout.

[0011] Embodiments according to the present invention may be embodied as an apparatus, a method, or a computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "module" or "system." Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer-applicable program code embodied in the medium.

[0012] Any combination of one or more computer-applicable or computer-readable media may be used. For example, a computer-readable medium may include one or more of the following: a portable computer diskette, a hard disk, a random access memory (RAM) device, a read-only memory (ROM) device, an erasable programmable ROM (EPROM or Flash memory) device, a portable compact disc ROM (CD-ROM), an optical storage device, and a magnetic storage device. In selected embodiments, a computer-readable medium may include any non-transitory medium that can contain, store, communicate, disseminate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0013] Computer program code for performing operations of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, or the like, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed entirely on a computer system as a standalone software package, on a standalone hardware unit, partially on a remote computer located some distance from the computer, or entirely on a remote computer or server.In the latter scenario, the remote computer can be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be to an external computer (for example, through the Internet using an Internet service provider).

[0014] The present invention will be described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer program instructions or code.These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions, executing via the processor of the computer or other programmable data processing device, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0015] These computer program instructions may also be stored in a non-transitory, computer-readable medium that can direct a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable medium produce a product that includes instruction means that implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0016] The computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions executing on the computer or other programmable device provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0017] With reference to Fig. 1, a controller 102 may be housed in a vehicle. The vehicle may include any vehicle known in the art. The vehicle may include all structures and features of any vehicle known in the art, including wheels, a drivetrain coupled to the wheels, an engine coupled to the drivetrain, a steering system, a braking system, and other systems known in the art to be incorporated into a vehicle.

[0018] As discussed in more detail herein, the controller 102 may perform autonomous navigation and collision avoidance. In particular, image data and audio data may be analyzed to identify motorcycles that may be filtering, as described in more detail below with reference to Fig. 3 and Fig. 4 is discussed.

[0019] The controller 102 may receive one or more image streams from one or more imaging devices 104. For example, one or more cameras may be mounted on the vehicle and output image streams that are received by the controller 102. The controller 102 may receive one or more audio streams from one or more microphones 106. For example, one or more microphones or microphone arrays may be mounted on the vehicle and output audio streams that are received by the controller 102. The microphones 106 may include directional microphones that have a sensitivity that varies with angle.

[0020] The controller 102 may execute a collision avoidance module 108 that receives the image streams and audio streams, identifies potential obstacles, and takes action to avoid them. In the presently disclosed embodiments, only image and audio data are used to perform collision avoidance. However, other sensors may also be used to detect obstacles, such as radar, lidar, sonar, and the like.

[0021] The collision avoidance module 108 may include a filtering module 110a. The filtering module 110a may include an audio preprocessing module 112a programmed to process the one or more audio streams to identify features that may correspond to a motorcycle. The filtering module 110a may include an image preprocessing module 112b programmed to process the one or more image streams to identify images, movements, behaviors, or other visual cues that may correspond to a motorcycle, motorcyclists, or a riding pattern characteristic of a motorcycle.The filtering module 110a may further include a correlation module 112c that receives audio features identified by the audio preprocessing module 112a and visual cues identified by the image preprocessing module 112b and estimates the presence of a motorcyclist near the autonomous vehicle and / or whether the motorcyclist is filtering and / or the position of the motorcyclist. The function of modules 112a-112c of the filtering module 110a is described below with reference to the method 400 of FIG. Fig. 4 described in more detail.

[0022] The collision avoidance module 108 may further include an obstacle identification module 110b, a collision prediction module 110c, and a decision module 110d. The collision identification module 110b analyzes the one or more image streams and identifies potential obstacles, including people, animals, vehicles, buildings, curbs, and other objects and structures. In particular, the obstacle identification module 110b may identify vehicle images in the image stream.

[0023] The collision prediction module 110c predicts which obstacle images are likely to collide with the vehicle based on its current trajectory or current intended path. The decision module 110d may make a decision to stop, accelerate, turn, etc., to avoid obstacles. The manner in which the collision prediction module 110c predicts potential collisions and the manner in which the decision module 110d takes action to avoid potential collisions may be performed according to any method or system known in the autonomous vehicle art.

[0024] The decision module 110d can control the vehicle's trajectory by actuating one or more actuators 114 that control the vehicle's direction and speed. For example, the actuators 114 can include a steering actuator 116a, an accelerator pedal actuator 116b, and a brake actuator 116c. The configuration of the actuators 116a-116c can follow any implementation of such actuators known in the autonomous vehicle art.

[0025] Fig. Figure 2 is a block diagram illustrating an example computing device 200. Computing device 200 may be used to perform various procedures, such as those discussed herein. Controller 102 may include some or all of the attributes of computing device 200.

[0026] Computing device 200 includes one or more processors 202, one or more memory devices 204, one or more interfaces 206, one or more mass storage devices 208, one or more input / output (I / O) devices 210, and a display device 230, all coupled to a bus 212. Processor(s) 202 include one or more processors or one or more controllers that execute instructions stored in one or more memory devices 204 and / or mass storage devices 208. Processor(s) 202 may also include various types of computer-readable media, such as cache memory.

[0027] The one or more storage devices 204 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 214) and / or non-volatile memory (e.g., read-only memory (ROM) 216). The one or more storage devices 204 may also include rewritable ROM, such as flash memory.

[0028] The one or more mass storage devices 208 include various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, semiconductor memories (e.g., flash memory), and so forth. As in Fig. 2, a particular mass storage device is a hard disk drive 224. Various drives may also be included in the mass storage device(s) 208 to enable reading from and / or writing to the various computer-readable media. The one or more mass storage devices 208 include removable media 226 and / or non-removable media.

[0029] The one or more I / O devices 210 include various devices that allow data and / or other information to be input to or retrieved from the computing device 200. The one or more example I / O devices 210 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.

[0030] The display device 230 includes any type of device capable of displaying information to one or more users of the computing device 200. Examples of the display device 230 include a monitor, a display station, a video projection device, and the like.

[0031] The one or more interfaces 206 include various interfaces that allow the computing device 200 to interact with other systems, devices, or computing environments. The one or more example interfaces 206 include any number of different network interfaces 220, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. One or more other interfaces include a user interface 218 and a peripheral device interface 222. The one or more interfaces 206 may also include one or more peripheral interfaces, such as interfaces for pointing devices (mice, trackpads, etc.), keyboards, and the like.

[0032] Bus 212 allows the one or more processors 202, the one or more memory devices 204, the one or more interfaces 206, the one or more mass storage devices 208, the one or more I / O devices 210, and the display device 230 to communicate with each other, as well as with other devices or components coupled to bus 212. Bus 212 represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so on.

[0033] For illustrative purposes, programs and other executable program components are depicted herein as discrete blocks, although it is understood that such programs and components may reside in different memory components of the computing device 200 and be executed by the one or more processors 202 at various times. Alternatively, the systems and procedures described herein may be implemented in hardware or in a combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) may be programmed to carry out the systems and / or methods described herein.

[0034] Now Fig. Turning to Figure 3, in multi-lane traffic, vehicles 300a-300f may travel in lanes on either side of a dividing line 302 between the lanes. A motorcyclist 304 may travel between the lanes, e.g., on or around the dividing line 302. The motorcyclist 304 will typically be traveling faster than the other vehicles 300a-300f, providing little time to react. Furthermore, the gaps between vehicles 300a-300f, where they are moving slowly, may be small, and as a result, visibility behind each vehicle may be limited.

[0035] In Fig. 3, the vehicle 300a includes the controller 102 housed therein, as well as imaging devices 104 and microphones 106 mounted thereon. As illustrated, the imaging devices 104 may include a rear-facing camera 104a, such as a conventional rearview camera. A forward-facing camera 104b may be mounted to the vehicle 300a. The illustrated embodiment further includes side cameras 104c, 104d mounted to the sides of the vehicle 300a, such as the front doors near the outside mirrors, or to the outside mirrors themselves. The side cameras 104c, 104d may have an optical axis angled relative to the longitudinal axis (front to rear) of the vehicle 300a to provide a wider field of view.

[0036] The microphones 106 may be mounted on a rear side of the vehicle. Each illustrated microphone 106 may be an individual microphone or an array of microphones. The microphones 106 may be directional microphones, which are sensitive to sounds originating from within a limited angular range. The illustrated microphones 106 are located on opposite sides of the vehicle 300a to enable estimation of the direction from which a detected sound originates. In some embodiments, the microphones 106 may include a microphone located within the vehicle's interior, such as a microphone used to detect voice commands or enable hands-free phone calls.

[0037] With reference to Fig.4, the illustrated method 400 may be performed by the controller 102 to provide enhanced collision avoidance with respect to motorcyclists. The illustrated method 400 may be performed, for example, by the collision avoidance module 108 and its filtering module 110a.

[0038] The method 400 may include receiving 402 audio data, such as in the form of audio streams, from the one or more microphones 106. The audio streams may be analog or digital streams and may be filtered or otherwise processed to remove noise. The method 400 may include receiving 404 image data, such as in the form of image streams from one or more cameras 104. The image streams may be in the form of a video signal according to any video coding and transmission protocol known in the art. Alternatively, each image stream may simply be a sequence of individual images according to any format known in the art, received in the order in which they were captured by one of the cameras 104.

[0039] The method 400 may include identifying 406 features in the audio data. In particular, audio streams may be filtered, analyzed, or otherwise processed in the time domain or frequency domain such that motorcycle-specific audio features likely generated by a motorcycle are identified or highlighted. Audio features may be short or long segments of the audio signal that include sounds likely generated by a motorcycle, particularly engine noise. Identifying 406 features in an audio signal may include filtering the audio signal in the time and / or frequency domain such that the output of the filtering is sounds likely originating from a motorcycle. Motorcycles (and motorcycle engines), for example, may have a characteristic frequency band.Accordingly, the audio stream may be bandpass filtered or otherwise filtered so that only noise in this frequency band remains. In many cases, noise from a motorcycle is detected along with engine and road noise from other vehicles, resulting in a low signal-to-noise ratio (SNR). However, due to the distinctive audio characteristics of motorcycles, the audio features corresponding to a motorcycle may be isolated at step 406.

[0040] The method 400 may further include identifying 408 objects, in particular vehicles, in the image data. Identifying 408 objects may, for example, include performing image recognition to identify images of motorcycles or motorcyclists. Identifying 408 objects may include performing video filtering on an image stream in the time domain and / or spatial domain. An image stream may, for example, be analyzed to identify fast-moving objects. In particular, when filtering occurs, the vehicle may be moving at the same speed as four (or more) other moving vehicles, i.e., slowly. In such situations, a motorcyclist is likely to perform filtering and is likely to be traveling much faster than the other vehicles. Accordingly, the image stream is analyzed to identify fast-moving objects.For example, identifying 408 objects may include identifying a set of objects moving at a speed relative to the vehicle 300a that exceeds a threshold. Identifying 408 objects may include identifying objects moving at a relative speed that exceeds a threshold in a particular direction, ie, parallel to the longitudinal axis (front to rear) of the vehicle 300a.

[0041] The method 400 may further include correlating 410 audio features and image objects identified at steps 406, 408. For example, features identified in multiple audio streams may be analyzed to determine a direction to the source of the features. For example, the same feature may be detected by multiple microphones at different times and with different intensities. Indeed, a feature in multiple audio streams emitted by the same source at the same time may be identified by determining that the features are highly correlated, e.g., the correlation function of the two features exceeds a threshold. The direction may be determined from a known distance between the microphones 106 that detected the highly correlated features.

[0042] The direction to the source of a feature may be correlated to a position of an object identified at step 408, i.e., an object located within a threshold angular range from the direction to the feature may be considered to be correlated with that feature. The correlating at step 410 may involve performing a temporal correlation. For example, it may be determined whether a time period at which an audio feature was detected corresponds to a time period at which the images in which an image object was found were detected. Accordingly, correlating audio features and image objects may involve identifying pairs of audio features and image objects that are temporally and / or spatially correlated. An audio feature and an image object that are temporally correlated may, for example, have a score that increases with the degree of spatial correlation between them, i.e.,how closely the direction to the source of the audio features corresponds to the position of the image object. Where the score exceeds a threshold, the audio feature and the image object can be considered correlated and therefore likely correspond to a motorcycle.

[0043] In some embodiments, correlating 410 audio features and image objects may be performed using a machine learning algorithm. For example, a model may be trained by detecting sounds from a motorcycle through an array of microphones 106 and cameras 104 having relative positions similar to the vehicle 300a, where the speed and position of the motorcycle are known. The model may then be trained using machine learning methods to correlate features and image objects detected using the microphones 106 and cameras 104 with the presence of a motorcycle and / or the position of a motorcycle.

[0044] Correlating audio features and image objects may then involve feeding them into the model, which then outputs whether the audio features and image objects correspond to a motorcycle and / or the likely position of a motorcycle that corresponds to the audio features and image objects.

[0045] If it is determined 412 that the correlation step 410 indicates that a motorcycle is nearby, a possible filtering motorcycle is added 414 to a set of potential obstacles. The set of potential obstacles may include other obstacles detected using the imaging devices 104 or any other sensors by any algorithm known in the art.

[0046] Method 400 may include performing 416 collision avoidance with respect to the obstacle set, which may include a likely filtering motorcycle identified according to steps 406-412. Where the obstacle set includes a possible filtering motorcycle, performing collision avoidance may include performing fewer lane changes, signaling longer before lane changes, performing lane changes more slowly, or otherwise driving differently, such as when a possible filtering motorcycle is not detected.

[0047] The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects as illustrative and not restrictive. The scope of the invention is therefore indicated by the appended claims, rather than by the foregoing description. All changes which come within the same meaning or scope as the claims are therefore intended to be embraced within their scope.

Claims

[1] Control system for an autonomous vehicle, comprising: one or more processing devices programmed to: Receiving one or more image streams; Receiving one or more audio streams; identifying a plurality of vehicle images in the one or more image streams; Identifying a plurality of audible features in the one or more audio streams and Identifying a potential filtering motorcycle by correlating the plurality of vehicle images and the plurality of audible features, wherein the one or the plurality of processing devices are further programmed to: identifying one or more audible features in the one or more audio streams, wherein the one or more audible features comprise a motorcycle-specific noise pattern; Identifying motorcycle-specific image patterns among the multiple vehicle images; Attempting to correlate the one or more audible features and the one or more image patterns; if the one or more audible features and the one or more image patterns are correlated, augmenting a confidence score that the potential filtering motorcycle is a potential obstacle. [2] The controller of claim 1, wherein the one or more processing devices are further programmed to receive environmental information from one or more cameras and / or RADAR sensors and / or LIDAR sensors and / or SONAR sensors. [3] The controller of claim 2, wherein the one or more processing devices are further programmed to: in response to identifying the potential filtering motorcycle, preventing the autonomous vehicle from making lane changes. [4] The controller of claim 1, wherein the one or more processing devices are programmed to attempt to correlate the one or more audible features and the one or more image patterns by inputting the one or more audible features and the one or more image patterns to a machine learning algorithm. [5] The controller of claim 4, wherein the one or more processing devices are further programmed to identify motorcycle-specific image patterns among the plurality of vehicle images, comprising identifying objects exhibiting motorcycle-specific motion among the plurality of vehicle images. [6] The controller of claim 4, wherein the one or more processing devices are further programmed to identify the one or more audible features by filtering the one or more audio streams to obtain a filtered signal including the one or more audible features. [7] The controller of claim 1, further comprising a microphone array coupled thereto, wherein the microphone array is configured to generate the one or more audio streams. [8] The controller of claim 1, further comprising at least one coupled microphone configured to generate the one or more audio streams, wherein the at least one microphone is located in a vehicle interior of the autonomous vehicle. [9] The controller of claim 1, further comprising one or more cameras configured to generate the one or more image streams, the one or more cameras coupled to the autonomous vehicle and directed rearwardly of the autonomous vehicle. [10] Collision avoidance procedure comprising: Receiving one or more image streams by a controller of an autonomous vehicle; Receiving, by the controller, one or more audio streams; identifying, by the controller, a plurality of vehicle images in the one or more image streams; Identifying, by the controller, a plurality of audible features in the one or more audio streams and Identifying, by the controller, a potential filtering motorcycle by correlating the plurality of vehicle images and the plurality of audible features, the method further comprising: (a) identifying, by the controller, one or more audible features in the one or more audio streams, wherein the one or more audible features comprise a motorcycle-specific noise pattern; (b) identifying, by the controller, image patterns characteristic of a motorcycle among the plurality of vehicle images; (c) attempting, by the controller, to correlate the one or more audible features and the one or more image patterns; in response to the attempt of (c) being successful, augmenting, by the controller, a confidence score that the potential filtering motorcycle is a potential obstacle. [11] The method of claim 10, further comprising: identifying, by the controller, a set of potential obstacles among the plurality of vehicle images, the set of potential obstacles including the potential filtering motorcycle; Evaluate, through the controller, possible collisions between the autonomous vehicle and the set of potential obstacles and Activating, by the controller, a steering actuator and / or an accelerator pedal actuator and / or a brake actuator of the autonomous vehicle that cause collisions with the set of potential obstacles to be avoided. [12] The method of claim 11, further comprising: In response to identifying the potential filtering motorcycle, preventing the autonomous vehicle from making lane changes by controlling the vehicle. [13] The method of claim 10, wherein attempting to correlate the one or more audible features and the one or more image patterns comprises inputting the one or more audible features and the one or more image patterns into a machine learning algorithm. [14] The method of claim 13, wherein identifying motorcycle-specific image patterns among the plurality of vehicle images comprises identifying objects exhibiting motorcycle-specific motion among the plurality of vehicle images. [15] The method of claim 13, wherein identifying the one or more audible features comprises filtering the one or more audio streams to obtain a filtered signal including the one or more audible features. [16] The method of claim 10, wherein receiving the one or more audio streams comprises receiving the one or more audio streams from a microphone array. [17] The method of claim 10, wherein receiving the one or more audio streams comprises receiving the one or more audio streams from at least one coupled microphone located in a vehicle interior of the autonomous vehicle. [18] The method of claim 10, wherein receiving the one or more image streams comprises receiving the one or more image streams from one or more cameras coupled to the autonomous vehicle and directed rearwardly of the autonomous vehicle.

Images