Dynamic vehicle warning signal generation

Abstract

The vehicle computing system may implement techniques for dynamically modifying warning signals from the vehicle so that an object (e.g., a dynamic object) is notified of vehicle behavior. The vehicle computing system may emit a first warning signal, including an audio signal and / or a visual signal, and may detect an object reaction to the first warning signal. Based on a determination that the object reaction does not substantially change the vehicle's ability to defeat the object, the vehicle computing system may modify the frequency, volume, brightness, color, shape, movement, etc. of the first warning signal and emit a second warning signal. The vehicle computing system may continuously modify the warning signal until the object reacts according to an expected reaction or becomes irrelevant to the vehicle.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to dynamic vehicle warning signal generation. [Background technology]

[0002] This PCT international patent application is a continuation of U.S. Patent Application No. 16 / 656,312, entitled "DYNAMIC VEHICLE WARNING SIGNAL EMISSION," filed October 17, 2019, and claims the benefit of priority, the entire contents of which are incorporated herein by reference.

[0003] Vehicles operated today are often equipped with horns that enable the driver of the vehicle to call attention to the vehicle, such as to warn others of potential dangers in the environment. Conventional vehicle horns are configured to emit a sound at a specific frequency and volume. However, the specific frequency and / or volume of the vehicle horn may often be insufficient to attract the attention of pedestrians, such as those listening to music through headphones or those who are hard of hearing. As such, the vehicle horn may be ineffective in warning others of potential danger. [Brief explanation of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number indicates the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical components or functions.

[0005] [Figure 1] FIG. 1 is an illustration of an environment in which a dynamic warning signal system may be used by an autonomous vehicle to warn an object of a potential conflict between the vehicle and an object in the environment, according to an embodiment of the present disclosure. [Figure 2] FIG. 10 is an illustration of a process for modifying a warning signal emitted by a vehicle based at least in part on a detected object reaction to the emitted warning signal. [Figure 3]FIG. 1 is an illustration showing an environment in which a vehicle emits signals based on a determination that an object is blocking the vehicle's vehicle path, the signals including a warning signal to alert the vehicle of the blockage and an object path signal to indicate the potential path of the object. [Figure 4] FIG. 1 is a block diagram of an example system for implementing the techniques described herein. [Figure 5] FIG. 1 illustrates an exemplary process for emitting different signals to alert an object of a potential conflict between the vehicle and the object. [Figure 6] 10 illustrates another example process for issuing a warning signal based at least in part on the location of a vehicle and the detection of an object associated with the vehicle. [Figure 7] FIG. 1 illustrates an exemplary process for issuing at least one of a warning signal or a route signal based on a determination that an object is blocking the path of a vehicle. DETAILED DESCRIPTION OF THE INVENTION

[0006] The present disclosure is directed to techniques for improving vehicle warning systems. The vehicle warning system may be configured to emit sound and / or light to alert objects (e.g., dynamic objects) in an environment proximate to the vehicle of a potential conflict with the vehicle. The vehicle may include an autonomous or semi-autonomous vehicle. The objects may include pedestrians, bicycles, animals (e.g., dogs, cats, birds, etc.), other vehicles (e.g., cars, trucks, motorcycles, mopeds, etc.), or any other object that may cause a conflict (e.g., a collision) with the vehicle. The vehicle computing system may be configured to identify objects in the environment and determine that a potential conflict between the vehicle and the object may occur. The vehicle computing system may emit a first signal to alert the object of the potential conflict and emit a second (different) signal based on a determination that the object's response did not substantially match the expected response. The vehicle computing system may determine whether the object is moving in a manner that is consistent with the expected response. The vehicle may react according to the response provided, or continue to modify the warning signal until the object is no longer relevant to the vehicle (e.g., the possibility of a collision no longer exists), thereby maximizing safe operation of the vehicle.

[0007] The vehicle computing system may be configured to identify objects in the environment. In some examples, the objects may be identified based on sensor data from sensors (e.g., cameras, motion detectors, lidar, radar, etc.) on the vehicle. In some examples, the objects may be identified based on sensor data received from remote sensors, such as sensors associated with another vehicle or sensors attached to the environment configured to share data with multiple vehicles. In various examples, the vehicle computing system may be configured to determine a classification associated with the object, such as whether the object is a pedestrian, a bicycle, an animal, another vehicle, etc.

[0008] The vehicle computing system may be configured to emit a first warning signal to alert one or more objects in the environment of the presence and / or movement of a vehicle. The first warning signal may include an audio signal and / or a light signal. The first warning signal may include a first set of characteristics, such as frequency, volume, brightness, color, shape, movement, etc. In various examples, the first warning signal may be emitted based on detection of an object in the environment and / or characteristics associated with the detection. In such examples, the characteristics associated with the detection may include a distance between the vehicle and the object, a relative speed between the vehicle and the object, etc. For example, the vehicle computing system may detect a bicycle on the road and determine that the bicycle may not hear a vehicle approaching from behind. The vehicle computing system may emit a warning signal to the cyclist to alert the cyclist of the approaching vehicle so that the cyclist does not swerve or otherwise maneuver.

[0009] In some examples, the first warning signal may be emitted based on a classification, sub-classification (e.g., age, height, etc.), and / or additional characteristics associated with the detected object. In such examples, the vehicle computing system may determine the classification, sub-classification, and / or additional characteristics associated with the detected object and may determine a first set of characteristics associated with the first warning signal based on the classification, sub-classification, and / or additional characteristics. For example, a first warning signal generated to alert pedestrians to vehicle activity may include a lower volume than a first warning signal generated to alert cyclists described in the example above. As another example, a first warning signal generated to attract the attention of automobile drivers may include a higher volume than a first warning signal generated to attract the attention of motorcycle drivers. As yet another example, a first warning signal generated for pedestrians wearing headphones may include a first frequency, and a first warning signal generated for pedestrians looking toward a vehicle (e.g., in an associated direction) may include a second frequency.

[0010] In various examples, the first warning signal may be emitted based on a location associated with the vehicle, such as a location associated with a pedestrian, bicycle, or other object (e.g., a school zone, near a playground, a construction zone, etc.). In some examples, the first warning signal may be emitted based on a speed associated with the vehicle (e.g., less than 15 miles per hour, less than 30 kilometers per hour, etc.). In some examples, the first warning signal may include an electric vehicle warning sound as required by law and / or regulation.

[0011] In some examples, the vehicle computing system may cause a first warning signal to be emitted based on a determination that the detected object is relevant to the vehicle (e.g., a potential conflict may exist between the vehicle and the object, and the object may potentially slow the vehicle's forward movement). In various examples, the vehicle computing system may be configured to determine the object's relevance utilizing techniques described in U.S. Patent Application No. 16 / 193,945, filed November 16, 2018, and entitled "Dynamic Sound Emission For Vehicles," the entire contents of which are incorporated herein by reference. In some examples, the determination of the object's relevance may be based on a location associated with the object being within a threshold distance of a vehicle path. In such examples, the path may correspond to a drivable surface along which the vehicle will travel from a first location to a destination. In some examples, the determination of the object's relevance may be based on the object's potential trajectory intersecting a trajectory associated with the vehicle (e.g., a trajectory associated with the vehicle path). In such examples, the vehicle computing system may determine the potential object's trajectory based on sensor data.

[0012] In various examples, the trajectory and / or intent of an object may be determined using techniques described in U.S. Patent No. 10,414,395, filed September 17, 2019, entitled "Feature-Based Prediction," the entire contents of which are incorporated herein by reference. For example, a vehicle computing system may detect a pedestrian running a red light on a roadway ahead of the vehicle. The vehicle computing system may determine that the pedestrian's trajectory may conflict with the vehicle's trajectory and that a collision between the vehicle and the pedestrian may occur without a correction to one or both trajectories. The vehicle computing system may cause a first warning signal to be emitted to alert the pedestrian of the vehicle's operation on the roadway. In some examples, the vehicle computing system may cause the first warning signal to be emitted simultaneously with or immediately before a correction to the vehicle's trajectory (e.g., yielding to the pedestrian) to maximize safe vehicle operation.

[0013] In various examples, the vehicle computing system may determine an object response to the first warning signal based on the sensor data. In some examples, the response may include a change in the object's trajectory (e.g., an increase in speed, a decrease in speed, a direction away from the vehicle, etc.), a movement of the object's head and / or shoulders, a gesture (e.g., a wave, etc.), the placement of the object's feet, an adjustment to an item the object is holding (e.g., an electronic device, a book, a magazine, or other item, etc.), and / or any other movement indicative of the object responding to the first warning signal, etc.

[0014] In various examples, the vehicle computing system may compare the object response to an expected response (also commonly referred to as an object action) associated with the first warning signal (also commonly referred to as a first signal). In various examples, the vehicle computing system may compare one or more characteristics of the first warning signal (e.g., volume, frequency, luminosity, color, movement (e.g., animated movement, light sequence, etc.), signal shape, etc.) and / or data associated with the object (e.g., object attributes (e.g., classification, location (e.g., toward / moving toward vehicle, away from vehicle, etc.), distance from vehicle, trajectory, etc.), object movement (e.g., walking, running, riding a scooter, specific movement implied by the object's trajectory (e.g., based on speed, etc.), reading a book, talking on a phone, viewing data on an electronic device, interacting with other vehicles, interacting with other objects (e.g., talking to another person, looking inside a stroller, etc.)). The vehicle computing system may be configured to determine the expected response based on the vehicle's behavior during the first warning signal (e.g., eating, drinking, operating a sensory-impaired device (e.g., a cane, hearing aid, etc.), listening to headphones, etc.). In some examples, the vehicle computing system may access a database of expected responses to determine the expected response associated with the first warning signal. In such examples, the expected responses in the database may be stored based, at least in part, on data associated with characteristics of the object and / or the first warning signal. In various examples, the vehicle computing system may utilize machine learning techniques to determine the expected response. In such examples, a model may be trained using training data including a plurality of warning signals and detected responses thereto.

[0015] Based on the comparison of the object response to the expected response, the vehicle computing system may determine whether the object responded as expected (e.g., whether there is a substantial match between the object response and the expected response). In response to determining that the object response substantially matches the expected response, the vehicle computing system may store the encounter (e.g., data associated with the first warning signal and the object response) in a database. In some examples, the database may be used for comparison of future object responses, such as to increase the reliability of the response to the first warning signal, to train a machine learning model, etc.

[0016] In various examples, determining a substantial match between the object response and the expected response may include a threshold number of actions (e.g., one matching action, two matching actions, etc.), a threshold percentage of actions (e.g., 90%, 50%, etc.) match, or the like. In some examples, a substantial match may be determined based on a threshold match and / or a threshold difference between the object response and the expected response. Actions may include trajectory corrections (e.g., increasing speed, decreasing speed, changing direction, etc.), body movements (e.g., foot placement, head rotation, shoulder movement, etc.), gestures, etc. For example, an expected response to a first warning signal may include head and / or shoulder movement and adjusting the position of an electronic device held by the object. An object response may include head movement toward the vehicle. Based on at least the match of head movement, the vehicle computing system may determine that the object response and the expected response substantially match. In another example, the vehicle computing system may determine that the object response matches the expected response 75% at a threshold match of 65%. Based on a determination that the percentage of match meets or exceeds a threshold match, the vehicle computing system may determine that the object response substantially matches the expected response.

[0017] In some examples, determining a substantial match between the object response and the expected response may include determining that a modification to the object's trajectory meets or exceeds a threshold modification. In some examples, the threshold modification may include a modification that renders the object irrelevant to the vehicle (e.g., does not impede the vehicle's progress, does not pose a potential conflict, etc.). In such examples, based at least in part on the determination of the modification, the vehicle computing system may cause the vehicle to proceed along the vehicle's trajectory (e.g., at a planned speed, direction, etc.). In some examples, the threshold modification may include a change in speed and / or direction (e.g., 45 degrees, 90 degrees, etc.) associated with the object's trajectory.

[0018] In response to determining that the object reaction did not substantially match (e.g., less than a threshold number of actions, percentage match, etc.), the vehicle computing system may determine that the object did not react in accordance with the expected reaction. In such an example, the vehicle computing system may determine that the object remains unaware of the vehicle action and / or the presence of the vehicle in the environment. Based on the determination that the object did not react in accordance with the expected reaction, the vehicle computing system may emit a second warning signal. In some examples, the second warning signal may include a signal of a different modality (e.g., light, sound, etc.) than the first warning signal. For example, the first warning signal may include the emission of a sound, and the second warning signal may include the emission of a light.

[0019] In some examples, the second warning signal may include a signal of the same style as the first warning signal. In such examples, the vehicle computing system may modify the frequency, volume, brightness, color, shape, movement, and / or other characteristics of the first warning signal to generate the second warning signal. For example, based on a determination that a detected object did not respond in accordance with an expected response to a first warning signal including a first frequency emitted at 50 decibels, the vehicle computing system may cause a second warning signal including a second frequency to be emitted at 70 decibels. As another example, based on a determination that a detected object did not respond to a first warning signal including red and green illumination, the vehicle computing system may cause a second warning signal including yellow and blue illumination. However, it is understood that the specific volume and color in the foregoing examples are merely illustrative, and that other signal characteristics (e.g., volume, frequency, brightness, color, shape, movement, etc.) are contemplated herein.

[0020] In various examples, the vehicle computing system may compare the second object response to a second expected response associated with the second warning signal. In response to determining that the second object response substantially matches the second expected response, the vehicle computing system may store the object response and / or data associated with the second warning signal in a database of object responses. As described above, in some examples, the database may be used for future comparisons of object responses, such as to increase the reliability of responses to warning signals, to train machine learning models, etc.

[0021] In response to determining that the second object reaction does not substantially match the second expected reaction, the vehicle computing system may cause a third warning signal to be emitted, where the third warning signal may be different from the first warning signal and the second warning signal (e.g., different style, different characteristics, etc.). Continuing with the example above, based on determining that the second object reaction to the second frequency emitted at 70 decibels does not substantially match the second expected reaction, the vehicle computing system may cause the second frequency to be emitted at 90 decibels.

[0022] In various examples, the vehicle computing system may continue to modify (e.g., iteratively modify) the emitted warning signal until the object response substantially matches the expected response to the warning signal. In various examples, the vehicle computing system may continue to modify the emitted warning signal based on a determination that the object is associated with the vehicle. In such examples, a modified warning signal may be emitted based on a determination that the detected object is associated with the vehicle. In some examples, the vehicle computing system may be configured to continuously and / or periodically (e.g., every 0.1 seconds, every 1.0 seconds, before generating a modified warning signal, etc.) determine whether the detected object is associated with the vehicle. For example, the vehicle computing system may determine that the detected object did not respond to the second warning signal in accordance with the second expected response. However, before issuing a third warning signal, the vehicle computing system may determine that the detected object is behind the vehicle and traveling in a different direction from the vehicle. As such, the vehicle computing system may determine that the detected object is no longer associated with the vehicle and decide not to issue the third warning signal.

[0023] In various examples, based on a determination that the detected object has responded in accordance with an expected response and / or a determination that the detected object is unrelated to the vehicle, the vehicle computing system may stop emitting a warning signal. In some examples, based on a determination that the detected object has responded in accordance with an expected response and / or that the detected object is unrelated to the vehicle, the vehicle computing system may cause a first warning signal to be emitted. In such examples, the first warning signal may include a baseline warning signal emitted to alert nearby objects to the presence and / or operation of the vehicle. For example, and as described above, the baseline warning signal may include an electric vehicle warning sound as required by law and / or regulation. As another example, the baseline warning signal may include a sound and / or light emitted based on a location associated with the vehicle.

[0024] In addition to providing warning signals to alert the object to the presence and / or movement of the vehicle, the vehicle computing system may be configured to generate path signals for the object in the environment. In various examples, the path signals may include a proposed path for the object to take to avoid a conflict with the vehicle (e.g., a collision, an obstruction, etc.). In various examples, the vehicle computing system may generate the path signal based on a determination that the object is an obstructing object. In such examples, the vehicle computing system may determine that the object is blocking the vehicle path. The object may block the vehicle path based on a determination that the object is stopped at a location that at least partially blocks the vehicle's progress toward the destination. In some examples, the vehicle computing system may determine that the object is a vehicle based on a determination that the vehicle may not be able to proceed toward the destination while remaining within a drivable path (e.g., a drivable surface along which the vehicle will travel along the path). For example, the object may be stopped at an intersection in the vehicle's path so that the vehicle cannot proceed through the intersection.

[0025] In various examples, based on a determination that the object is a blocking object, the vehicle computing system may be configured to identify potential route options for the blocking object. The potential route options may include clear (e.g., unmanned) paths that the blocking object can follow to move out of the vehicle's path. In some examples, the potential route options may include areas through which the blocking object can move. In some examples, the areas may include areas that the operator of the blocking object cannot see, such as because another object is positioned between the blocking object and the area. Using the above example, the blocking object may be making a left turn at an intersection in front of the vehicle and may be stopped in the first lane at an intersection behind the delivery vehicle. The blocking object cannot see that the area beyond the intersection in the second lane is unmanned, and therefore the blocking object may not know the area into which the blocking object can move to clear the intersection. The vehicle computing system may be configured to identify the area into which the blocking object can move.

[0026] In various examples, based on identifying an area into which the occluding object may move (e.g., a clear path that the occluding object may follow to move out of the vehicle path), the vehicle computing system may cause a path signal to be emitted. The path signal may indicate to an operator of the occluding object that the area is clear. In some examples, the path signal may include a light emitted in the direction of the area, an arrow, or other means by which the vehicle computing system may communicate a clear area into which the occluding object may move.

[0027] The technology described herein may substantially improve the safe operation of autonomous and semi-autonomous vehicles operating within an environment. An increasing number of pedestrians, cyclists, scooter riders, and the like operate on drivable surfaces, often listening to music, podcasts, and the like through headphones. The sounds emitted through the headphones drown out the sounds of nearby vehicles, potentially causing people to be unaware of the presence and / or operation of a vehicle, even one emitting an electric vehicle warning sound. To enhance the awareness, and thus safety, of autonomous and / or semi-autonomous vehicles, the technology described herein recognizes that an object is not responding to a first warning signal emitted by the vehicle and adjusts multiple characteristics of one of the first warning signals in an attempt to alert the object to the vehicle's operation and / or presence. The vehicle computing system continues to modify (e.g., iteratively modify) the warning signal until the object responds according to an expected response or is no longer associated with the vehicle, thereby maximizing the safe operation of the vehicle in the environment.

[0028] The techniques described herein may be implemented in many ways. Exemplary implementations are provided below with reference to the following figures. Although discussed in the context of an autonomous vehicle, the methods, apparatus, and systems described herein may be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In another example, the techniques may be utilized in an aviation or nautical context, or in any system that uses machine vision (e.g., a system that uses image data). Furthermore, the techniques described herein may be used with real data (e.g., captured using a sensor), simulated data (e.g., generated by a simulator), or any combination thereof.

[0029] 1 is an illustration of an environment 100 in which one or more computing systems 102 of an autonomous vehicle 104 (e.g., vehicle 104) may utilize a dynamic warning signal system to alert one or more objects 106 of the presence and / or movement of the vehicle 104 in the environment 100. The computing system 102 may detect the object 106 based on sensor data captured by one or more sensors 108 of the vehicle 104 and / or one or more remote sensors (e.g., sensors mounted on another vehicle 104 and / or sensors mounted in the environment 100 for traffic monitoring, collision avoidance, etc.). The sensors 108 may include data captured by lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.), microphones, time of flight sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc.

[0030] In some examples, the sensor data can be provided to a perception component 110 configured to determine a classification 112 associated with the object 106 (e.g., car, truck, pedestrian, bicycle, motorcycle, animal, etc.). In various examples, the perception component 110 may determine the object's classification 112 based on one or more features associated with the object 106. The features may include the object's 106's size (e.g., width, height, depth, etc.), shape (e.g., geometry, symmetry, etc.), and / or other distinctive features. For example, the perception component 110 may recognize that the size and / or shape of an object 106, such as object 106(1), corresponds to a pedestrian, while the size and / or shape of another object 106, such as object 106(2), corresponds to a bicyclist.

[0031] In various examples, based in part on the detection of one or more objects 106 in the environment 100, the warning signal component 114 of the computing system 102 may be caused to generate and / or emit a first warning signal to alert the object 106 to the presence and / or movement of the vehicle 104. The first warning signal may include an audio signal and / or a visual signal. The first warning signal may include a first set of characteristics, such as frequency, volume, brightness, color, shape, movement, etc. In some examples, the first set of characteristics may include a predetermined set of characteristics. In such examples, the first warning signal may include a baseline warning signal associated with alerting the object 106 to the presence and / or movement of the vehicle 104. For example, the first warning signal may include an electric vehicle warning sound having a predetermined frequency and emitted at a predetermined volume.

[0032] In various examples, the first set of characteristics may be dynamically determined, such as based on one or more real-time conditions associated with the environment 100. The real-time conditions may include data associated with the object 106 (e.g., object attributes (e.g., classification, location (e.g., toward / moving toward a vehicle, away from / facing a vehicle, etc.), distance from a vehicle, trajectory, etc.), object movement (e.g., walking, running, riding a scooter (e.g., a particular movement implied by the object's trajectory, e.g., based on speed), reading a book, talking on a phone, viewing data on an electronic device, interacting with other vehicles, interacting with other objects (e.g., talking to another person, looking into a stroller, etc.), eating, drinking, operating a sensory-impaired device (e.g., cane, hearing aid, etc.), listening to headphones, etc.), environmental factors (noise level in the environment 100, traffic volume, road conditions, etc.), weather (rain, snow, hail, wind, etc.), vehicle considerations (speed, passengers in the vehicle 104, etc.), etc.). For example, a first set of characteristics associated with a first warning signal generated for a pedestrian wearing headphones may include a first frequency, and a first set of characteristics associated with a first warning signal generated for a pedestrian looking in a direction associated with a vehicle may include a second frequency.

[0033] In various examples, the frequency (e.g., one or more frequencies) of the first warning signal may include a frequency (or set / range of frequencies) perceptible to the object 106, such as based on the classification 112 of the object 106. For example, the warning signal component 114 may determine that the object 106 is a dog. Based in part on the classification 112 as a dog, the warning signal component 114 may determine to emit the first warning signal at a frequency perceptible to dogs but not humans, so as to cause the dog to avoid the vehicle 104 and / or the vehicle path. In various examples, the one or more characteristics of the first set (e.g., volume and / or volume range, one or more frequencies, brightness, shape, action, and / or color) may be determined based on the urgency of the warning (e.g., low urgency (e.g., alert), medium urgency (e.g., caution), high urgency (e.g., warning)), the possibility of a conflict between the vehicle 104 and the object 106, the message to be conveyed to the object 106 (e.g., vehicle 104 approaching, please stop, trajectories rapidly converging, etc.).

[0034] In various examples, one or more characteristics of the first set of characteristics may be determined based on object movement (e.g., detected motion associated with object 106). The detected motion may include haptic use of a mobile phone, a determination that a potentially conflicting object is engaged in a conversation (e.g., with another object in proximity to object 106, such as on a mobile phone), a determination that object 106 is wearing headphones, earmuffs, earplugs, or any other device configured to fit in or around the auditory canal, etc.

[0035] In some examples, one or more characteristics of the first set of characteristics may be determined based on weather conditions in the environment. Weather conditions may include rain, wind, sleet, hail, snow, temperature, humidity, large pressure changes, or any other weather phenomenon that may affect the hearing of the object 106 in the environment 100. In various examples, one or more characteristics of the warning signal may be determined based on road conditions in the environment. Road conditions may include the smoothness of the road surface (e.g., concrete, asphalt, gravel, etc.), the number of potholes, uneven terrain (e.g., rumble strips, washboard, road corrugations, etc.), or the like. For example, an object 106 and / or vehicle 104 operating on a gravel road may generate a greater amount of noise than when operating on a smooth surface. An increase in the noise generated by the object 106 and / or vehicle 104 (e.g., the amount of noise impact from driving) may result in a subsequent increase in the determined volume and / or volume range of the warning signal.

[0036] In various examples, one or more characteristics of the first set of characteristics may be determined based on the location of the object 106 in the environment 100. For example, if the object 106 is located in a roadway shared with the vehicle 104, such as indicating an intent to enter the roadway, the volume and / or volume range and / or luminosity may be higher than if the object 106 is located on a sidewalk. As another example, if the object 106 is a pedestrian standing in a median between opposing traffic, the volume and / or volume range and / or luminosity may be higher than if the object 106 is located in a bike lane close to a curb.

[0037] In some examples, one or more characteristics of the first set of characteristics may be determined based on a detected loss of one or more sensors 108 on the vehicle 104. For example, the vehicle computing system may determine that a speaker on the vehicle is not functioning at optimal capacity. In response, the vehicle computing system may increase the volume of a warning signal to compensate for the reduced speaker capacity. As another example, the vehicle computing system may determine that a light on the vehicle 104 is not functioning. In response, the vehicle computing system may increase the brightness and / or flashing frequency of a visual warning signal to compensate for the non-functioning light.

[0038] In various examples, one or more characteristics of the first set of characteristics may be determined based on detecting a passenger within the vehicle 104. In some examples, detecting a passenger may be based on sensor data received from one or more sensors 108 of the vehicle. In some examples, detecting a passenger may be based on a signal indicative of the passenger's presence within the vehicle, such as received from a computing device associated with the passenger. In various examples, the vehicle computing system may reduce the volume and / or volume range and / or frequency of an audio warning signal based on the detection of a passenger, for example, to avoid annoying the passenger with loud noise emissions.

[0039] In various examples, the warning signal component 114 may generate the first warning signal (e.g., determine the first set of characteristics) and / or cause the first warning signal to be transmitted based on a location associated with the vehicle 104. The location may include a school zone, a construction zone, near a playground, a business district, a downtown area, etc. In some examples, the warning signal component 114 may generate and / or cause the first warning signal to be transmitted based on a time of day, a day of the week, a season, a date (e.g., a holiday, etc.), etc. In some examples, the warning signal component 114 may generate the first warning signal and / or cause the first warning signal to be transmitted based on a speed associated with the vehicle 104. In such examples, the warning signal component 114 may cause the first warning signal to be transmitted based on a determination that the vehicle 104 is traveling at or below a threshold speed (e.g., 28 kilometers per hour, 22 miles per hour, 15 miles per hour, etc.).

[0040] In various examples, computing system 102 may be configured to determine that objects 106, such as objects 106(1) and 106(2), are relevant to vehicle 104 (e.g., a potential conflict may exist between vehicle 104 and object 106, or object 106 may slow the forward movement of vehicle 104). In various examples, object relevance may be determined using techniques described in U.S. Patent Application No. 16 / 389,720, filed April 19, 2019, entitled "Dynamic Object Relevance Determination," U.S. Patent Application No. 16 / 417,260, filed May 20, 2019, entitled "Dynamic Object Relevance Determination," and U.S. Patent Application No. 16 / 530,515, filed August 2, 2019, entitled "Object Relevance Determination," the entire contents of which are incorporated herein by reference.

[0041] In some examples, the relevance of an object may be determined based on the distance (D) between the object 106(1) and the drivable surface 116 (e.g., a roadway, a lane on which the vehicle 104 is operating, etc.). In such examples, the object 106 may be determined to be relevant based on the distance (D) being equal to or less than a threshold distance (e.g., 18 inches, 1 foot, 4 meters, etc.). In various examples, the threshold distance may be determined based on the classification 112 associated with the object 106. For example, a first threshold distance associated with a pedestrian may be 1 meter, and a second threshold distance associated with a bicycle may be 5 meters.

[0042] In various examples, an object 106 may be determined to be relevant based on the trajectories of associated objects. In such examples, the computing system 102 may be configured to determine a predicted object trajectory (e.g., an object trajectory), such as based on sensor data. In some examples, the object trajectory may be based on a top-down representation of the environment, utilizing techniques described in U.S. Patent Application No. 16 / 151,607, filed October 4, 2018, and entitled "Trajectory Prediction on Top-Down Scenes," and U.S. Patent Application No. 16 / 504,147, filed July 5, 2019, and entitled "Prediction on Top-Down Scenes based on Action Data," the entire contents of which are incorporated herein by reference. In some examples, the predicted object trajectory may be determined using probabilistic heat maps (e.g., discretized probability distributions), tree search methods, temporal logic formulas, and / or machine learning techniques for predicting object behavior, such as those described in U.S. Patent Application No. 15 / 807,521, filed November 8, 2017, and entitled "Probabilistic Heat Maps for Behavior Prediction," the entire contents of which are incorporated herein by reference.

[0043] In various examples, an object 106 may be associated with a vehicle 104 based on an intersection between the object's trajectory and the vehicle's trajectory. In some examples, an object 106 may be associated based on the predicted locations of the object 106 and the vehicle 104 on their respective trajectories. In some examples, an object 106 may be associated with a vehicle 104 based on a determination that a predicted future object location associated with the object 106 traveling on the object's trajectory is within a threshold distance (e.g., 2 feet, 10 feet, 2 meters, 4 meters, etc.) of a predicted future vehicle location associated with the vehicle 104 traveling on the vehicle's trajectory.

[0044] In various examples, the object 106 may be associated with the vehicle 104 based on a probability of a conflict (e.g., a likelihood of a collision) between the object 106 and the vehicle 104. The probability of a conflict may be based on a determined likelihood that the object 106 will continue the object's trajectory and / or will change the object's trajectory to one that conflicts with the vehicle 104. In some examples, the probability of a conflict may correspond to the possibility (e.g., probability) of a conflict between the vehicle 104 and the object 106 being equal to or greater than a threshold level of conflict (e.g., a threshold probability).

[0045] In various examples, the vehicle computing system may determine the probability of a conflict using a top-down representation of the environment, such as that described in the incorporated U.S. patent applications. In some examples, the vehicle computing system may input the top-down representation of the environment into a machine learning model configured to output a heat map display that predicts probabilities associated with a future position of the object 106 (e.g., predicting the object's trajectory and / or the probabilities associated therewith). In such examples, the vehicle computing system may forward project the movement of the vehicle 104 over time and determine the probability of a conflict between the amount of overlap between the heat map associated with the object 106 and the future position of the vehicle 104 determined by forward projecting over time.

[0046] In some examples, the probability of a conflict may be determined based on the classification 112 associated with the object 106. In such examples, the classification 112 associated with the object 106 may assist in determining the likelihood that the object 106 will maintain or change trajectory. For example, a deer detected on the side of the roadway may be unpredictable and therefore more likely to change trajectory to conflict with the vehicle 104. As such, the deer may be determined to be an object 106 that may potentially conflict with (e.g., be associated with) the vehicle 104.

[0047] In some examples, based on the determination of relevance, the warning signal component 114 of the computing system 102 may generate a first warning signal to alert the associated object 106 in the environment of the presence and / or operation of the vehicle 104. As described above, a first set of characteristics (e.g., frequency, volume, brightness, color, shape, movement, etc.) of the first warning signal may be determined based on the classification 112 associated with the associated object 106.

[0048] In various examples, the first warning signal may be emitted via one or more emitters 118 on the vehicle 104. The emitters 118 may include speakers, lights, displays, projectors, and / or any other devices configured to emit a signal. In some examples, the first warning signal may be emitted in multiple directions around the vehicle (e.g., substantially evenly in front, behind, and to the sides of the vehicle 104). In some examples, the first warning signal may be emitted uniformly in multiple directions around the vehicle 104. For example, an electric vehicle warning sound may be emitted via speakers mounted on the corners of the vehicle 104 and configured to diffuse the first warning signal substantially evenly around the vehicle 104.

[0049] In some examples, the warning signal component 114 may be configured to emit a first warning signal toward the associated object 106 and / or toward the detected object 106 in the environment 100. In some examples, the first warning signal may be emitted via one or more emitters 118 that substantially face a direction in which the object 106 (e.g., the detected object, the associated object, etc.) is detected. For example, the object 106 may be detected in front of and to the right of the vehicle 104 (e.g., on a sidewalk adjacent to the drivable surface 116). Based on the detection of the object in front of and to the right of the vehicle 104, the warning signal component 114 may cause the first warning signal to be emitted via the emitters 118 mounted on the front and right side of the vehicle 104. In some examples, the first warning signal may be transmitted to relevant objects 106 within the environment 100 using beam steering and / or beam forming array techniques, such as those described in U.S. Patent No. 9,878,664, issued May 4, 2017, and entitled "Method for Robotic Vehicle Communication with an External Environment via Acoustic Beam Forming," the entire contents of which are incorporated herein by reference.

[0050] In some examples, the warning signal component 114 may be configured to continuously and / or periodically (e.g., every 0.5 seconds, every 3.0 seconds, etc.) modify the second warning signal to generate a second warning signal. In some examples, the modification to the warning signal may be based at least in part on additional sensor data processed by the computing system 102. For example, the warning signal component 114 may emit a first warning signal at a first time, at least one characteristic of which is determined based on a baseline noise level in the environment. The vehicle computing system may determine an increase in the baseline noise level at a second time, and the warning signal component 114 may generate the second warning signal to be emitted at a higher volume.

[0051] In various examples, the warning signal component 114 may be configured to determine an object reaction to the first warning signal. In such examples, the warning signal component 114 may be configured to determine a real-time object reaction to the warning signal. In some examples, the warning signal component 114 may receive processed sensor data from the perception component 110, such as that associated with an object reaction of the object 106 in the environment 100. The object reaction may be a change (or lack thereof) in the object's trajectory (e.g., an increase in speed, a decrease in speed, a heading away from the vehicle, etc.), a movement of the object's head and / or shoulders, a gesture (e.g., a wave, etc.), the placement of the object's feet, an adjustment of an item held by the object (e.g., an adjustment of an electronic device, book, magazine, or other item), and / or any other movement indicative of the object reacting to the first warning signal, etc.

[0052] The warning signal component 114 may compare the object response to an expected response 120 . The expected response may be based on characteristics of the first warning signal (e.g., volume, frequency, brightness, color, movement (e.g., animated movement, light sequence, etc.), signal shape, etc.) and / or data associated with the object 106 (e.g., object attributes (e.g., classification 112, location (e.g., toward / moving towards vehicle 104, away from / facing vehicle 104, etc.), distance (D) from vehicle 104, object trajectory, etc.), object movement (e.g., walking, running, riding a scooter, (a specific movement implied by the object's trajectory, e.g., based on speed, etc.), reading a book, talking on a phone, viewing data on an electronic device, interacting with other vehicles, interacting with other objects 106 (e.g., talking to other people, looking inside a stroller, etc.), eating, drinking, operating a sensory-impaired device (e.g., cane, hearing aid, etc.), listening to headphones, etc.). In some examples, the warning signal component 114 may access a database 122 containing a plurality of expected responses to determine an expected response 120 associated with the first warning signal. In such examples, the database 122 may include a database 122 containing a plurality of expected responses. The expected response 120 in database 122 may be stored based, at least in part, on data associated with characteristics of object 106 and / or the first warning signal. For example, database 122 may include expected response 120(1) of object 106(1), a pedestrian looking at an electronic device, to include object 106(1) lowering the electronic device it is looking at and / or moving its head and / or shoulders toward vehicle 104 (e.g., emitting the first warning signal). In another example, database 122 may include expected response 120(2) of object 106(2), a cyclist, to include a change in the object's trajectory (e.g., increasing speed, decreasing speed, changing direction, etc.) and / or head movement. In an illustrative example, database 122 may be located on autonomous vehicle 104 separate from computing system 102. In such an example, database 122 may be accessible to computing system 102 via a wired and / or wireless connection.In some examples, database 122 may be remote from computing system 102, such as stored on a remote computing system and accessible via a wireless connection. In yet other examples, database 122 may be located on computing system 102.

[0053] As described with respect to object 106(2), the predicted response 120(2) may include a change in the trajectory associated with object 106(2). The change in trajectory may include a modification to the speed (e.g., increasing speed, decreasing speed, changing speed by a threshold amount, etc.) and / or direction at which object 106(2) moves. In various examples, computing system 102 may determine an updated predicted object trajectory based on additional sensor data from sensors at a time after issuing the first warning signal. In some examples, the updated predicted object trajectory may be determined utilizing a top-down representation of the environment and / or a heat map associated therewith, such as those described in the U.S. patent applications incorporated herein by reference above. In various examples, computing system 102 may determine a modification to the object's trajectory (e.g., a difference between the predicted object trajectory and the updated predicted object trajectory determined after issuing the first warning signal). In some examples, the predicted response may be based on the modification. For example, the expected response may include the object slowing its forward speed or changing its direction of travel (e.g., from a trajectory that intersects with vehicle 104 to a trajectory that is parallel to vehicle 104). Computing system 102 may compare the modified and / or updated object's trajectory with expected response 120(2) to determine whether object 106(2) reacts in accordance with the expected response.

[0054] In some examples, computing system 102 may determine that the first warning signal successfully alerted object 106 based on determining that the object responds (e.g., modifies its behavior) within a threshold time (e.g., 1 second, 2 seconds, etc.) after emitting the first warning signal. In some examples, computing system 102 may store a response associated with the response in database 122. In various examples, despite detecting an unexpected response (e.g., not an expected response), computing system 102 may continue to iteratively modify the warning signal until an expected response is detected. In such examples, the computing system may store data associated with each iteration of the warning signal and the corresponding response (expected and / or unexpected) in database 122.

[0055] In various examples, the vehicle computing system may utilize machine learning techniques to determine the expected response 120. As discussed in more detail below with respect to Figure 4, in some examples, the computing system 102 may include a response training component configured to train a model utilizing machine learning techniques to determine the expected response 120 to the warning signal. In such examples, the model may be trained with training data including a plurality of warning signals and detected responses thereto.

[0056] Based on a comparison of the object response (e.g., actual response, real-time response, etc.) with the expected response 120, the warning signal component 114 may determine whether the object 106 responded as expected to the first warning signal (e.g., whether there is a substantial match between the object response and the expected response 120). In various examples, determining a substantial match between the object response and the expected response 120 may include a threshold number of actions (e.g., one matching action, two matching actions, etc.), a threshold percentage of actions (e.g., 80%, 55%, etc.) match, or the like. In some examples, the substantial match may be determined based on a threshold match and / or a threshold difference between the object response and the expected response 120. The actions may include trajectory corrections (e.g., increasing speed, decreasing speed, changing direction, etc.), body movements (e.g., foot placement, head rotation, shoulder movement, etc.), gestures, etc. For example, the expected response 120(1) to the first warning signal may include head and / or shoulder movement and adjustment of an electronic device held by the object 106(1). The actual object response may include a head movement toward the vehicle 104. Based on at least the match of the head movement, the warning signal component 114 may determine that the object response substantially matches the expected response 120. As another example, for a first warning signal, the expected response 120(2) may include a head movement, a correction to the object's trajectory, and / or the magnitude of the correction to the object's trajectory. The warning signal component 114 may receive an indication, such as from a prediction component, that the speed associated with the object's trajectory has decreased by 5 miles per hour. Based in part on the object's trajectory correction and its magnitude, the warning signal component may determine that the object response substantially matches the expected response 120(2) 85% of the time, which exceeds a 75% threshold percentage of the time.

[0057] In response to determining that the object response substantially matches the expected response 120, the vehicle computing system may store the encounter (e.g., data associated with the first warning signal and the object response) in database 122. In some examples, database 122 may be used for comparison of future object responses, such as to increase the reliability of the object response to the first warning signal, to train a machine learning model, etc.

[0058] In response to determining that the object reaction did not substantially match (e.g., less than a threshold number of actions, percentage match, etc.), the warning signal component 114 may determine that the object 106 did not react in accordance with the expected reaction 120. In such an example, the warning signal component 114 may determine that the object 106 remains unaware of the operation of the vehicle 104 and / or the presence of the vehicle 104 in the environment 100. Based on the determination that the object 106 did not react in accordance with the expected reaction 120, the warning signal component 114 may generate a second warning signal. The second signal may include an audio signal and / or a visual signal. The second warning signal may include a signal of the same or a different modality (e.g., light, sound, etc.) as the first warning signal. For example, the first warning signal may include an audio emission and the second warning signal may include an audio emission. In another example, the first warning signal may include an audio emission and the second warning signal may include an audio emission.

[0059] In various examples, the warning signal component 114 may determine a second set of characteristics (e.g., frequency, volume, brightness, color, movement (e.g., animated movement, light sequence, etc.), signal shape, etc.) of the second warning signal. In some examples, the second set of characteristics may include predetermined modifications to one or more of the characteristics of the first warning signal. In such examples, the warning signal component 114 may modify one or more of the frequency, volume, and / or brightness of the first warning signal to generate the second warning signal. For example, the volume associated with the second warning signal may include an increase of 10 decibels from the first warning signal.

[0060] In various examples, the set of second characteristics may be dynamically determined, such as based on real-time conditions in the environment 100, as described above. In some examples, the warning signal component 114 may process real-time considerations and a failure in response to the first warning signal and determine a set of second characteristics associated with the second warning signal. In some examples, the warning signal component 114 may access the database 122 and determine a set of second characteristics associated with the second signal. In some examples, the set of second characteristics may be stored in the database 122 based on the real-time considerations. In various examples, the set of second characteristics may be determined using machine learning techniques. In some examples, the warning signal component 114 may input the real-time considerations and the first set of characteristics indicating a failure of the first warning signal into a machine learning model configured to output a second set of characteristics. In such examples, the warning signal component 114 may generate a second warning signal according to the output set of second characteristics.

[0061] In various examples, the second set of characteristics may be based in part on an increasing level of urgency, such as from low urgency to medium or high urgency, an increasing likelihood and / or probability of a conflict, such as from medium probability to high probability, or the like. For example, warning signal component 114 may cause a first warning signal to be emitted at a first frequency and a first volume to alert an object 106, such as object 106(1), within a threshold distance of drivable surface 116 of vehicle operation. Based on a determination that object 106(1) did not respond in accordance with expected response 120(1) and that vehicle 104 approached a location associated with object 106(1), the vehicle computing system may determine that the urgency and / or likelihood of a conflict associated with the notification has increased. In response, warning signal component 114 may determine to modify the frequency and / or increase the volume of the first warning signal to generate a second warning signal.

[0062] In various examples, the second set of characteristics may include a predetermined modification to the first set of characteristics. In such examples, one or more of the first set of characteristics may be modified by a predefined amount to determine the second set of characteristics. In some examples, the predefined modification may be stored in database 122 in association with the first warning signal and / or its characteristics, data associated with object 106, etc. For example, the volume associated with the second warning signal may include a 10 decibel increase over the first warning signal. As another example, the second warning signal may include a 100 lumen increase over the first warning signal.

[0063] In various examples, the warning signal component 114 may compare the second object reaction to a second expected reaction 120 associated with the second warning signal. In response to determining that the second object reaction substantially matches the second expected reaction 120, the warning signal component 114 may store the object reaction and / or data associated with the second warning signal in a database 122. As described above, in some examples, the database 122 may be used for comparison of future object reactions, such as to increase the reliability of reactions to warning signals, to train machine learning models, etc.

[0064] In response to determining that the second object response does not substantially match the second expected response 120, the warning signal component 114 may generate and emit a third warning signal, which is different (e.g., different style, different characteristics, etc.) from the first and second warning signals. For example, based on a determination that the object 106 did not respond to a first audio warning signal emitted at 50 decibels and a second audio warning signal emitted at 70 decibels, the warning signal component 114 may determine to emit a visual warning signal as the third warning signal. In response, the warning signal component 114 may determine the color and brightness of the visual warning signal (e.g., a third set of characteristics associated with the third warning signal). The warning signal component 114 may utilize similar or the same techniques as those described above for determining the third set of characteristics associated with the third warning signal.

[0065] In various examples, the vehicle computing system may continue to modify the emitted warning signal until the object response substantially matches the expected response 120 to the warning signal. In various examples, the warning signal component 114 may continue to modify the emitted warning signal based on a determination that the object 106 remains associated with the vehicle 104. In such examples, a modified warning signal may be emitted based on a determination that the object 106 is associated with the vehicle 104. In some examples, the warning signal component 114 may be configured to determine whether the object 106 is associated with the vehicle 104 continuously and / or periodically (e.g., every 0.1 second, every 1.0 second, before generating a modified warning signal, etc.). In some examples, the warning signal component 114 may receive an indication of the relevance of the object 106 from another component of the computing system 102. In various examples, the indication of the relevance of the object 106 may be received in response to a relevance query sent by the warning signal component 114. In such examples, the warning signal component 114 may send a relevance query before generating the modified warning signal, such as to verify the relevance of the object 106 before consuming computing resources to generate the modified warning signal. In such examples, the techniques described herein may improve the functionality of the computing system 102 by making additional computing resources (processing power, memory, etc.) available to other functions of the computing system 102 based at least on the determination of the relevance of the object 106.

[0066] In various examples, based on a determination that the object 106 has responded in accordance with the expected response 120 and / or that the object 106 is unrelated to the vehicle 104, the warning signal component 114 may stop emitting a warning signal. In some examples, based on a determination that the object 106 has responded in accordance with the expected response 120 and / or that the object 106 is unrelated to the vehicle 104, the warning signal component 114 may cause a first warning signal to be emitted. In such examples, the first warning signal may include a baseline warning signal emitted to alert nearby objects of the presence and / or operation of the vehicle 104. For example, and as described above, the baseline warning signal may include an electric vehicle warning sound as required by law and / or regulation. As another example, the baseline warning signal may include a sound and / or light emitted based on a location associated with the vehicle 104 and / or real-time conditions as described above.

[0067] FIG. 2 is a diagram illustrating a process 200 for modifying a warning signal 202 emitted by a vehicle 104 based at least in part on an object reaction 204 of an object 106 to the emitted warning signal 202.

[0068] In operation 206, the process may include emitting a first warning signal 202(1) in an environment, such as environment 100. The first warning signal 202(1) may include an audio signal and / or a visual signal. The first warning signal 202(1) may include a first set of characteristics (e.g., frequency, volume, brightness, color, movement (e.g., animated movement, light sequence, etc.), signal shape, etc.). In some examples, the first set of characteristics associated with the first warning signal 202(1) may be predefined, such as based on a baseline sound emitted from the vehicle 104 to alert the object 106 to the presence and / or movement of the vehicle 104. For example, the first warning signal 202(1) may include an electric vehicle warning sound having a frequency of 528 hertz emitted at 100 decibels.

[0069] In various examples, a vehicle computing system associated with the vehicle 104 may determine the first set of characteristics based on one or more real-time conditions in the environment. As described above, the real-time conditions may include data associated with the object 106 (e.g., object attributes (e.g., classification, location (e.g., toward / moving toward the vehicle, away from / facing the vehicle, etc.), distance from the vehicle, trajectory, etc.), object movement (e.g., walking, running, riding a scooter, (e.g., specific movement implied by the object's trajectory, e.g., based on speed, etc.), reading a book, talking on a phone, viewing data on an electronic device, interacting with other vehicles, interacting with other objects (e.g., talking to another person, looking into a stroller, etc.), eating, drinking, sensory-impaired devices (e.g., canes, hearing aids, etc.), etc.). The vehicle computing system may determine that a detected object 106 is a pedestrian walking in the rain toward the vehicle 104 and is within a threshold distance of the vehicle 104. Due to the object's trajectory, the object's location within the threshold distance of the vehicle 104, and the rain, the vehicle computing system may determine to emit a 1000 Hz signal at 100 decibels.

[0070] In the illustrated example, first warning signal 202(1) may be emitted on a side of vehicle 104 associated with detected object 106. In such an example, first warning signal 202(1) may be configured to alert object 106 in a bike lane, on a sidewalk, and / or in other areas where the vehicle computing system might reasonably expect object 106 to be detected and / or operated (e.g., legal operating areas, typical operating areas, etc.). In some examples, first warning signal 202(1) may be emitted around the vehicle, such as toward the front, rear, right, and left sides. In some examples, first warning signal 202(1) may include an acoustic and / or optical signal in the form of a beam directed toward object 106.

[0071] In operation 208, the vehicle computing system may compare the object reaction 204 to the expected reaction 120 to the first warning signal 202(1). As described above, the vehicle computing system may determine the object reaction 204 based on sensor data from one or more sensors. The sensors may include sensors mounted on the vehicle, sensors mounted on other vehicles, and / or sensors mounted in the environment. The object reaction 204 may include a real-time object reaction to the first warning signal 202(1), such as a change to, or lack thereof, the object's trajectory, position, and / or movement. In an illustrative example, the object reaction 204 includes a substantial lack of movement. For example, the position (e.g., head, shoulders, arms, legs, etc.) of the object 106 remains substantially the same, and the object 106 continues to hold the electronic device in substantially the same position.

[0072] The expected response 120 may be based on characteristics of the first warning signal 202(1) (e.g., volume, frequency, brightness, color, movement (e.g., animation movement, light sequence, etc.), signal shape, etc.) and / or data associated with the object 106 (e.g., object attributes (e.g., classification, location (e.g., toward / moving towards a vehicle, moving away from a vehicle, etc.), distance from a vehicle, trajectory, etc.), object movement (e.g., walking, running, riding a scooter, (e.g., specific movement implied by the object's trajectory, e.g., based on speed, etc.), reading a book, talking on a phone, viewing data on an electronic device, interacting with other vehicles, interacting with other objects (e.g., talking to other people, looking inside a stroller, etc.), eating, drinking, sensory impairment devices (e.g., canes, hearing aids, etc.)). The expected response 120 may be based on the vehicle's response to the first warning signal 202(1), such as a movement of the head toward the vehicle 104 (e.g., a rotation), a movement of the electronic device from a higher position (e.g., in front of the object's head) toward the vehicle 104, or a movement of the electronic device from a lower position (e.g., in front of the object's head). In some examples, the vehicle computing system may access a database including a plurality of expected responses to determine the expected response 120 associated with the first warning signal 202(1). In such examples, the expected response 120 in the database may be stored based, at least in part, on data associated with the object 106 and / or characteristics of the first warning signal 202(1). In various examples, the vehicle computing system may utilize machine learning techniques to determine the expected response 120. In an illustrative example, the expected response 120 includes a movement of the head toward the vehicle 104 (e.g., a rotation), and a movement of the electronic device from a higher position (e.g., in front of the object's head) toward the vehicle 104.

[0073] At operation 210, the process includes issuing a second warning signal based on the object reaction 204 substantially differing from the expected reaction 120. In various examples, the vehicle computing system may determine that the object 106 did not process the first warning signal 202(1) (e.g., the object 106 did not hear and / or see the first warning signal 202(1)) based on the substantial difference between the object reaction 204 and the expected reaction 120.

[0074] The determination of substantial difference may be based on one or more actions associated with the object reaction 204 differing from one or more actions of the expected reaction 120. The actions may include trajectory corrections (e.g., increasing speed, decreasing speed, changing direction, etc.), body movements (e.g., foot placement, head rotation, shoulder movement, etc.), gestures, etc. In some examples, the object reaction 204 may be determined to be substantially different from the expected reaction 120 based on a determination that a threshold number of actions and / or a threshold percentage of actions differ (e.g., a threshold difference). For example, the expected reaction may include foot placement, head movement, and shoulder movement. Based on a determination that the object reaction includes only foot placement, the vehicle computing system may determine that a threshold number of actions (2) has not been met and, therefore, the object reaction is substantially different from the expected reaction.

[0075] The second warning signal 202(2) may include a signal of the same style as the first warning signal 202(1) or a signal of a different style. The second warning signal 202(2) may include a second set of characteristics. In various examples, the vehicle computing system may determine the second set of characteristics based on predefined adjustments to the first set of characteristics. In such examples, the vehicle computing system may modify the frequency, volume, brightness, color, movement, and / or shape of the first warning signal 202(1) based on the predefined adjustments. For example, the first warning signal 202(1) may include an audio signal of a first frequency emitted at 65 decibels. Based on a determination that the object 106 did not respond according to an expected response, the vehicle computing system may increase the volume by 15 decibels and emit the second warning signal 202(2) at 80 decibels.

[0076] In an illustrative example, the vehicle computing system may cause the second warning signal 202(2) to be emitted on the same side as the first warning signal 202(1) (e.g., the side of the vehicle 104 associated with the detected object 106). In other examples, the vehicle computing system may cause the second warning signal 202(2) to be directed toward the object 106, such as in the form of an acoustic and / or optical signal in a beam. In yet other examples, the second warning signal 202(2) may be emitted around the vehicle, such as toward the front, rear, right side, and left side of the vehicle 104.

[0077] In operation 212, the process includes issuing different warning signals until the object response 204 matches (e.g., substantially matches) the expected response 120 or the object 106 is no longer associated with the vehicle 104. In various examples, the vehicle computing system may be configured to continuously modify the warning signals based on the real-time object response to optimize safe operation of the vehicle 104 in the environment.

[0078] In various examples, the vehicle computing system may generate different warning signals 202 in the same and / or different modes. In some examples, the vehicle computing system may emit a predetermined number of warning signals 202 in a first mode and change to a second mode. In such examples, the vehicle computing system may determine that the first mode is ineffective in alerting the object 106 to the presence and / or movement of the vehicle 104. For example, the vehicle computing system may emit three audio warning signals 202 and, based on a substantial difference between the object's reaction 204 to the audio signals and its expected reaction 120 thereto, determine that the audio signals are ineffective because the object 106 is hard of hearing, listens to loud music, etc. The vehicle computing system may modify the fourth warning signal 202 (and subsequent warning signals 202) to a visual warning signal 202, such as a flashing light of various colors, behaviors (e.g., sequences), shapes, and / or intensities.

[0079] In various examples, the vehicle computing device may be configured to continuously and / or periodically (e.g., every 0.2 seconds, every 0.5 seconds, before generating a modified warning signal, etc.) determine whether the object 106 is relevant to the vehicle 104. In some examples, before generating a subsequent warning signal 202 (e.g., after determining that the object response 204 substantially differs from the expected response 120), the vehicle computing device may determine the object's relevance to the vehicle 104. The object 106 may be relevant to the vehicle 104 based on the intersection between the object's trajectory and the vehicle's trajectory. The object's trajectory may be determined utilizing the techniques described above and in the patent applications incorporated by reference. In some examples, the object 106 may be relevant based on the predicted locations of the object 106 and the vehicle 104 on their respective trajectories.

[0080] In some examples, an object 106 may be associated with a vehicle 104 based on a determination that a predicted future object location associated with the object 106 traveling on the object's trajectory is within a threshold distance (e.g., 4 feet, 12 feet, 1 meter, 3 meters, etc.) of a predicted future vehicle location associated with the vehicle 104 traveling on the vehicle's trajectory. In various examples, an object 106 may be determined to be associated with a vehicle 104 based on a location where the object 106 is ahead of the vehicle 104 (e.g., ahead of the vehicle 104 traveling in a direction) and a distance between the object 106 and a drivable surface (e.g., a road) on which the vehicle 104 is traveling on the trajectory (e.g., a distance from the object to the path of the vehicle 104). In such examples, the object 106 may be associated based on a determination that the distance is equal to or less than the threshold distance.

[0081] In various examples, based on a determination that the object 106 has responded in accordance with the expected response 120 and / or that the object 106 is unrelated to the vehicle 104, the vehicle computing device may stop emitting the warning signal 202. In some examples, based on a determination that the object 106 has responded in accordance with the expected response 120 and / or that the object 106 is unrelated to the vehicle 104, the warning signal component 114 may cause a first warning signal 202(1) to be emitted. In such examples, the first warning signal may include a baseline warning signal 202 emitted to alert nearby objects 106 of the presence and / or movement of the vehicle 104. For example, the baseline warning signal may include a sound and / or a light emitted based on a location associated with the vehicle 104 and / or real-time conditions, such as environmental factors, weather conditions, vehicle considerations, data associated with the object 106, etc.

[0082] 3 is an illustration showing an environment 300 in which a vehicle 302, such as vehicle 104, emits signals 304 and 306 based on a determination that an object 308(1) is blocking a vehicle path 310 of the vehicle 302. The signals may include a warning signal 304, such as warning signal 202, to alert the object 106 to the presence and / or movement of the vehicle 302, and a path signal 306 indicating a potential object path for the object 106. A vehicle computing system associated with the vehicle may be configured to detect the object 308, such as the object 106, in the environment based at least in part on sensor data received from one or more sensors of the vehicle and / or one or more remote sensors (e.g., sensors associated with other vehicles, sensors mounted in the environment 300, etc.).

[0083] In various examples, a vehicle computing system, such as computing system 102, may be configured to determine that an object 308 (e.g., obstructing object 308(1)) is blocking a vehicle path 310 (path 310) associated with the movement of the vehicle 302 through the environment 300. In some examples, the vehicle path 310 may include a path for the vehicle 302 from a current location 312 to a destination. In some examples, the vehicle path 310 may include a drivable surface (e.g., a drivable area) associated with the movement of the vehicle 302 to the destination. In some examples, the drivable surface may include a width of the vehicle 302 and / or a safety margin on either side of the vehicle 302. In some examples, the drivable surface may include a width of the lane 314 in which the vehicle 302 is traveling.

[0084] In some examples, the vehicle computing system may determine that blocking object 308(1) blocks vehicle path 310 based on a determination that object location 316 associated with blocking object 308(1) is, at least in part, within the drivable area and / or vehicle path 310. In various examples, the vehicle computing system may determine that blocking object 308(1) blocks vehicle path 310 based on a determination that vehicle 302 cannot proceed around blocking object 308(1) in lane 314. In the illustrated example, blocking object 308(1) is stopped across vehicle path 310, blocking lane 314. In such an example, vehicle 302 may be unable to proceed along vehicle path 310 in lane 314 (or an adjacent lane 318). In other examples, blocking object 308(1) may block a smaller portion of lane 314 (e.g., a smaller percentage of blocking object 308(1) blocking vehicle path 310); however, the vehicle computing system may determine object 308 to be blocking object 308(1) based on a determination that vehicle 302 cannot go around blocking object 308(1) while remaining within lane 314.

[0085] In various examples, the vehicle computing system may be configured to identify a region 320 within which the obstructing object 308(1) may move. In some examples, the region 320 may include locations that are not in the vehicle path 310, the lane 314, and / or an adjacent lane 318. In such examples, the region 320 may include locations within which the obstructing object 308(1) may move so that it no longer obstructs the progress of the vehicle 302 and / or other vehicles / objects 308 traveling in the lane 314 and / or the adjacent lane 318.

[0086] In some examples, region 320 may include a location that may not be visible to the operator of blocking object 308(1), such as based on the blocking of the visible path by another object, such as object 308(2). For example, blocking object 308(1) may be making a left turn at intersection 322 ahead of vehicle 302. Also, blocking object 308(1) may be traveling in the left lane behind object 308(2). Due to the object's location 316 and position (e.g., orientation partially through the left turn), the operator of blocking object 308(1) may not be able to verify that region 320 in the right lane is clear of object 308.

[0087] In various examples, the vehicle computing system may cause a path signal 306 to be emitted from an emitter, such as emitter 118. In some examples, the path signal 306 may include an indication of an object path (route) outside of vehicle path 310. In some examples, the path signal 306 may indicate to an operator of obstructing object 308(1) that area 320 exists and is clear. In an illustrative example, the path signal 306 includes an arrow pointing to area 320, and the path signal 306 is projected (e.g., displayed) on the road surface so that an operator of obstructing object 308(1) can view the path signal 306 from the object's location 316. In another example, the path signal 306 may include a holographic image that provides an indication of area 320 through which obstructing object 308(1) may travel. While depicted as an arrow in FIG. 3 , this is for illustrative purposes only, and other designs, shapes, symbols, etc. are contemplated herein. For example, the path signal 306 may include a flashing sequence of lights configured to indicate a path, such as approach lighting.

[0088] In various examples, the vehicle computing system may emit path signal 306 and may determine that the operator of blocking object 308(1) has not responded in accordance with an expected response (e.g., the operator has not moved blocking object 308(1) toward area 320). In some examples, based on a determination that the operator of blocking object 308(1) has not responded in accordance with an expected response to path signal 306, the vehicle computing system may cause warning signal 304 to be emitted to alert the operator of blocking object 308(1) to the path signal 306 and / or the presence and / or movement of a vehicle. In some examples, the vehicle computing system may emit warning signal 304 to attract the attention of the operator of blocking object 308(1) before emitting path signal 306. Warning signal 304 may include an audio signal and / or a visual signal having a first set of characteristics (e.g., frequency, volume, brightness, color, movement (e.g., animated movement, light sequence, etc.), signal shape, etc.).

[0089] As described above, the vehicle computing system may be configured to detect a reaction (e.g., operator response) of the operator of the blocking object 308(1) to the warning signal 304. The vehicle computing system may detect the operator response based on sensor data collected from sensors associated with the vehicle and / or remote sensors. Due in part to the limited visibility of the vehicle operator provided by the sensor data, the operator response may include body movements, such as head movements, shoulder movements, hand gestures (e.g., waves, etc.). As described above, the vehicle computing system may be configured to compare the operator response to an expected response, such as expected response 120. Further, in examples where the vehicle computing system causes a route signal 306 to be emitted simultaneously with or before the warning signal 304, the expected response may include movement of the blocking object 308(1) toward the area 320. Based on the comparison, the vehicle computing system may determine whether the operator of the blocking object 308(1) is aware of the presence and / or movement of the vehicle 302 and / or route signal 306.

[0090] Based on a determination that the operator of blocking object 308(1) was unaware of the presence and / or movement of vehicle 302 (e.g., the operator's response did not substantially match the expected response) and / or path signal 306, the vehicle computing system may modify warning signal 304. The modification may include a change in style (e.g., from an audio signal to a visual signal, frequency, volume, brightness, color, movement, shape, etc.). In various examples, the vehicle computing system may emit the modified warning signal 304 to alert the operator of blocking object 308(1) to the presence and / or movement of vehicle 302 and / or path signal 306. As described above, the vehicle computing system may be configured to continually modify warning signal 304 until the operator's response substantially matches the expected response or until the vehicle computing system determines that blocking object 308(1) is no longer associated with vehicle 302 (e.g., no longer blocking path 310).

[0091] While FIG. 3 is described with respect to obstructing object 308(1), this is not intended to be limiting, and the vehicle computing system may be configured to generate and transmit path signals 306 for other (non-obstructing) objects 308. For example, the vehicle computing system may detect object 308 within a threshold distance of vehicle 302. Based on the object 308 being within the threshold distance, the vehicle computing system may determine to slow down the forward speed of vehicle 302 if object 308 maintains a first location within the threshold distance to maximize operational safety. The vehicle computing system may determine that if the object moves to a second location outside the threshold location, vehicle 302 may not need to slow down for optimal operational safety. Thus, the vehicle computing system may generate path signals 306 to indicate to object 308 a second location to which object 308 should move so as not to obstruct the forward movement of vehicle 302.

[0092] 4 is a block diagram of an example system 400 for implementing the techniques described herein. In at least one example, the system 400 may include a vehicle 402, such as the vehicle 104.

[0093] The vehicle 402 may include one or more vehicle computing devices 404 (e.g., vehicle computing systems), such as computing system 102, one or more sensor systems 406, such as sensor 108, one or more emitters 408, such as emitter 118, one or more communication connections 410, at least one direct connection 412, and one or more drive systems 414.

[0094] Vehicle computing device 404 may include one or more processors 416 and memory 418 communicatively coupled to the one or more processors 416. In the illustrated example, vehicle 402 is an autonomous vehicle, although vehicle 402 may be any other type of vehicle, such as a semi-autonomous vehicle, or any other system having at least an image capture device (e.g., a smartphone with a camera). In the illustrated example, memory 418 of vehicle computing device 404 stores an orientation component 420, a perception component 422, a planning component 424, one or more system controllers 426, and a warning signal component 428 that includes a signal emission component 430, a response determination component 432, a machine learning component 434, a response database 436, and an object path determination component 438. While depicted in FIG. 4 as residing in memory 418 for illustrative purposes, it is contemplated that the localization component 420, perception component 422, planning component 424, one or more system controllers 426, and warning signal component 428 (and / or the components and / or databases illustrated therein) may additionally or alternatively be accessible to the vehicle 402 (e.g., stored in or otherwise accessible by memory remote from the vehicle 402, e.g., memory 440 of one or more (remote) computing devices 442).

[0095] In at least one example, the localization component 420 may include functionality to receive data from the sensor system 406 to determine the position and / or orientation (e.g., one or more of x-, y-, z-position, roll, pitch, or yaw) of the vehicle 402. For example, the localization component 420 may include and / or request / receive one or more maps of the environment and may continuously determine the location and / or orientation of the autonomous vehicle within the maps. For purposes of this discussion, a map may be any number of data structures modeled in two, three, or N dimensions that may provide information about the environment, such as topology (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some examples, the map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information, etc.), intensity information (e.g., lidar information, radar information, etc.), spatial information (e.g., image data projected onto a mesh, individual “surfels” (e.g., polygons associated with individual colors and / or intensities)), reflectance information (e.g., specular reflectance information, retroreflectance information, BRDF information, BSSRDF information, and the like). In at least one example, the map may include a three-dimensional mesh of the environment. In some examples, the vehicle 402 may be controlled based at least in part on the map. That is, the map may be additionally used in conjunction with the perception component 422 and / or the planning component 424 to determine the location of the vehicle 402, detect objects in the environment, and / or generate a path and / or trajectory to navigate through the environment.

[0096] In some examples, one or more maps may be stored on a remote computing device (such as computing device 442) accessible via network 444. In some examples, multiple maps may be stored, for example, based on characteristics (e.g., type of entity, time of day, day of the week, season of the year, etc.). Storing multiple maps may have similar memory requirements but may increase the speed at which data in the maps can be accessed.

[0097] In various examples, the localization component 420 may be configured to utilize SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping, simultaneous), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc. to receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc. to accurately determine the location of the vehicle 402. In some examples, the localization component 420 may provide data to various components of the vehicle 402 to determine an initial position of the autonomous vehicle 402, to determine the likelihood (e.g., probability) of a conflict with an object, such as whether the object is associated with the vehicle 402, as discussed herein.

[0098] In some examples, perception component 422 may include functionality for performing object detection, segmentation, and / or classification. In some examples, perception component 422 may provide processed sensor data indicating the presence of an object (e.g., an entity, a dynamic object) in proximity to vehicle 402 and / or a classification of the object as an object type (e.g., a car, a pedestrian, a bicyclist, a dog, a cat, a deer, an unknown, etc.). In some examples, perception component 422 may provide processed sensor data indicating the presence of a stationary entity in proximity to vehicle 402 and / or a classification of the stationary entity as a type (e.g., a building, a tree, a road surface, a curb, a sidewalk, an unknown, etc.). In additional or alternative examples, perception component 422 may provide processed sensor data indicative of one or more characteristics associated with a detected object (e.g., a tracked object) and / or an environment in which the object is located. In some examples, characteristics associated with an object may include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), object type (e.g., classification), object velocity, object acceleration, object range (size), etc. Characteristics associated with an environment may include, but are not limited to, the presence of other objects in the environment, the state of other objects in the environment, time of day, day of the week, season, weather (e.g., rain, sleet, hail, snow, temperature, humidity, etc.), darkness / brightness indication, etc.

[0099] In general, the planning component 424 may determine a path for the vehicle 402 to follow to traverse an environment. For example, the planning component 424 may determine various paths and trajectories as well as various levels of detail. For example, the planning component 424 may determine a path for traveling from a first location (e.g., a current location) to a second location (e.g., a destination). For purposes of this discussion, the path may include a sequence of waypoints for traveling between the two locations. As non-limiting examples, the waypoints may include streets, intersections, Global Positioning System (GPS) coordinates, etc. Additionally, the planning component 424 may generate instructions for guiding the vehicle 402 along at least a portion of the path from the first location to the second location. In at least one example, the planning component 424 may determine how to guide the vehicle 402 from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions may be a trajectory, or a portion of a trajectory. In some examples, multiple trajectories may be generated substantially simultaneously (e.g., within technical tolerances) according to a receding horizon technique, and one of the multiple trajectories is selected for the vehicle 402 to navigate.

[0100] In some examples, the planning component 424 may include a prediction component for generating predicted trajectories of objects in the environment. For example, the prediction component may generate one or more predicted trajectories for objects within a threshold distance from the vehicle 402. In some examples, the prediction component may measure the tracking of the object and generate a trajectory for the object based on the observed and predicted behavior. In various examples, the trajectory and / or intention of the object may be determined using techniques described in U.S. Patent No. 10,414,395 and / or U.S. Patent Application Nos. 16 / 151,607, 16 / 504,147, and / or 15 / 807,521, which are incorporated by reference above.

[0101] In at least one example, vehicle computing device 404 may include one or more system controllers 426 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 402. System controller 426 may communicate with and / or control corresponding systems of drive system 414 and / or other components of vehicle 402.

[0102] 4 , the vehicle computing device 404 may include a warning signal component 428. The warning signal component 428 may include a signal transmission component 430. In various examples, the signal transmission component 430 may be configured to determine when to transmit a warning signal, such as warning signals 202 and 304. In some examples, the signal transmission component 430 may determine to transmit the warning signal based on a location associated with the vehicle 402. In such examples, the signal transmission component 430 may receive location data from the localization component 420 and, based on the location data, determine to transmit the warning signal. In various examples, the location may include an area associated with a school zone, an urban area, a business district, a construction zone, and / or other area where pedestrians, scooters, bicycles, etc. typically travel.

[0103] In various examples, the signaling component 430 may determine to emit a warning signal based on a speed associated with the vehicle 402. In some examples, the signaling component 430 may receive an indication of the vehicle speed and / or an indication that the vehicle speed exceeds or falls below a threshold speed (e.g., 15 miles per hour, 30 kilometers per hour, etc.), such as from the perception component 422, and may cause a warning signal to be emitted. In various examples, the warning signal may include an electric vehicle warning sound, such as required by law or regulation to alert objects in the electric (silent) vehicle's environment. In such an example, the signaling component 430 may receive an indication that the speed of the vehicle 402 is at or below a threshold speed and cause a warning signal to be emitted based on the indication.

[0104] In various examples, the signal emission component 430 may determine to emit a warning signal based on detecting an object in the environment and / or determining the object's relevance to the vehicle. In various examples, an object may be determined to be associated with the vehicle based on the distance between the object and the vehicle being less than a threshold distance. In some examples, an object may be determined to be relevant based on a determination that the object's predicted object trajectory intersects with the trajectory of a vehicle associated with the vehicle 402. In such examples, the object may be relevant based on a determination that a conflict (e.g., a collision) may exist between the vehicle 402 and the object.

[0105] The signal emission component 430 may be configured to determine a set of characteristics associated with the warning signal. In various examples, the set of characteristics may be predetermined (e.g., a predetermined frequency, volume, luminosity, color, movement, shape, etc.) based on the location, the speed of the vehicle 402, etc. In such examples, the signal emission component 430 may emit a predetermined warning signal. For example, the warning signal may include an electric vehicle warning sound that is emitted based on a determination that the speed of the vehicle 402 is less than 23 miles per hour. The signal emission component 430 may cause the warning signal to be emitted via one or more emitters 408 having a predetermined frequency and volume designated for the electric vehicle warning sound.

[0106] In various examples, the signal emission component 430 may dynamically determine a set of characteristics associated with the warning signal, such as based on real-time conditions. The real-time conditions may be based on one or more environmental factors (e.g., noise level in the environment 100, traffic volume, proximity of the object 106, etc.), weather conditions (e.g., rain, snow, hail, wind, etc.), vehicle considerations (e.g., speed, passengers in the vehicle 104, etc.), data associated with the object 106 (e.g., object attributes (e.g., classification, location (e.g., toward / moving toward the vehicle, away from the vehicle, etc.), distance from the vehicle, trajectory, etc.), object movement (e.g., walking, running, riding a scooter, etc.), and / or other factors (e.g., based on speed, trajectory, etc.). The activities may include interacting with other vehicles, interacting with other objects (e.g., talking to other people, looking inside a stroller, etc.), eating, drinking, operating a sensory-impaired device (e.g., a cane, hearing aid, etc.), listening to headphones, etc. In such an example, the signal emission component 430 may receive data associated with the environment, such as from the localization component 420 and / or the perception component 422, and may dynamically determine a set of characteristics associated with the alert signal.

[0107] In various examples, the signaling component 430 may be configured to determine that one or more of the environmental factors include an unknown environmental factor. The unknown environmental factor may include a condition in the environment that the signaling component 430 is not trained to understand, such as the number of pedestrians in proximity to the vehicle exceeding a threshold or the noise level in the environment exceeding a threshold noise level. For example, the signaling component 430 may determine that the vehicle is surrounded by a large group of pedestrians. In various examples, the signaling component 430 may determine a set of characteristics based on the unknown environmental factor. In such examples, the set of characteristics may include a set of event-specific characteristics based on the unknown environmental factor. In various examples, the signaling component 430 may cause data associated with the unknown environmental factor, the set of characteristics associated with the warning signal, and the object response to the warning signal to be stored in a response database. In such examples, the data may be utilized to train the system to optimize for expected (desired) responses to the unknown environmental factor.

[0108] In various examples, the signal emission component 430 may be configured to determine a direction in which the warning signal should be emitted. In some examples, the signal emission component 430 may emit the warning signal in a direction surrounding the vehicle, such as in all directions around the vehicle. In some examples, the signal emission component 430 may emit the warning signal in a direction associated with an object in the environment. For example, the warning signal may be emitted via a speaker on the right side of the vehicle toward pedestrians on a sidewalk adjacent to the roadway and / or cyclists on a bicycle lane. In various examples, the signal emission component 430 may emit the warning signal toward a particular object, such as an associated object. In such examples, the warning signal may be emitted via an emitter 408 directed toward the particular object. In some examples, the audio warning signal may be directed toward the particular object using beam steering and / or beam forming array techniques.

[0109] In various examples, the reaction determination component 432 of the warning signal component 428 can be configured to determine an object reaction to the warning signal. The object reaction can include a change in the object's trajectory (e.g., an increase in speed, a decrease in speed, a direction away from the vehicle, etc.), a movement of the object's head and / or shoulders, a gesture (e.g., a wave, etc.), the placement of the object's feet, an adjustment to an item the object is holding (e.g., an adjustment to an item such as an electronic device, a book, a magazine, etc.), and / or any other movement that indicates that the object has reacted to the first warning signal. In various examples, the reaction determination component 432 can receive sensor data from the perception component 422 and determine the object reaction based on the sensor data. In other examples, the reaction determination component 432 can receive an indication of the object reaction, such as from the perception component 422. In such examples, the perception component 422 can process the sensor data to determine the object reaction.

[0110] In various examples, the reaction determination component 432 may compare the detected object reaction to an expected reaction. In various examples, the reaction determination component 432 may access the reaction database 436 (and / or the reaction database 456 on the computing device 442) to determine the expected reaction. In such examples, the expected reaction may be stored based on data associated with the object and / or a set of characteristics associated with the warning signal.

[0111] In various examples, the response determination component 432 may receive the predicted response from the machine learning component 434 or the machine learning component 454 of the computing device 442. In such examples, the machine learning component 434 and / or 454 may be configured to receive data related to a set of characteristics associated with the object and / or warning signal and output the predicted response. The machine learning component 434 and / or 454 may include one or more models trained using training data consisting of multiple object responses to multiple warning signals.

[0112] In various examples, machine learning component 434 and / or 454 may be trained to determine an optimal signal for alerting an object to the presence of a vehicle. The optimal signal may be based on one or more real-time considerations present in the environment, such as environmental factors, weather conditions, object activity, etc. (as described above). The optimal signal may include a signal that has the highest probability of being successful in alerting a particular object to the presence and / or movement of a vehicle.

[0113] In some examples, the machine learning components 434 and / or 454 may be trained utilizing training data including previously emitted warning signals, object reactions thereto, and / or real-time considerations associated therewith. In such examples, the machine learning components 434 and / or 454 may be configured to receive input including the real-time considerations and may output an optimal warning signal (e.g., characteristics associated with the optimal warning signal) and / or a predicted response thereto. In various examples, the training data may include previously emitted signals and associated reactions and / or real-time considerations that were successful in moving objects away from and / or out of the way of the vehicle 402. In such examples, the optimal signal output by the machine learning components 434 and / or 454 to alert a particular object may include a signal that resulted in another object having similar attributes to the particular object reacting in accordance with a predicted response (e.g., staying in the vehicle's path, moving out of the vehicle's path, acknowledging the presence of the vehicle 402, etc.).

[0114] Based on the comparison of the object response to the expected response, the response determination component 432 may be configured to determine whether the object response substantially matches the expected response. In some examples, the object response may substantially match the expected response based on a determination that the object response and the expected response share a threshold number of actions (e.g., characteristics). The threshold number of actions may be one or more actions. In some examples, the threshold number of actions may be dynamically determined based on the scenario (e.g., urgency, object classification, vehicle speed, etc.). For example, an emergency warning signal for an intersecting associated object may involve a threshold of three matching actions to determine that the object response substantially matches the expected response, while a non-emergency warning signal directed at a stationary object located on the sidewalk may involve one matching action to determine a substantial match.

[0115] In some examples, the object response may substantially match the expected response based on determining that a threshold percentage of motion matches between the object response and the expected response. Continuing with the example above, an emergency warning signal to an intersecting associated object may include a 90% match, while a non-emergency warning signal may include a 50% match.

[0116] In response to determining that the object reaction substantially matches the expected reaction, reaction determination component 432 determines that the object has been alerted to the presence and / or movement of vehicle 402. In various examples, based on the determination of substantial match, reaction determination component 432 may cause data associated with the warning signal and the object reaction to be stored in reaction database 436 and / or reaction database 456. In some examples, based on the determination of substantial match, reaction determination component 432 may provide the data associated with the object reaction and the warning signal to machine learning component 434 and / or 454 to train machine learning component 434 and / or 454 and output the associated expected reaction.

[0117] In response to determining that the object response does not substantially match the expected response, the response determination component 432 may modify the set of characteristics associated with the warning signal. In various examples, the response determination component 432 may cause a second (modified) warning signal to be emitted. The second (modified) warning signal may include a second set of characteristics. The second (modified) warning signal may include the same or a different signal style as the first warning signal. In some examples, the response determination component 432 may modify one or more of the frequency, volume, brightness, color, movement, and / or shape of the warning signal to generate the second (modified) warning signal. The response determination component 432 may cause the second (modified) warning signal to be emitted via one or more emitters 408 in a latest attempt to alert the object to the presence and / or movement of the vehicle 402.

[0118] In various examples, the reaction determination component 432 may continue to modify the set of characteristics associated with the warning signal until the object response substantially matches the expected response. In some examples, the reaction determination component 432 may modify the set of characteristics based on a determination of the object's relevance to the vehicle 402. In such examples, the reaction determination component 432 may be configured to determine the object's relevance, such as by utilizing the techniques described above. In various examples, the reaction determination component 432 may determine whether the object is relevant before generating and / or causing the modification signal to be emitted.

[0119] In various examples, the reaction determination component 432 may determine that the associated object is an occluding object. As described above with respect to FIG. 3 , the occluding object may be in a location that at least partially blocks the path of the vehicle 402. In some examples, in response to determining that the associated object is an occluding object, the reaction determination component 432 may send an indication of the occluding object to the object path determination component 438. In various examples, the object path determination component 438 may be configured to determine whether an area on the road is clear such that the occluding object can be moved out of the way of the vehicle 402.

[0120] As described above, the region may include locations that are not in the vehicle path, the lane associated with the vehicle, and / or adjacent lanes. In some examples, the region may include locations where the occluding object may move because it no longer obstructs the progress of the vehicle 402 and / or other vehicles / objects traveling in the same direction (same lane or on the same road) as the vehicle 402. In some examples, the region may include locations where the operator of the occluding object may not be able to see, such as based on the line of sight being blocked by other objects.

[0121] In various examples, the object path determination component 438 may send an indication of a clear area to the signal emission component 430 where the obstructing object may move out of the way of the vehicle. In some examples, the signal emission component 430 may emit a path signal via the emitter 408. The path signal may include an indication of the clear area where the obstructing object may move, a path to it, and / or additional information. In various examples, the path signal may include characteristics (e.g., frequency, volume, brightness, color, movement, shape, etc.) to indicate to the operator that the path to the area is clear. For example, the path guidance signal may include a green arrow projected onto the roadway surface, such as a lane associated with the clear area. As another example, the path guidance signal may be projected onto the roadway surface to appear as a series of sequential flashing lights leading to an accessible area, similar to approach lights.

[0122] In some examples, the reaction determination component 432 may be configured to determine whether the operator's reaction matches an expected reaction. In such examples, the expected reaction may include the operator of the obstructing object following the path signal (e.g., moving toward the area, deviating from the vehicle path). In some examples, based on a determination that the operator's reaction substantially matches the expected reaction and / or that the object is unrelated to the vehicle 402, the reaction determination component 432 may send instructions to the signal emission component 430 and / or the emitter 408 to stop emitting the path signal.

[0123] As can be appreciated, the components discussed herein (e.g., localization component 420, perception component 422, planning component 424, one or more system controllers 426, warning signal component 428 including signal emission component 430, response determination component 432, machine learning component 434, response database 436, and object path determination component 438) are described separately for illustrative purposes. However, the operations performed by the various components may be combined or performed in any other component.

[0124] In some examples, some or all aspects of the components discussed herein may include any model, technique, and / or machine learning technique. For example, in some examples, the components in memory 418 (and memory 440, described below) may be implemented as a neural network. As described herein, an exemplary neural network is a biologically inspired technique that passes input data through a series of connected layers to generate an output. Each layer of a neural network may also constitute another neural network, or may constitute any number of layers (convolutional or not). As may be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of such techniques in which an output is generated based on trained parameters.

[0125] In some examples, the vehicle computing device 404 may utilize machine learning techniques to determine one or more characteristics (e.g., frequency, volume, brightness, color, shape, movement, etc.) of the warning signal to be emitted from the vehicle 402. In some examples, one or more data models may be trained to determine the characteristics of the warning signal based on one or more conditions in the environment. The conditions may include environmental factors (e.g., noise level in the environment, traffic volume, proximity of objects, etc.), weather conditions (e.g., rain, snow, hail, wind, etc.), data associated with the object (e.g., object attributes (e.g., classification, location (e.g., toward / moving towards a vehicle, moving away from a vehicle, etc.), distance from a vehicle, trajectory, etc.), object movement (e.g., walking, running, riding a scooter, (e.g., specific movement implied by the object's trajectory, e.g., based on speed), reading a book, talking on a phone, viewing data on an electronic device, interacting with other vehicles, interacting with other objects (e.g., talking to another person, looking inside a stroller, etc.), eating, drinking, operating a sensory-impaired device (e.g., a cane, hearing aid, etc.), listening to headphones, etc.), etc. In various examples, the data model may be trained to output characteristics of a warning signal based at least in part on conditions present in the environment.

[0126] Although discussed in the context of neural networks, any type of machine learning consistent with this disclosure may be used.For example, machine learning techniques include regression techniques (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally weighted scatterplot smoothing (LOESS)), instance-based techniques (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree techniques (e.g., classification and regression trees (CART), iterative binary tree 3 (ID3), chi-squared automated interaction detection (CHAI)), and others. D), decision stumps, conditional decision trees), Bayesian techniques (e.g., Naive Bayes, Gaussian Naive Bayes, Multinomial Naive Bayes, Average Ordinary Attribute Classifier (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering techniques (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), association rule learning techniques (e.g., Perceptron, Backpropagation, Hopfield Network, Radial Basis Function Network (RBFN)), deep learning techniques (Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), Stacked Auto-Encoders), dimensionality reduction techniques (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), ensemble techniques (e.g., Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forest), Support Vector Machine (SVM), supervised learning, unsupervised learning, semi-supervised learning, etc., but are not limited to these.Additional example architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0127] In at least one example, the sensor system 406 may include a lidar sensor, a radar sensor, an ultrasonic transducer, a sonar sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, etc.), a camera (e.g., RGB, IR, intensity, depth, time of flight, etc.), a microphone, a wheel encoder, an environmental sensor (e.g., temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), etc. The sensor system 406 may include various instances of each of these or other types of sensors. For example, the lidar sensor may include individual lidar sensors positioned on the corners, front, back, sides, and / or top of the vehicle 402. As another example, the camera sensor may include various cameras positioned in various locations on the exterior and / or interior of the vehicle 402. The sensor system 406 may provide input to the vehicle computing device 404. Additionally or alternatively, the sensor system 406 may transmit sensor data over one or more networks 444 to one or more computing devices 442 at a particular frequency, after a predetermined period of time, or in near real time.

[0128] Vehicle 402 may include one or more emitters 408 for emitting light and / or sound, as described above. Emitters 408 in this example include internal audio emitters and video emitters for communicating with passengers of vehicle 402. By way of example and not limitation, internal emitters may include speakers, lights, signs, display screens, touchscreens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.), etc. Emitters 408 in this example also include external emitters. By way of example and not limitation, external emitters in this example include lights (e.g., signal lights, signs, light arrays, etc.) emitted as warning signals and / or to indicate the direction of travel of objects and / or vehicle 402 and / or other indicators of vehicle operation, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, including one or more acoustic beam steering technologies.

[0129] Vehicle 402 may include communications connection 410 that enables communication between vehicle 402 and one or more other local or remote computing devices 442. For example, communications connection 410 may facilitate communication with other local computing devices on vehicle 402 and / or drive system 414. Communications connection 410 may also provide for the vehicle to communicate with other nearby computing devices (e.g., computing device 442, other nearby vehicles, etc.) and / or one or more remote sensor systems 446 to receive sensor data.

[0130] The communications connection 410 may include a physical and / or logical interface for connecting the vehicle computing device 404 to another computing device or network, such as network 444. For example, the communications connection 410 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing devices.

[0131] In at least one example, the vehicle 402 may include one or more drive systems 414. In some examples, the vehicle 402 may have a single drive system 414. In at least one example, if the vehicle 402 has multiple drive systems, the individual drive systems 414 may be located at opposite ends of the vehicle 402 (e.g., the front and rear, etc.). In at least one example, the drive system 414 may include one or more sensor systems for detecting conditions surrounding the drive system 414 and / or the vehicle 402. By way of example and not limitation, the sensor systems may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive system, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other imaging sensors, ultrasonic sensors for acoustically detecting objects in the vicinity of the drive system, lidar sensors, radar sensors, etc. Some sensors, such as the wheel encoders, may be specific to the drive system 414. In some cases, sensor systems on drive system 414 may overlap or complement corresponding systems on vehicle 402 (e.g., sensor system 406).

[0132] The drive system 414 may include a high-voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use in other vehicle systems, a steering system including a steering motor and steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking force to mitigate loss of traction and maintain controllability, an HVAC system, lighting (e.g., head / tail lamps, lighting to illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, a DC / DC converter, a high-voltage junction, a high-voltage cable, a charging system, a charge port, etc.). Additionally, the drive system 414 may include a drive system controller to receive and preprocess data from the sensor system 406 and control the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory 418 may store one or more modules for performing various functions of the drive system 414. Additionally, drive system 414 may include one or more communication connections that enable the respective drive system to communicate with one or more other local or remote computing devices 442 .

[0133] In at least one example, direct connection 412 may provide a physical interface for coupling one or more drive systems 414 with the body of vehicle 402. For example, direct connection 412 may provide for the transfer of energy, fluid, air, data, etc. between drive system 414 and the vehicle. In some examples, direct connection 412 may also releasably secure drive system 414 to the body of vehicle 402.

[0134] In at least one example, the localization component 420, the perception component 422, the planning component 424, the one or more system controllers 426, and the warning signal component 428, and various components thereof, may process sensor data as described above and transmit their respective outputs to the computing device 442 via one or more networks 444. In at least one example, the localization component 420, the perception component 422, the planning component 424, the one or more system controllers 426, and the warning signal component 428 may transmit their respective outputs to the computing device 442 at a particular frequency, after a predetermined period of time, or in near real time.

[0135] In some examples, vehicle 402 may transmit sensor data to computing device 442 over network 444. In some examples, vehicle 402 may receive sensor data from computing device 442 and / or one or more remote sensor systems 446 over network 444. The sensor data may include raw sensor data and / or processed sensor data and / or representations of sensor data. In some examples, sensor data (raw or processed) may be transmitted and / or received as one or more log files.

[0136] The computing device 442 may include a processor 448 and a memory 440 that stores a map component 450, a sensor data processing component 452, a machine learning component 454, and a response database 456 (as described above). In some examples, the map component 450 may include functionality for generating maps of various resolutions. In such examples, the map component 450 may transmit one or more maps to the vehicle computing device 404 for navigation. In various examples, the sensor data processing component 452 may be configured to receive data from one or more remote sensors, such as the sensor system 406 and / or the remote sensor system 446. In some examples, the sensor data processing component 452 may be configured to process the data and transmit the processed sensor data to the vehicle computing device 404, such as for use by the warning signal component 428. In some examples, the sensor data processing component 452 may be configured to transmit raw sensor data to the vehicle computing device 404.

[0137] Processor 416 of vehicle 402 and processor 448 of computing device 442 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, processors 416 and 448 may be comprised of one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into other electronic data that can be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0138] Memories 418 and 440 are examples of non-transitory computer-readable media. Memories 418 and 440 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functionality attributed to the various systems described herein. In various implementations, memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which the ones shown in the accompanying figures are merely examples relevant to the discussion herein.

[0139] In some examples, memories 418 and 440 may include at least working memory and storage memory. For example, working memory may be high-speed memory with limited capacity (e.g., cache memory) used to store data manipulated by processors 416 and 448. In some examples, memories 418 and 440 may include storage memory, which may be slower memory with a relatively large capacity used for long-term storage of data. In some cases, processors 416 and 448 may not be able to operate directly on data stored in storage memory, and the data may need to be loaded into working memory to perform operations based on the data, as discussed herein.

[0140] 4 is illustrated as a distributed system, it should be noted that in alternative examples, components of vehicle 402 may be associated with computing device 442 and / or components of computing device 442 may be associated with vehicle 402. That is, vehicle 402 may perform one or more of the functions associated with computing device 442, and vice versa.

[0141] 5-7 illustrate example processes according to embodiments of the present disclosure. These processes are illustrated as logical flow graphs, each operation of which represents a sequence of actions that may be implemented in hardware, software, or a combination thereof. In the software context, the actions represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described actions. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The described order of actions is not intended to be construed as a limitation, and any number of the described actions may be combined in any order and / or in parallel to perform processing.

[0142] 5 shows an example process 500 for issuing different signals to warn an object of a potential conflict between the vehicle and the object. For example, some or all of process 500 may be performed by one or more components in FIG. 4 as described herein. For example, some or all of process 500 may be performed by vehicle computing device 404.

[0143] At operation 502, the process may include detecting an object in an environment of the vehicle based at least in part on sensor data. The sensor data may include data received from one or more sensors on the vehicle and / or one or more remote sensors, such as sensors mounted on the environment or sensors mounted on other vehicles. In various examples, a vehicle computing system of the vehicle may be configured to determine a classification (e.g., type) associated with the object.

[0144] At operation 504, the process may include issuing a first warning signal based at least in part on detecting the object. In various examples, the first warning signal may be issued based on a determination of the relevance of the object. In such examples, the vehicle computing system may be configured to determine whether the object is relevant to the vehicle. In various examples, the relevance of the object may be determined using techniques described in U.S. Patent Application Nos. 16 / 389,720, 16 / 417,260, and 16 / 530,515, all of which are incorporated herein by reference.

[0145] In some examples, the relevance of an object may be determined based on the distance between the object and the drivable surface on which the vehicle operates (e.g., a roadway, a lane on which the vehicle operates, etc.). In such examples, the object may be determined to be relevant based on the distance being equal to or less than a threshold distance (e.g., 23 inches, 5 feet, 5 meters, etc.). In various examples, the threshold distance may be determined based on a classification associated with the object and / or the object's movement. For example, a first threshold distance associated with a walking pedestrian may be 2 meters, and a second threshold distance associated with a running pedestrian may be 3 meters.

[0146] In various examples, an object may be determined to be relevant based on the trajectory of an object associated with it. In such examples, the vehicle computing system may be configured to determine a predicted object trajectory (e.g., object trajectory), such as based on sensor data. As mentioned above, the object trajectory may be determined according to the techniques described in U.S. Patent Application Nos. 16 / 151,607, 16 / 504,147, and 15 / 807,521, all of which are incorporated herein by reference.

[0147] In various examples, an object may be determined to be relevant to a vehicle based on an intersection between the object's trajectory and the vehicle's trajectory. In some examples, an object may be relevant based on the predicted locations of the object and the vehicle on their respective trajectories. In some examples, an object may be relevant to a vehicle based on a determination that a predicted future object location associated with the object traveling on the object's trajectory is within a threshold distance (e.g., 3 feet, 9 feet, 1.5 meters, 3.3 meters, etc.) of a predicted future vehicle location associated with the vehicle traveling on the vehicle's trajectory.

[0148] In various examples, the first warning signal may include an audio signal and / or a visual signal. The first warning signal may include a first set of characteristics, such as frequency, volume, brightness, color, shape, movement, etc. In some examples, the first set of characteristics may include a predetermined set of characteristics. In such examples, the first warning signal may include a baseline warning signal associated with alerting an object to the presence and / or movement of a vehicle. In various examples, the first set of characteristics may be dynamically determined, such as based on one or more real-time conditions associated with the environment. The real-time conditions may include data associated with the object (e.g., object attributes (e.g., classification, location (e.g., toward / moving towards vehicle, away / facing vehicle, etc.), distance from vehicle, trajectory, etc.), object movement (e.g., walking, running, riding a scooter (e.g., specific movement implied by the object's trajectory, e.g., based on speed), reading a book, talking on the phone, viewing data on an electronic device, interacting with other vehicles, interacting with other objects (e.g., talking to another person, looking inside a stroller, etc.), eating, drinking, operating a sensory-impaired device (e.g., cane, hearing aid, etc.), listening to headphones, etc.), environmental factors (noise level in the environment, traffic volume, road conditions, etc.), weather conditions (rain, snow, hail, wind, etc.), vehicle considerations (speed, passengers in the vehicle, etc.), etc.).

[0149] In various examples, the first warning signal may be emitted in a direction associated with the object. For example, the vehicle computing system may cause the first warning signal to be emitted via an emitter substantially facing the object. In some examples, the first warning signal may be directed toward the object, such as in a beam-type array.

[0150] At operation 506, the process may include determining whether the object reacts in accordance with the expected response to the (first) warning signal and whether the object maintains association with the vehicle. In various examples, the vehicle computing system may verify the object's association with the vehicle before or simultaneously with determining whether the object reacts in accordance with the expected response.

[0151] In various examples, the vehicle computing system may be configured to determine an object response to the first warning signal based on the sensor data. In some examples, the response may include a change in the object's trajectory (e.g., an increase in speed, a decrease in speed, a direction away from the vehicle, etc.), a head and / or shoulder movement of the object, a gesture (e.g., a wave, etc.), a positioning of the object's feet, an adjustment of the object's position relative to an item it holds (e.g., an adjustment of an electronic device, book, magazine, or other item), and / or any other movement indicative of the object responding to the first warning signal.

[0152] In various examples, the vehicle computing system may compare the object response to an expected response associated with the first warning signal. In various examples, the vehicle computing system may compare one or more characteristics (e.g., volume, frequency, brightness, color, shape, movement, etc.) of the first warning signal and / or data associated with the object (e.g., object attributes (e.g., classification, location (e.g., toward / moving toward the vehicle, away from the vehicle, etc.), distance from the vehicle, trajectory, etc.), object movement (e.g., walking, running, riding a scooter, (e.g., specific movement implied by the object's trajectory, such as based on speed), reading a book, talking on a phone, viewing data on an electronic device, interacting with other vehicles, interacting with other objects (e.g., talking to other people, looking inside a stroller, etc.), eating, drinking, etc.). The vehicle computing system may use machine learning techniques to determine the expected response. In such examples, training data including a plurality of warning signals and detected responses thereto may be used to train a model.

[0153] Based on the comparison of the object response to the expected response, the vehicle computing system may determine whether the object responds in accordance with the expected response (e.g., whether there is a substantial match between the object response and the expected response).

[0154] Based on a determination that the object reacts in accordance with the expected reaction (e.g., "Yes" at 506), the process may include storing the object reaction in a reaction database at operation 508. In some examples, the reaction database may be used for comparison of future object reactions, such as to train a machine learning model to increase the reliability of the reaction to the first warning signal. In various examples, data related to the first warning signal, the object reaction, and / or real-time considerations associated with the environment may be used to train a machine learning model to select an optimal signal to notify (e.g., alert) the object.

[0155] Based on a determination that the object does not respond in accordance with the expected response (e.g., "No" at 506), the process may include, at operation 510, issuing a second warning signal based at least in part on the object response. In some examples, the second signal may include modifications to the first warning signal. In such examples, the second signal may include a signal having a different frequency, volume, brightness, color, shape, movement, etc. compared to the first warning signal.

[0156] In some examples, the vehicle computing system may determine the second set of characteristics based on predetermined modifications to frequency, volume, brightness, color, shape, movement, etc. For example, subsequent warning signals may include an increase in volume, such that a first warning signal is an audio signal emitted at 50 decibels, a second warning signal is an audio signal emitted at 60 decibels, etc. In various examples, the vehicle computing system may determine the second set of characteristics based on one or more real-time conditions. As described above, real-time conditions may include environmental factors, weather conditions, vehicle considerations, data associated with the object, etc.

[0157] In various examples, the second warning signal may be emitted in a direction associated with the object. For example, the vehicle computing system may cause the second warning signal to be emitted via an emitter that substantially faces the object. In some examples, the second warning signal may be directed toward the object, such as with an array in the form of a beam.

[0158] In various examples, the vehicle computing system may store data associated with the first warning signal, the object response, and / or real-time considerations based on a determination that the object did not respond in accordance with the expected response (“No” at operation 510). In some examples, the data may be utilized to compare the relative effectiveness of different warning signals, such as to determine an optimized signal for a given scenario.

[0159] After transmitting the second warning signal, the process may include again determining whether the object responded to the warning signal in accordance with the expected response and whether the object remains relevant to the vehicle, as illustrated at operation 506. In various examples, the vehicle computing system may continuously modify (e.g., iteratively modify) the warning signal until the vehicle computing system determines that the object responded in accordance with the expected response or determines that the object is irrelevant to the vehicle. In some examples, the vehicle computing system may modify the warning signal a predetermined number of times until a set of characteristics associated with the warning signal includes a maximum volume, frequency, and / or luminosity. In some examples, the vehicle computing system may cause the last modified warning signal to be transmitted until the object is no longer relevant to the vehicle. In some examples, the vehicle computing system may cause the last modified warning signal to be transmitted for a predetermined period of time (e.g., 30 seconds, 2 minutes, etc.).

[0160] 6 illustrates an example process 600 for issuing a warning signal based at least in part on the location of a vehicle and the detection of an object associated with the vehicle. For example, some or all of process 600 may be performed by one or more components in FIG. 4 as described herein. For example, some or all of process 600 may be performed by vehicle computing device 404.

[0161] At operation 602, the process may include determining a speed and / or location of the vehicle in the environment. In various examples, the vehicle computing system may determine the speed and / or location of the vehicle based on data provided by one or more sensors of the vehicle.

[0162] In operation 604, the process may include determining whether a speed and / or a location is associated with issuing a warning signal. In various examples, the speed of the vehicle may be associated with issuing a warning signal. In such examples, based on the speed of the vehicle being less than a threshold speed, the vehicle computing system may cause the vehicle to issue a warning signal.

[0163] In various examples, the location of the vehicle may be associated with the emission of a warning signal. In various examples, the location may be associated with a classification of the object (e.g., pedestrian, bicycle, etc.). In various examples, the location may be associated with a school zone, proximity to a playground, a downtown area, a business district, a construction zone, a popular cycling route, etc.

[0164] In various examples, a location may be associated with a classification of an object, area, etc. based on a time of day, a day of the week, a date (e.g., a holiday, a season, etc.). In such examples, the vehicle computing system may determine the time of day, the day of the week, the date, etc., and determine whether the location is associated with the emission of a warning signal. For example, the vehicle computing system may operate in a school zone associated with pedestrians. Based on a determination that the day of the week and / or the date is associated with a school day, the vehicle computing system may determine that the location is associated with the emission of a warning signal.

[0165] Based on a determination that the location is not associated with a location associated with the speed and / or issuance of the warning signal (“No” at operation 604), at operation 606 the process may include determining whether an associated object is detected within the environment.

[0166] As described above, an object may be detected based on sensor data received from one or more sensors on the vehicle and / or one or more remote sensors. In various examples, the vehicle computing system may determine whether the object is relevant to the vehicle. As described above, the relevance determination may be based on the distance between the object and the vehicle, the distance between the object and the vehicle path (e.g., a drivable surface, a lane, etc. associated with the vehicle path), the trajectory of one or more objects, the trajectory of the vehicle, etc.

[0167] Based on a determination that the associated object is not detected within the area (“No” at operation 606), the process may include determining the speed and / or location of the vehicle within the environment, as described with respect to operation 602.

[0168] Based on a determination that the vehicle's speed and / or location are relevant to issuing a warning signal ("Yes" at operation 604) or that an associated object is detected in the environment ("Yes" at operation 606), the process may include issuing a first signal (e.g., a first warning signal) based in part on the speed, location, and / or associated object. The first signal may include an audio and / or visual warning signal. The first signal may include a first set of characteristics (e.g., frequency, volume, brightness, color, shape, movement, etc.). The first set of characteristics may include one or more predetermined characteristics and / or one or more dynamically determined characteristics. The predetermined characteristics may be based on the speed, location, and / or associated object (e.g., classification, proximity, etc.). The dynamically determined characteristics may be based on one or more real-time conditions in the environment (e.g., data associated with the object, environmental factors, weather conditions, vehicle considerations, etc.).

[0169] In various examples, the first signal may be emitted in a direction associated with the object. For example, the vehicle computing system may cause the first signal to be emitted via an emitter substantially facing the object. In some examples, the first signal may be directed toward the object by a beam-type array or the like.

[0170] At operation 610, the process may include determining whether the object reacts (to the first signal) according to an expected reaction. In various examples, the vehicle computing system may determine the object reaction, such as based on sensor data. The object reaction may include a change (or lack thereof) in the object's trajectory (e.g., an increase in speed, a decrease in speed, a direction away from the vehicle, etc.), a movement of the object's head and / or shoulders, a gesture (e.g., a wave, etc.), the placement of the object's feet, an adjustment to an item the object holds (e.g., an electronic device, a book, a magazine, or other item, adjusting its position), and / or any other movement that indicates the object has reacted to the first signal.

[0171] The vehicle computing system may compare the object response to the expected response to determine whether the object responds in accordance with the expected response. In some examples, the computing system may access a database of expected responses to determine the expected response. In various examples, the expected response may be stored in the database based on data associated with the object, characteristics of the first signal, etc. In some examples, the vehicle computing system may utilize machine learning techniques to determine the expected response. In such examples, the vehicle computing system may input the data associated with the object and / or characteristics of the first signal into a machine learning model trained to determine the expected response of the object and may receive an output of the expected response.

[0172] As described above, the object may respond according to the expected response based on a substantial match between the (observed, detected) object response and the expected response. The vehicle computing system may determine the substantial match based on the number of actions (e.g., features) and / or the percentage of actions between the match between the object response and the expected response.

[0173] Based on a determination that the object responds in accordance with the expected response (“Yes” at operation 610), at operation 612 the process may include storing the object response in a response database, such as database 122. In some examples, the database may be used for comparison of future object responses, such as to increase the reliability of the object response to the first signal, to train a machine learning model, etc.

[0174] Based on a determination that the object does not respond in accordance with the expected response ("No" at operation 610), the process may include determining whether the object remains relevant to the vehicle at operation 614. The determination of continued relevance may be based on the relevance determination techniques described above, such as in the description of operation 606.

[0175] In various examples, the vehicle computing system may store data associated with the first signal, the object response, and / or real-time considerations based on a determination that the object does not respond in accordance with the expected response (“No” at operation 610). In some examples, the data may be utilized to compare the relative effectiveness of different warning signals, such as to determine an optimized signal for a given scenario.

[0176] Based on a determination that the object is unrelated to the vehicle ("No" at operation 614), at operation 602 the process may include determining the speed and / or location of the vehicle in the environment.

[0177] Based on a determination that the object is associated with the vehicle ("Yes" at operation 614), the process may include emitting a second signal at operation 616 based at least in part on the object reaction. The second signal may include an audio signal and / or a visual signal emitted to alert the object to the presence and / or movement of the vehicle. The second signal may include the same or a different modality as the first signal. The second signal may include a second set of characteristics. In various examples, the second set of characteristics may include one or more characteristics that differ from the first set of characteristics. In some examples, the vehicle computing system may modify the first set of characteristics to generate the second signal (e.g., the second set of characteristics).

[0178] In various examples, the second signal may be emitted in a direction associated with the object. For example, the vehicle computing system may cause the second signal to be emitted via an emitter that substantially faces the object. In some examples, the second signal may be directed toward the object, such as by a beam-type array.

[0179] Based at least in part on emitting the second signal, the process may include determining whether the object reacts (to the second signal) in accordance with an expected response at operation 610. In various examples, the vehicle computing system may continue to modify the emitted signal until the object becomes disassociated with the vehicle or until the object reacts in accordance with an expected response. In some examples, the vehicle computing system may modify the signal a predetermined number of times (e.g., 7 times, 10 times, etc.). In such examples, the vehicle computing system may stop modifying the emitted signal. In some examples, the vehicle computing system may modify the signal for a predetermined period of time. In such examples, the vehicle computing system may stop modifying the signal after the period of time has expired.

[0180] 7 illustrates an example process 700 for issuing at least one of a warning signal or a route signal based on a determination that an object is blocking a vehicle route. For example, some or all of process 700 may be performed by one or more components in FIG. 4 as described herein. For example, some or all of process 700 may be performed by vehicle computing device 404.

[0181] At operation 702, the process may include determining that an object in the environment is blocking a vehicle path. The vehicle computing system may determine that the object is a blocking object based on sensor data received from one or more sensors in the vehicle and / or remote sensors in the environment. In various examples, the vehicle path may include a drivable area of ​​a road associated with a route from the vehicle's current location to a destination. In some examples, the drivable area may include the width of the vehicle and / or a buffer distance (e.g., 12 centimeters, 6 inches, 1 foot, etc.) on either side of the vehicle.

[0182] In various examples, the vehicle computing system may determine that the object blocks the vehicle path based on a determination that the location of the object associated with the object is at least partially within the vehicle path. In various examples, the vehicle computing system may determine that the object blocks the vehicle path based on a determination that the vehicle cannot go around the object in a lane associated with the vehicle path.

[0183] At operation 704, the process may include emitting a first signal based on the object blocking the vehicle path. The first signal may include an audio and / or visual warning signal. The first signal may include a first set of characteristics (e.g., frequency, volume, brightness, color, shape, movement, etc.). The first set of characteristics may include one or more predetermined characteristics and / or one or more dynamically determined characteristics. The predetermined characteristics may be based on speed, location, and / or associated object (e.g., classification, proximity, etc.). The dynamically determined characteristics may be based on one or more real-time conditions in the environment (e.g., data associated with the object, environmental factors, weather conditions, vehicle considerations, etc.).

[0184] In various examples, the first signal may be emitted in a direction associated with the object. For example, the vehicle computing system may emit the first signal via an emitter that substantially faces the object. In some examples, the first signal may be directed toward the object by a beam-type array or the like.

[0185] At operation 706, the process may include determining whether the object reacts (to the first signal) according to an expected reaction. In various examples, the vehicle computing system may determine the object reaction, such as based on sensor data. The object reaction may include a change (or lack thereof) in the object's trajectory (e.g., increasing speed, decreasing speed, moving away from the vehicle, etc.), movement of the object's head and / or shoulders, a gesture (e.g., a wave, etc.), placement of the object's feet, adjustment of the object's position to an item it holds (e.g., adjusting the position of an electronic device, book, magazine, or other item), and / or any other movement indicative of the object reacting to the first signal.

[0186] The vehicle computing system may compare the object response to the expected response to determine whether the object responds in accordance with the expected response. In some examples, the computing system may access a database of expected responses to determine the expected response. In various examples, the expected response may be stored in the database based on data associated with the object, characteristics of the first signal, etc. In some examples, the vehicle computing system may utilize machine learning techniques to determine the expected response. In such examples, the vehicle computing system may input the data associated with the object and / or characteristics of the first signal into a machine learning model trained to determine the expected response of the object and may receive an output of the expected response.

[0187] As described above, the object may respond according to the expected response based on a substantial match between the (observed, detected) object response and the expected response. The vehicle computing system may determine the substantial match based on the number of actions (e.g., features) and / or the percentage of actions between the match between the object response and the expected response.

[0188] Based on a determination that the object does not respond according to the expected response (“No” at operation 706), the process may include transmitting a second signal toward the object at operation 708. The second signal may include an audio and / or visual warning signal. The second signal may include a second set of characteristics (e.g., frequency, volume, brightness, color, shape, movement, etc.). The second set of characteristics may include one or more predetermined characteristics and / or one or more dynamically determined characteristics. The predetermined characteristics may be based on speed, location, and / or associated objects (e.g., classification, proximity, etc.). The dynamically determined characteristics may be based on one or more real-time conditions in the environment (e.g., data associated with the object, environmental factors, weather conditions, vehicle considerations, etc.).

[0189] In various examples, the second signal may be emitted in a direction associated with the object. For example, the vehicle computing system may cause the second signal to be emitted via an emitter facing the object. In some examples, the second signal may be directed toward the object by a beam-type array or the like.

[0190] Based on a determination that the object reacts according to the expected response (“Yes” at operation 706), at operation 710 the process may include determining whether an area is identified for the object to move from the vehicle path. In some examples, the area may include locations not in the vehicle path, a lane associated with the vehicle, and / or adjacent lanes. In such examples, the area may include locations where the blocking object may move to avoid impeding the progress of the vehicle and / or other vehicles / objects traveling in the lane and / or adjacent lanes. In some examples, the area may include a size large enough for the object to move to avoid impeding the progress of the vehicle and / or other vehicles / objects. In some examples, the area may include locations that the operator of the object may not be able to see, such as based on being blocked from view by other objects.

[0191] Based on a determination that an area exists for the object to move out of the vehicle path ("Yes" at operation 710), the process may include emitting a third signal at operation 712 that includes an indication of the area. In some examples, the third signal may include an indication of the object's path (route) out of the vehicle path. In some examples, the third signal may indicate to an operator of the object that the area exists and is clear. In various examples, the third signal may include a symbol or other indicator, such as an arrow, to indicate to an operator of the object a location associated with the area. In various examples, the symbol or other indicator may be projected onto a drivable surface proximate the object and / or area. In some examples, the symbol or other indicator may include a holographic image projected into the field of view of the operator of the object.

[0192] At operation 714, the process may include determining that the object is irrelevant to the vehicle. Further, the vehicle computing system may determine that the object is irrelevant based on a determination that no area exists for the object to move off the path ("No" at operation 710). In various examples, the determination that the object is irrelevant to the vehicle may be based on a determination that the object no longer blocks the vehicle path. In such examples, the vehicle computing system may determine that the object has moved into or towards an area (e.g., pursuant to a third signal) and / or has moved to another area outside the vehicle path.

[0193] At operation 716, the process may include controlling the vehicle according to the head. In various examples, controlling the vehicle according to the vehicle path may be based on traffic rules, laws, etc. For example, the vehicle computing system may determine that a traffic light has turned red by the time the object no longer blocks the vehicle path. Based on determining that the light is red, the vehicle may maintain its position and wait for the light to turn green.

[0194] (Example section) A. A vehicle including a sensor; an emitter; one or more processors; and one or more computer-readable media having stored thereon instructions that, when executed, cause the vehicle to: determine an object in an environment associated with the vehicle based at least in part on sensor data from the sensor; determine the object is associated with progression of the vehicle based at least in part on a trajectory of the object; emit, via the emitter, a first signal based at least in part on determining the object is associated with progression of the vehicle, the first signal including a first characteristic; determine an object reaction to the first signal based at least in part on the sensor data; and emit a second signal based at least in part on the object reaction, the second signal including a second characteristic different from the first characteristic.

[0195] B. The vehicle of paragraph A, wherein transmitting the second signal is further based at least in part on determining that the object response is different from an expected response, the object response being the first object response and the expected response being the first expected response, and the instructions cause the vehicle to determine a second object response to the second signal and store data associated with the second object response in a database based at least in part on the second object response.

[0196] C. The vehicle of paragraph A or B, wherein the first characteristic includes at least one of one or more first frequencies, one or more first volume levels, one or more first brightness levels, one or more first colors, one or more first shapes, or one or more first movements, and the second characteristic includes at least one of one or more second frequencies, one or more second volume levels, one or more second brightness levels, one or more second colors, one or more second shapes, or one or more second movements.

[0197] D. The vehicle of any of paragraphs A-C, wherein the first characteristic or the second characteristic is based at least in part on a movement associated with the object, the movement including at least one of listening to headphones, viewing data on a mobile device, reading a book, talking on a cell phone, eating, drinking, a specific movement implied by a predicted trajectory, operating a device for a sensory impaired person, the object's head pointing away from a location associated with the vehicle, interacting with other vehicles in the environment, or interacting with other objects in proximity to the object.

[0198] E. The vehicle of any of paragraphs A-C, wherein the instructions further cause the vehicle to determine the expected response based at least in part on at least one of machine learning techniques or expected response data stored in a database, wherein the expected response is associated with at least one of the first characteristic, a classification of the object, a location of the object, or a movement of the object.

[0199] F. A computer-implemented method, comprising: detecting an object in an environment based on sensor data from a sensor on a vehicle, the object including object attributes; causing a first signal to be emitted via an emitter of the vehicle at a first time based at least in part on the object attributes, the first signal including a first characteristic; determining an object response of the object at a second time after the first time based at least in part on additional sensor data from the sensor; and causing a second signal to be emitted via an emitter of the vehicle based on the object response, the second signal including a second characteristic.

[0200] G. The computer-implemented method of paragraph F, wherein the sensor data is first sensor data, the method further including: determining that the object is associated with the vehicle's movement; and, based at least in part on determining that the object is associated with the vehicle's movement, causing at least one of the first signal or the second signal to be emitted in a direction associated with the object.

[0201] H. The computer-implemented method of either paragraph F or G, wherein causing the second signal to be emitted is further based on determining that the object continues to impede the vehicle's progress after the second time.

[0202] I. The computer-implemented method of any of paragraphs F-H, wherein at least one of the first characteristic or the second characteristic is based at least in part on at least one of an environmental factor in the environment, a location of the vehicle in the environment, a speed of the vehicle in the environment, a motion associated with an object, a relative position of the object with respect to the vehicle, a day on which the vehicle is operating, a year on which the vehicle is operating, or a day of the week on which the vehicle is operating.

[0203] J. The computer-implemented method of paragraph I, wherein the movements include one or more of listening to headphones, viewing data on a mobile device, reading a book, talking on a cell phone, eating, drinking, a specific movement implied by a predicted trajectory, operating a device for a sensory impaired person, the object's head pointing away from a location associated with the vehicle, interacting with other vehicles in the environment, or interacting with other objects in proximity to the object.

[0204] K. The computer-implemented method of any of paragraphs F through I, wherein causing the second signal to be emitted is further based at least on determining that the object response differs from an expected response, wherein the object response is the first object response and the expected response is the first expected response, the method further including determining a second object response of the object to the second signal and storing data associated with at least one of the second signal or the second object response in a database based at least in part on the second object response.

[0205] L. The computer-implemented method of any of paragraphs F-K, wherein the emitter includes at least one of a speaker, a light, or a projector.

[0206] M. The computer-implemented method of any of paragraphs F through L, wherein the method further includes repeatedly issuing additional signals until at least one of determining an object not associated with the vehicle, determining that a timer associated with the warning signal has expired, or determining that the number of issued warning signals meets or exceeds a threshold.

[0207] N. The computer-implemented method of any of paragraphs F through L, the method further including: determining that the object at least partially obstructs a head associated with the vehicle; identifying a location for the object to move to, the location being outside the head and clear of other objects; and, based at least in part on identifying the location, causing a third signal to be emitted via the second emitter, the third signal providing instructions to the object of a location for the vehicle to move to.

[0208] O. A system or device comprising: a processor; and a non-transitory computer-readable medium storing instructions that, when executed, cause the processor to perform the computer-implemented method of any of paragraphs F through M.

[0209] P. A system or device comprising: means for processing; and means for storing, coupled to the means for processing, wherein the means for storing comprises instructions for configuring one or more devices to perform the computer-implemented method set forth in any of paragraphs F through M.

[0210] Q. One or more non-transitory computer-readable media storing instructions that, when executed, cause a vehicle to perform actions, including detecting an object in an environment based at least in part on sensor data from a sensor, the object including object attributes, emitting a first signal via an emitter of the vehicle at a first time based at least in part on the object attributes, the first signal including a first characteristic, determining an object response of the object at a second time after the first time based at least in part on additional sensor data from the sensor, emitting a second signal via an emitter of the vehicle based on the object response, the second signal including a second characteristic.

[0211] R. The one or more non-transitory computer-readable media described in paragraph Q, wherein the operations further include determining that the object is associated with the vehicle's movement and, based at least in part on determining that the object is associated with the vehicle's movement, causing at least one of the first signal or the second signal to be emitted in a direction associated with the object.

[0212] S. The one or more non-transitory computer-readable media described in paragraph Q or R, wherein the operation further includes determining a movement associated with the object and determining the first characteristic based at least in part on the movement, the movement including one or more of listening to headphones, viewing data on a mobile device, reading a book, talking on a cell phone, eating, drinking, a specific movement implied by the predicted trajectory, operating a sensory impairment device, the object's head turning in a direction away from a location associated with the vehicle, interacting with another vehicle in the environment, or interacting with another object proximate to the object.

[0213] T. The one or more non-transitory computer-readable media of any of paragraphs Q to S, wherein the operations further include utilizing a machine learning model to determine a second characteristic, the machine learning model being trained at least in part based on signals emitted prior to causing additional objects having similar attributes to perform an action to unblock the vehicle.

[0214] U. The one or more non-transitory computer-readable media of any of paragraphs Q through T, wherein the object attribute includes an object trajectory, the object response includes a modification to at least one of a velocity or direction associated with the object trajectory, and wherein issuing the second signal is based at least in part on determining that the modification to at least one of the velocity or direction is less than a threshold modification associated with the expected response.

[0215] V. The one or more non-transitory computer-readable media of paragraph U, further including the operation of repeatedly emitting additional signals until at least one of determining that the object is not associated with the vehicle's progress, determining that a timer associated with the warning signal has expired, or determining that the number of emitted warning signals meets or exceeds a threshold.

[0216] Although the above example section AV is described with respect to a particular implementation, it should be understood that in the context of this document, the contents of the example section AV may also be implemented via a method, apparatus, system, computer-readable medium, and / or another implementation.

[0217] (Conclusion) Having described one or more examples of the technology described herein, various modifications, additions, permutations, and equivalents thereof fall within the scope of the technology described herein.

[0218] In describing the examples, reference is made to the accompanying drawings that form a part of this specification, which show, by way of example, specific examples of the claimed subject matter. It is to be understood that other examples may be used and that changes or modifications, such as structural changes, may be made. Such examples, modifications, or modifications do not necessarily constitute a departure from the intended scope of the claimed subject matter. While steps herein may be presented in a certain order, in some cases the order may be changed, such that some inputs are provided at different times or in a different order without changing the functionality of the systems and methods described. The procedures disclosed may also be performed in a different order. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to being reordered, calculations may also be decomposed into sub-calculations that have the same result.

Claims

1. A vehicle, The sensor and The emitter and one or more processors; When executed, the vehicle: determining objects in an environment associated with the vehicle based at least in part on sensor data from the sensors; determining an object trajectory associated with the object based at least in part on the sensor data and a heat map associated with the object; determining that the object is associated with the vehicle's progress based at least in part on the heat map and an overlap between the object trajectory and a future position of the vehicle; emitting, via the emitter, a first signal based at least in part on determining that the object is associated with the travel of the vehicle, the first signal including a first characteristic; determining an object response to the first signal based at least in part on the sensor data; and generating a second signal based at least in part on the match value not meeting or exceeding a threshold, wherein generating the second signal includes: determining a match value associated with the object response by comparing the object response to an expected response; determining that the object response differs from the expected response based on the match value not meeting or exceeding a threshold; and the second signal includes a second characteristic different from the first characteristic. one or more computer-readable media having stored thereon instructions configured to: A vehicle equipped with:

2. The object reaction is a first object reaction, and the predicted reaction is a first predicted reaction; The instructions may include: determining a second object response to the second signal; storing data associated with the second object reaction in a database based at least in part on the second object reaction; The vehicle of claim 1 .

3. The first characteristic is one or more first frequencies; one or more first volume levels; one or more first intensities; one or more first colors; one or more first shapes; or one or more first movements; At least one of The second characteristic is one or more second frequencies; one or more second volumes; one or more second intensities; one or more second colors; one or more second shapes; or one or more second movements; At least one of 3. A vehicle according to claim 1 or 2.

4. The first characteristic or the second characteristic is based at least in part on a movement associated with the object, the movement comprising: Listening to headphones, Viewing data on mobile devices Reading books, Talking on a cell phone, Eating, Drinking, the specific motion implied by the predicted trajectory, operating devices for sensory impaired persons; a head of the object facing away from a location associated with the vehicle; interacting with other vehicles in the environment; or interacting with other objects in the vicinity of said object; 4. A vehicle according to claim 1, comprising at least one of:

5. The instructions further include: Machine learning techniques, or The predicted reaction is the first characteristic, a classification of said object; the position of the object, or the movement of the object, predicted reaction data stored in a database associated with at least one of The vehicle of claim 2 , wherein the anticipated response is determined based, at least in part, on at least one of:

6. 1. A method comprising: Detecting objects in an environment based on sensor data from sensors on a vehicle, the objects including object attributes; determining an object trajectory associated with the object based at least in part on a heat map associated with the object; causing a first signal to be emitted via an emitter of a vehicle at a first time based at least in part on the object attribute and the object trajectory, the first signal including a first characteristic; determining an object response of the object at a second time after the first time based at least in part on additional sensor data from the sensor; and and causing a second signal to be emitted via the emitter of the vehicle based on the match value not meeting or exceeding a threshold value, wherein causing the second signal to be emitted includes: determining a match value associated with the object response by comparing the object response to an expected response; determining that the object response differs from the expected response based on the match value not meeting or exceeding a threshold; and the second signal includes a second characteristic; and A method for providing the above.

7. the sensor data is first sensor data, and the method further comprises: determining that the object is associated with the vehicle's progress; causing at least one of the first signal or the second signal to be emitted in a direction associated with the object based at least in part on determining that the object is associated with the travel of the vehicle; and The method of claim 6 further comprising:

8. The method of claim 6 or 7, wherein causing the second signal to be emitted is further based on determining that the object continues to impede the vehicle's progress after the second time period.

9. At least one of the first characteristic or the second characteristic is environmental factors in said environment; weather conditions in said environment; the location of the vehicle in the environment; the speed of the vehicle in the environment; a movement associated with the object; the relative position of the object with respect to the vehicle; the day the vehicle is in operation, the year the vehicle has been in operation; or the day of the week that the vehicle is in operation; 9. The method of claim 6, wherein the method is based, at least in part, on at least one of:

10. The movement is Listening to headphones, Viewing data on mobile devices Reading books, Talking on a cell phone, Eating, Drinking, the specific motion implied by the predicted trajectory, operating devices for sensory impaired persons; a head of the object facing away from a location associated with the vehicle; interacting with other vehicles in the environment; or interacting with other objects in the vicinity of said object; The method of claim 9 , comprising one or more of:

11. The object reaction is a first object reaction, the predicted reaction is a first predicted reaction, and the method comprises: determining a second object response of the object to the second signal; storing data associated with at least one of the second signal or the second object response in a database based at least in part on the second object response; The method of any one of claims 6 to 10, further comprising:

12. The emitter is speaker, Light, or Projector, The method according to any one of claims 6 to 11, comprising at least one of:

13. The method comprises: determining that the object is not associated with the vehicle; determining that a timer associated with the warning signal has expired; or determining whether the number of emitted warning signals meets or exceeds a threshold; 13. The method of claim 6, further comprising repeatedly emitting additional signals up to at least one of:

14. The method comprises: determining that the object at least partially blocks a vehicle path associated with the vehicle; identifying a location for the object to move to, the location being outside the vehicle path and away from other objects; causing a third signal to be emitted via a second emitter based at least in part on determining the location, the third signal providing instructions to the object at the location to cause the object to move; 14. The method of claim 6, further comprising:

15. 15. A non-transitory computer readable medium storing instructions that, when executed, cause one or more processors to perform the method of any one of claims 6 to 14.

Images