Display system and method

Abstract

Methods, computer program products, and computing systems are provided for: monitoring one or more machine vision sensors to obtain perception information regarding one or more third-party vehicles in the vicinity of an autonomous vehicle; identifying one or more vehicle shapes within the perception information to define one or more detected vehicle shapes; generating nearby object display information that positions the one or more detected vehicle shapes relative to the autonomous vehicle; and presenting the nearby object display information on a visual display system to confirm the autonomous vehicle's perception of the one or more third-party vehicles.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 028,953, filed May 22, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to display systems, and more particularly to display systems for use in autonomous vehicles. Background Art

[0004] As the transportation industry moves toward autonomous (i.e., driverless) vehicles, manufacturers and designers of these vehicles must address issues that traditional vehicles don't. Unfortunately, autonomous vehicles are still a rarity on the streets, and people are still unsure how they operate.

[0005] For example, when a person is waiting to cross the street at a crosswalk and a vehicle is approaching the same crosswalk, the person will typically wait until there is a signal from the driver of the vehicle that they see them. For example, the person might wait until the driver of the other vehicle looks at them... or waves at them... or flashes their headlights at them.

[0006] However, autonomous vehicles do not have drivers. Therefore, there is no one in the vehicle who can provide a signal to people waiting to cross the street at the crosswalk. Summary of the Invention

[0007] Concept 3

[0008] In one embodiment, a computer-implemented method is executed on a computing device and includes: monitoring one or more machine vision sensors to obtain perception information regarding one or more third-party vehicles in a vicinity of an autonomous vehicle; identifying one or more vehicle shapes within the perception information to define one or more detected vehicle shapes; generating nearby object display information that positions the one or more detected vehicle shapes relative to the autonomous vehicle; and presenting the nearby object display information on a visual display system to confirm the autonomous vehicle's perception of the one or more third-party vehicles.

[0009] One or more of the following features may be included. Identifying one or more vehicle shapes within the perception information may include comparing one or more defined vehicle shapes to one or more unidentified objects within the perception information to identify the one or more vehicle shapes within the perception information. The nearby object display information may include dynamic nearby object display information that changes as the position of the one or more detected vehicle shapes relative to the autonomous vehicle changes. Presenting the nearby object display information on the visual display system may include presenting the dynamic nearby object display information on the visual display system to dynamically confirm the autonomous vehicle's perception of the one or more third-party vehicles. The one or more machine vision sensors may include a lidar system. The visual display system may be configured to be mounted on the roof of the autonomous vehicle. The visual display system may be a cylindrical visual display system. The cylindrical visual display system may include an illuminated portion and an unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle. The visual display system may be a 360-degree visual display system. The visual display system may be integrated into the autonomous vehicle.

[0010] In another embodiment, a computer program product resides on a computer-readable medium and has a plurality of instructions stored on the computer-readable medium. The instructions, when executed by a processor, cause the processor to perform operations comprising: monitoring one or more machine vision sensors to obtain perception information regarding one or more third-party vehicles in a vicinity of an autonomous vehicle; identifying one or more vehicle shapes within the perception information to define one or more detected vehicle shapes; generating nearby object display information that positions the one or more detected vehicle shapes relative to the autonomous vehicle; and presenting the nearby object display information on a visual display system to confirm the autonomous vehicle's perception of the one or more third-party vehicles.

[0011] One or more of the following features may be included. Identifying one or more vehicle shapes within the perception information may include comparing one or more defined vehicle shapes to one or more unidentified objects within the perception information to identify the one or more vehicle shapes within the perception information. The nearby object display information may include dynamic nearby object display information that changes as the position of the one or more detected vehicle shapes relative to the autonomous vehicle changes. Presenting the nearby object display information on the visual display system may include presenting the dynamic nearby object display information on the visual display system to dynamically confirm the autonomous vehicle's perception of the one or more third-party vehicles. The one or more machine vision sensors may include a lidar system. The visual display system may be configured to be mounted on the roof of the autonomous vehicle. The visual display system may be a cylindrical visual display system. The cylindrical visual display system may include an illuminated portion and an unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle. The visual display system may be a 360-degree visual display system. The visual display system may be integrated into the autonomous vehicle.

[0012] In another embodiment, a computing system includes a processor and a memory configured to perform operations comprising: monitoring one or more machine vision sensors to obtain perception information regarding one or more third-party vehicles in a vicinity of an autonomous vehicle; identifying one or more vehicle shapes within the perception information to define one or more detected vehicle shapes; generating nearby object display information that positions the one or more detected vehicle shapes relative to the autonomous vehicle; and presenting the nearby object display information on a visual display system to confirm the autonomous vehicle's perception of the one or more third-party vehicles.

[0013] One or more of the following features may be included. Identifying one or more vehicle shapes within the perception information may include comparing one or more defined vehicle shapes to one or more unidentified objects within the perception information to identify the one or more vehicle shapes within the perception information. The nearby object display information may include dynamic nearby object display information that changes as the position of the one or more detected vehicle shapes relative to the autonomous vehicle changes. Presenting the nearby object display information on the visual display system may include presenting the dynamic nearby object display information on the visual display system to dynamically confirm the autonomous vehicle's perception of the one or more third-party vehicles. The one or more machine vision sensors may include a lidar system. The visual display system may be configured to be mounted on the roof of the autonomous vehicle. The visual display system may be a cylindrical visual display system. The cylindrical visual display system may include an illuminated portion and an unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle. The visual display system may be a 360-degree visual display system. The visual display system may be integrated into the autonomous vehicle.

[0014] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of an autonomous vehicle according to one embodiment of the present disclosure.

[0016] Figure 2A According to one embodiment of the present disclosure, Figure 1 A schematic diagram of one embodiment of various systems within an autonomous vehicle.

[0017] Figure 2B According to one embodiment of the present disclosure, Figure 1 A schematic diagram of another embodiment of various systems within an autonomous vehicle.

[0018] Figure 3 According to one embodiment of the present disclosure, Figure 1 A schematic diagram of another embodiment of various systems within an autonomous vehicle.

[0019] Figures 4A-4C According to one embodiment of the present disclosure Figure 1 Schematic diagram of the status indication system of an autonomous vehicle.

[0020] Figure 5 According to one embodiment of the present disclosure Figure 1 A flow chart of one embodiment of a status indication process performed on one or more systems of an autonomous vehicle.

[0021] Figures 6A-6C According to one embodiment of the present disclosure Figure 1 Schematic diagram of the status indication system of an autonomous vehicle.

[0022] Figure 7 According to one embodiment of the present disclosure Figure 1 A flow chart of another embodiment of a status indication process performed on one or more systems of an autonomous vehicle.

[0023] Figure 8 According to one embodiment of the present disclosure Figure 1 A schematic diagram of a status indication system for an autonomous vehicle; and

[0024] Figure 9A FIG. 9B is a diagram according to one embodiment of the present disclosure. Figure 1 Schematic diagram of the status indication system of an autonomous vehicle.

[0025] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0026] Autonomous Vehicle Overview

[0027] Reference Figure 1 , showing an autonomous vehicle 10. As is known in the art, an autonomous vehicle (e.g., autonomous vehicle 10) is a vehicle that is capable of sensing its environment and moving with little or no human input. An autonomous vehicle (e.g., autonomous vehicle 10) may incorporate various sensor systems to sense its surroundings, examples of which may include, but are not limited to, radar, computer vision, laser radar (LIDAR), global positioning system (GPS), odometer, temperature, and inertia, wherein these sensor systems may be configured to interpret lanes and markings on the road, road signs, traffic lights, pedestrians, other vehicles, roadside objects, hazards, etc.

[0028] Autonomous vehicle 10 may include multiple sensors (e.g., sensors 12), multiple electronic control units (e.g., ECU 14), and multiple actuators (e.g., actuator 16). Thus, sensors 12 within autonomous vehicle 10 may monitor the environment in which autonomous vehicle 10 is located, wherein sensors 12 may provide sensor data 18 to ECU 14. ECU 14 may process sensor data 18 to determine how autonomous vehicle 10 should move. ECU 14 may then provide control data 20 to actuators 16 so that autonomous vehicle 10 moves in the manner determined by ECU 14. For example, a machine vision sensor included within sensor 12 may "read" a speed limit sign indicating that the speed limit on the road autonomous vehicle 10 is currently 35 miles per hour. This machine vision sensor included within sensor 12 may provide sensor data 18 to ECU 14 indicating that the speed limit on the road autonomous vehicle 10 is currently 35 miles per hour. Upon receiving sensor data 18, ECU 14 may process sensor data 18 and may determine that ego vehicle 10 (currently traveling at 45 mph) is traveling too fast and needs to slow down. Accordingly, ECU 14 may provide control data 20 to actuators 16, where control data 20 may, for example, apply the brakes of ego vehicle 10 or remove any actuator signal currently applied to the accelerator (thereby allowing ego vehicle 10 to coast until the speed of ego vehicle 10 decreases to 35 mph).

[0029] System redundancy

[0030] It is conceivable that since autonomous vehicle 10 is controlled by various electronic systems included therein (e.g., sensors 12, ECU 14, and actuators 16), possible failures of one or more of these systems should be considered when designing autonomous vehicle 10, and appropriate contingency plans may be employed.

[0031] For example, also refer to Figure 2A The various ECUs (e.g., ECU 14) included in autonomous vehicle 10 may be partitioned so that the responsibilities of the various ECUs (e.g., ECU 14) may be logically grouped. For example, ECU 14 may include autonomous control unit 50 that receives sensor data 18 from sensor 12.

[0032] Autonomous control unit 50 may be configured to perform various functions. For example, autonomous control unit 50 may receive and process exteroceptive sensor data (e.g., sensor data 18), may estimate the position of autonomous vehicle 10 within its operating environment, may calculate a representation of the surroundings of autonomous vehicle 10, may calculate a safe trajectory for autonomous vehicle 10, and may command other ECUs (particularly vehicle control units) to cause autonomous vehicle 10 to perform desired maneuvers.

[0033] The autonomous control unit 50 may include powerful computing capabilities, persistent storage, and memory.

[0034] Thus, autonomous control unit 50 can process sensor data 18 to determine how autonomous vehicle 10 should operate. Autonomous control unit 50 can then provide vehicle control data 52 to vehicle control unit 54, which can then process vehicle control data 52 to determine how various control systems (e.g., powertrain system 56, braking system 58, and steering system 60) should respond to achieve the trajectory defined by autonomous control unit 50 within vehicle control data 52.

[0035] The vehicle control unit 54 can be configured to control other ECUs included in the autonomous vehicle 10. For example, the vehicle control unit 54 can control steering, powertrain, and brake controller units. For example, the vehicle control unit 54 can provide a powertrain control signal 62 to a powertrain control unit 64, a brake control signal 66 to a brake control unit 68, and a steering control signal 70 to a steering control unit 72.

[0036] The powertrain control unit 64 may process the powertrain control signal 62 so that appropriate control data (generally represented by the control data 20) may be provided to the powertrain system 56. Furthermore, the brake control unit 68 may process the brake control signal 66 so that appropriate control data (generally represented by the control data 20) may be provided to the brake system 58. Furthermore, the steering control unit 72 may process the steering control signal 70 so that appropriate control data (generally represented by the control data 20) may be provided to the steering system 60.

[0037] Powertrain control unit 64 may be configured to control a transmission (not shown) and an engine / traction motor (not shown) within autonomous vehicle 10 ; brake control unit 68 may be configured to control a mechanical / regenerative braking system (not shown) within autonomous vehicle 10 ; and steering control unit 72 may be configured to control a steering column / steering rack (not shown) within autonomous vehicle 10 .

[0038] The autonomous control unit 50 can be a highly complex computing system that provides extensive processing capabilities (e.g., a workstation-class computing system with a multi-core processor, discrete co-processing units, several gigabytes of memory, and persistent storage). In contrast, the vehicle control unit 54 can be a much simpler device that provides processing capabilities comparable to other ECUs included in the autonomous vehicle 10 (e.g., a computing system with a modest microprocessor (CPU frequency less than 200 MHz), less than 1 megabyte of system memory, and no persistent storage). Due to this simpler design, the vehicle control unit 54 can potentially have greater reliability and durability than the autonomous control unit 50.

[0039] To further improve redundancy and reliability, one or more ECUs (ECU 14) included in the autonomous vehicle 10 may be configured in a redundant manner. Figure 2B , shows one embodiment of the ECU 14 in which multiple vehicle control units are utilized. For example, this particular embodiment is shown as including two vehicle control units, a first vehicle control unit (e.g., vehicle control unit 54) and a second vehicle control unit (e.g., vehicle control unit 74).

[0040] In this particular configuration, the two vehicle control units (e.g., vehicle control units 54, 74) can be configured in various ways. For example, the two vehicle control units (e.g., vehicle control units 54, 74) can be configured in an active-passive configuration, wherein, for example, the vehicle control unit 54 performs the active role of processing the vehicle control data 52, while the vehicle control unit 74 assumes the passive role and is essentially in standby mode. In the event of a failure of the vehicle control unit 54, the vehicle control unit 74 can transition from the passive role to the active role and assume the role of processing the vehicle control data 52. Alternatively, the two vehicle control units (e.g., vehicle control units 54, 74) can be configured in an active-active configuration, wherein, for example, both the vehicle control unit 52 and the vehicle control unit 74 perform the active role of processing the vehicle control data 54 (e.g., sharing the workload), wherein, in the event of a failure of the vehicle control unit 54 or the vehicle control unit 74, the surviving vehicle control unit can process all of the vehicle control data 52.

[0041] Although Figure 2BWhile one example of how various ECUs (e.g., ECU 14) included within autonomous vehicle 10 may be configured in a redundant manner is described, this is for illustrative purposes only and is not intended to be limiting of the present disclosure, as other configurations are possible and are considered within the scope of the present disclosure. For example, autonomous control unit 50 may be configured in a redundant manner, wherein a second autonomous control unit (not shown) is included within autonomous vehicle 10 and configured in an active-passive or active-active manner. Furthermore, it is contemplated that one or more sensors (e.g., sensor 12) and / or one or more actuators (e.g., actuator 16) may be configured in a redundant manner. It will be appreciated, therefore, that the degree of redundancy achievable with respect to autonomous vehicle 10 may be limited only by the design criteria and budget constraints of autonomous vehicle 10.

[0042] Autonomous computing subsystem

[0043] Also refer to Figure 3 , the various ECUs of the autonomous vehicle 10 may be grouped / arranged / configured to implement various functions.

[0044] For example, one or more of the ECUs 14 may be configured to implement / form a perception subsystem 100, wherein the perception subsystem 100 may be configured to process data from onboard sensors (e.g., sensor data 18) to compute a concise representation of objects of interest near the autonomous vehicle 10 (examples of which may include, but are not limited to, other vehicles, pedestrians, traffic signals, traffic signs, road markings, hazards, etc.), and identify environmental features that may be helpful in determining the position of the autonomous vehicle 10. Furthermore, one or more of the ECUs 14 may be configured to implement / form a state estimation subsystem 102, wherein the state estimation subsystem 102 may be configured to process data from onboard sensors (e.g., sensor data 18) to estimate the position, orientation, and velocity of the autonomous vehicle 10 in its operating environment. Furthermore, one or more of the ECUs 14 may be configured to implement / form a planning subsystem 104, wherein the planning subsystem 104 may be configured to compute a desired vehicle trajectory (using perception output 106 and state estimation output 108). Additionally, one or more of ECUs 14 may be configured to implement / form a trajectory control subsystem 110 , where trajectory control subsystem 110 uses planning output 112 and state estimation output 108 (in conjunction with feedback and / or feedforward control techniques) to compute actuator commands (e.g., control data 20 ) that may cause autonomous vehicle 10 to execute its desired trajectory within its operating environment.

[0045] For redundancy purposes, the above-described subsystems can be distributed across various devices (e.g., the autonomous control unit 50 and the vehicle control units 54, 74). Additionally or alternatively, due to increased computational requirements, the perception subsystem 100 and the planning subsystem 104 can be located almost entirely within the autonomous control unit 50, which (as described above) has more computational horsepower than the vehicle control unit 54, 74. Conversely, due to their lower computational requirements, the state estimation subsystem 102 and the trajectory control subsystem 110 can be located entirely on the vehicle control unit 54, 74, if the vehicle control unit 54, 74 has the necessary computational power; and / or can be located partially on the vehicle control unit 54, 74 and partially on the autonomous control unit 50. However, the location of the state estimation subsystem 102 and the trajectory control subsystem 110 can be critical in the design of any contingency planning architecture, as the location of these subsystems can determine how the contingency plan is computed, transmitted, and / or executed.

[0046] Status indication system

[0047] Also refer to Figure 4A 、 4B 4C shows an external view of the autonomous vehicle 10, wherein the autonomous vehicle 10 may include a status indication system 200 for transmitting status information about a movable vehicle (such as the autonomous vehicle 10).

[0048] Status indication system 200 may include an interface system (e.g., interface system 202) configured to receive perception information (e.g., perception information 204) about one or more objects (e.g., objects 206, 208, 210, 212) near a movable vehicle (e.g., autonomous vehicle 10). The one or more objects (e.g., objects 206, 208, 210, 212) near a movable vehicle (e.g., autonomous vehicle 10) may include one or more of the following:

[0049] Pedestrians (e.g., objects 208, 210, 212) in the vicinity of a movable vehicle (e.g., autonomous vehicle 10), e.g., a person walking / passing by autonomous vehicle 10;

[0050] Third-party vehicles (e.g., object 206 ) near a movable vehicle (e.g., autonomous vehicle 10 ), such as other vehicles traveling / stopped near autonomous vehicle 10 .

[0051] As described above, sensors 12 within autonomous vehicle 10 can monitor the environment in which autonomous vehicle 10 is located, wherein sensors 12 can provide sensor data 18 to ECU 14, which can be processed to determine how autonomous vehicle 10 should operate. Interface system 202 can be configured to interface with sensors 12 (in general) and one or more machine vision sensors (in particular) included within a mobile vehicle (e.g., autonomous vehicle 10). Examples of such machine vision sensors can include, but are not limited to, lidar systems.

[0052] As known in the art, lidar is a method of determining range (a variable distance) by pointing a laser at an object and measuring the time it takes for the reflected light to return to a receiver. Due to the differences in laser return times and by varying the laser's wavelength, lidar can also be used to create digital 3-D representations of areas on the Earth's surface and ocean floor. It has terrestrial, aerial, and mobile applications. Lidar is an acronym for "light detection and ranging" or "laser imaging, detection, and ranging." Lidar is sometimes referred to as 3-D laser scanning, a special combination of 3-D scanning and laser scanning. Lidar is commonly used to create high-resolution maps and has applications in surveying, geodesy, geography, geology, geomorphology, seismology, forestry, atmospheric physics, laser guidance, airborne laser scanning mapping (ALSM), and laser altimetry. The technology is also used for the control and navigation of some autonomous vehicles.

[0053] The status indication system 200 may include a processing system (e.g., processing system 214) configured to process sensory information (e.g., sensory information 204) to generate nearby object display information (e.g., nearby object display information 216). The processing system 214 may be configured in various ways. One example of the processing system 214 may include a standalone processing system that includes one or more processors (not shown) and one or more memory architectures (not shown). Another example of the processing system 214 may include a portion of the ECU 14. The processing system 214 may be coupled to a storage device (e.g., storage device 114). Examples of the storage device 114 may include, but are not limited to: a hard drive; a RAID device; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices.

[0054] Status indication system 200 may include a visual display system (e.g., visual display system 218) configured to present nearby object display information (e.g., nearby object display information 216). As will be discussed in greater detail below, visual display system (e.g., visual display system 218) may be configured to convey status information about autonomous vehicle 10 in a visual manner that is easily understood by a person near autonomous vehicle 10.

[0055] In addition, the interface system (e.g., interface system 202) can be configured to receive movable vehicle state information (e.g., movable vehicle state information 220). As described above, sensors 12 within autonomous vehicle 10 can monitor the environment in which autonomous vehicle 10 is located, wherein sensors 12 can provide sensor data 18 to ECU 14. Therefore, the interface system (e.g., interface system 202) can obtain such movable vehicle state information (e.g., movable vehicle state information 220) from ECU 14.

[0056] Thus, the movable vehicle status information (e.g., movable vehicle status information 220) may identify one or more of the following:

[0057] Whether the movable vehicle is decelerating;

[0058] Whether the movable vehicle is accelerating;

[0059] · whether the movable vehicle is stationary; and

[0060] Whether the movable vehicle is turning.

[0061] A processing system (e.g., processing system 214) can be configured to process movable vehicle status information (e.g., movable vehicle status information 220) to generate vehicle status display information (e.g., vehicle status display information 222), wherein a visual display system (e.g., visual display system 218) can be configured to present the vehicle status display information (e.g., vehicle status display information 222). As will be discussed in more detail below, a visual display system (e.g., visual display system 218) can be configured to convey status information about autonomous vehicle 10 in a visual manner that is easily understood by a person near autonomous vehicle 10.

[0062] A visual display system (e.g., visual display system 218) can be configured to be mounted on a roof (e.g., roof 224) of a movable vehicle (e.g., autonomous vehicle 10). A visual display system (e.g., visual display system 218) can be configured in a variety of ways, all of which are considered within the scope of the present disclosure.

[0063] Column: The visual display system (e.g., visual display system 218) can be a columnar visual display system (e.g., Figure 4A-4BWhen visual display system 218 is configured in a cylindrical manner, visual display system 218 may include an illuminated portion (e.g., illuminated portion 226) and an unilluminated portion (e.g., unilluminated portion 228) located between the illuminated portion (e.g., illuminated portion 226) and the roof (e.g., roof 224) of the movable vehicle (e.g., autonomous vehicle 10), thereby providing a "floating" appearance relative to the roof (e.g., roof 224) of the movable vehicle (e.g., autonomous vehicle 10).

[0064] Disk-shaped: The visual display system (e.g., visual display system 218) can be a disk-shaped visual display system (e.g., Figure 4C When visual display system 218 is configured in a disk-like manner, visual display system 218 can be generally illuminated near the roof (e.g., roof 224) of a movable vehicle (e.g., autonomous vehicle 10), thereby providing a lower profile, which may be desirable when used with taller-profile vehicles (e.g., SUVs and vans).

[0065] Regardless of the configuration, the visual display system (e.g., visual display system 218 ) may be a 360-degree visual display system, thereby visually conveying status information about autonomous vehicle 10 regardless of whether humans are located near autonomous vehicle 10 .

[0066] While the visual display system (e.g., visual display system 218) is described above as being mounted on the roof (e.g., roof 224) of a movable vehicle (e.g., autonomous vehicle 10), this is for illustrative purposes only and is not intended to be a limitation of the present disclosure, as other configurations are possible and are considered to be within the scope of the present disclosure. For example, the visual display system (e.g., visual display system 218) can be integrated into the movable vehicle (e.g., autonomous vehicle 10) in various ways. Thus, the visual display system (e.g., visual display system 218) can be part of (or incorporated into) a window (e.g., window 230) of the movable vehicle (e.g., autonomous vehicle 10).

[0067] Status indication process (for pedestrians)

[0068] As described above, a visual display system (e.g., visual display system 218 ) may be configured to convey information about the status of autonomous vehicle 10 in a visual manner that is easily understood by a person in the vicinity of autonomous vehicle 10 .

[0069] Please also refer to Figure 5 To convey this status information in such a visual manner, the processing system 214 may execute a status indication process 250. As described above, the processing system 214 may be configured in various ways, examples of which may include, but are not limited to:

[0070] A separate processing system comprising one or more processors (not shown) and one or more memory architectures (not shown); and / or

[0071] A part of the ECU 14.

[0072] The instruction set and subroutines of the status indication process 250 may be stored on a storage device 114 coupled to the ECU 14 and may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within the ECU 14. Examples of the storage device 114 may include, but are not limited to: a hard drive; a RAID device; random access memory (RAM); read-only memory (ROM); and all forms of flash memory storage devices.

[0073] The status indication process 250 can be executed on a single ECU, or can be executed collaboratively across multiple ECUs. For example, the status indication process 250 can be executed solely by the autonomous control unit 50, the vehicle control unit 54, or the vehicle control unit 74. Alternatively, the status indication process 250 can be executed collaboratively across a combination of the autonomous control unit 50, the vehicle control unit 54, and the vehicle control unit 74. Therefore, in the latter configuration, if one of the autonomous control unit 50, the vehicle control unit 54, or the vehicle control unit 74 fails, the surviving control unit can continue to execute the status indication process 250.

[0074] State indication process 250 can monitor 252 one or more machine vision sensors (e.g., sensor 12 in general and lidar sensors in particular) to obtain perception information (e.g., perception information 204) about one or more pedestrians (e.g., objects 208, 210, 212) in the vicinity of an autonomous vehicle (e.g., autonomous vehicle 10). Perception information 204 can be a three-dimensional image that generally identifies all objects in the geographic vicinity of autonomous vehicle 10.

[0075] State indication process 250 may identify 254 one or more humanoid shapes within the sensory information (e.g., sensory information 204), thereby defining one or more detected humanoid shapes. Upon identifying 254 one or more humanoid shapes within the sensory information (e.g., sensory information 204), state indication process 250 may compare 256 the one or more defined humanoid shapes (e.g., defined humanoid shape 232) with one or more unidentified objects within the sensory information (e.g., sensory information 204) to identify the one or more humanoid shapes within the sensory information (e.g., sensory information 204).

[0076] Defined humanoid shape 232 can be manually defined (e.g., by a designer / programmer of state indication process 250) and / or automatically defined (e.g., by artificial intelligence / machine learning (AI / ML), similar to how AI / ML can identify faces within a photograph). Generally speaking, perception information 204 can be a three-dimensional image that collectively identifies a plurality of unidentified objects in the geographic vicinity of autonomous vehicle 10. Thus, state indication process 250 can compare 256 defined humanoid shape 232 to one or more unidentified objects within perception information (e.g., perception information 204) to identify one or more humanoid shapes within the perception information (e.g., perception information 204).

[0077] Upon identifying humanoid shapes 254 within perception information 204, status indication process 250 can generate 258 nearby object display information (e.g., nearby object display information 216) that locates the one or more detected humanoid shapes relative to an autonomous vehicle (e.g., autonomous vehicle 10). Status indication process 250 can then present 260 the nearby object display information (e.g., nearby object display information 216) on a visual display system (e.g., visual display system 218) to confirm the autonomous vehicle's (e.g., autonomous vehicle 10) perception of one or more pedestrians (e.g., objects 208, 210, 212).

[0078] For example, Figure 6A The following illustrates a situation where two pedestrians (e.g., pedestrians 300 and 302) are standing stationary near the sides of autonomous vehicle 10. Therefore, nearby object display information 216 can position one or more detected humanoid shapes (e.g., pedestrians 300 and 302) relative to autonomous vehicle 10. Furthermore, visual display system 218 of status indication system 200 can convey status information about autonomous vehicle 10 in a visual manner that is easily understood by persons (e.g., pedestrians 300 and 302) near autonomous vehicle 10. Therefore, visual display system 218 can present a first sign (e.g., sign 304) pointing toward (or corresponding to) pedestrian 300. Since pedestrian 300 is stationary, sign 304 can be stationary within visual display system 302. Furthermore, visual display system 218 can present a second sign (e.g., sign 306) pointing toward (or corresponding to) pedestrian 302. Since pedestrian 302 is stationary, sign 306 can be stationary within visual display system 302.

[0079] also, Figure 6BA situation is illustrated in which a pedestrian (e.g., pedestrians 300, 308) is standing stationary near the front of autonomous vehicle 10. Therefore, nearby object display information 216 may position one or more detected humanoid shapes (e.g., pedestrian 308) relative to autonomous vehicle 10. Furthermore, visual display system 218 of status indication system 200 may convey status information about autonomous vehicle 10 in a visual manner that is easily understood by a person (e.g., pedestrian 308) near autonomous vehicle 10. Therefore, visual display system 218 may present a sign (e.g., sign 310) pointing toward (or corresponding to) pedestrian 308. Because pedestrian 308 is stationary, sign 310 may be stationary within visual display system 302. Furthermore, because pedestrian 310 is in the path of autonomous vehicle 10 (i.e., obstructing it), sign 310 may provide instructions to pedestrian 310, such as a left-pointing arrow, to move left and out of the path of autonomous vehicle 10.

[0080] The nearby object display information (e.g., nearby object display information 216) can include dynamic nearby object display information (e.g., nearby object display information 216) that changes as the position of the one or more detected humanoid shapes changes relative to the autonomous vehicle (e.g., autonomous vehicle 10). Therefore, when presenting 260 the nearby object display information (e.g., nearby object display information 216) on the visual display system (e.g., visual display system 218), the status indication process 250 can present 262 the dynamic nearby object display information (e.g., nearby object display information 216) on the visual display system (e.g., visual display system 218), thereby dynamically confirming the autonomous vehicle (e.g., autonomous vehicle 10)'s perception of the one or more pedestrians (e.g., objects 208, 210, 212).

[0081] For example, Figure 6CThe following illustrates a situation in which two pedestrians (e.g., pedestrians 312 and 314) are walking in front of autonomous vehicle 10. Therefore, nearby object display information 216 can position one or more detected humanoid shapes (e.g., pedestrians 312 and 314) relative to autonomous vehicle 10. Furthermore, visual display system 218 of status indication system 200 can convey status information about autonomous vehicle 10 in a visual manner that is easily understood by people (e.g., pedestrians 312 and 314) approaching autonomous vehicle 10. Accordingly, visual display system 218 can present a first indicator (e.g., indicator 316) pointing toward (or corresponding to) pedestrian 312. As pedestrian 312 moves from left to right, indicator 316 can also move from left to right within visual display system 302. Furthermore, visual display system 218 can present a second indicator (e.g., indicator 318) pointing toward (or corresponding to) pedestrian 314. As pedestrian 314 moves from left to right, indicator 318 can also move from left to right within visual display system 302.

[0082] Status indication process (for vehicles)

[0083] In a manner similar to how state indication process 10 tracks pedestrians near an autonomous vehicle (eg, autonomous vehicle 10 ), state indication process 10 may also track vehicles near an autonomous vehicle (eg, autonomous vehicle 10 ).

[0084] For example, also refer to Figure 7 , the status indication process 250 can monitor 350 one or more machine vision sensors (e.g., sensor 12 in general and lidar sensor in particular) to obtain perception information (e.g., perception information 204) about one or more third-party vehicles (e.g., object 206) in proximity to the autonomous vehicle (e.g., autonomous vehicle 10).

[0085] State indication process 250 may identify 352 one or more vehicle shapes within the perception information (e.g., perception information 204) to define one or more detected vehicle shapes. When identifying 352 one or more vehicle shapes within the perception information (e.g., perception information 204), state indication process 250 may compare 354 the one or more defined vehicle shapes (e.g., defined vehicle shape 234) with one or more unidentified objects within the perception information (e.g., perception information 204) to identify the one or more vehicle shapes within the perception information (e.g., perception information 204).

[0086] Defined vehicle shape 234 can be manually defined (e.g., by a designer / programmer of state indication process 250) and / or automatically defined (e.g., by artificial intelligence / machine learning (AI / ML), similar to how AI / ML can identify vehicles within a photograph). Generally speaking, perception information 204 can be a three-dimensional image to generally identify a plurality of unidentified objects in the geographic vicinity of autonomous vehicle 10. Thus, state indication process 250 can compare 354 defined vehicle shape 234 to one or more unidentified objects within perception information (e.g., perception information 204) to identify one or more vehicle shapes within the perception information (e.g., perception information 204).

[0087] Once the vehicle shapes 352 are identified within perception information 204, status indication process 250 can generate 356 nearby object display information (e.g., nearby object display information 216) that locates the one or more detected vehicle shapes relative to the autonomous vehicle (e.g., autonomous vehicle 10). Status indication process 250 can then present 358 the nearby object display information (e.g., nearby object display information 216) on a visual display system (e.g., visual display system 218) to confirm the autonomous vehicle's (e.g., autonomous vehicle 10) perception of the one or more third-party vehicles (e.g., object 206).

[0088] The nearby object display information (e.g., nearby object display information 216) can include dynamic nearby object display information (e.g., nearby object display information 216) that changes as the position of one or more detected vehicle shapes changes relative to the autonomous vehicle (e.g., autonomous vehicle 10). Accordingly, when presenting 358 the nearby object display information (e.g., nearby object display information 216) on the visual display system (e.g., visual display system 218), the status indication process 250 can present 360 the dynamic nearby object display information (e.g., nearby object display information 216) on the visual display system (e.g., visual display system 218), thereby dynamically confirming the autonomous vehicle (e.g., autonomous vehicle 10)'s perception of the one or more third-party vehicles (e.g., object 206).

[0089] like Figure 6A 、 Figure 6B 、 Figure 6CAs shown, the visual display system 218 can present various signs (e.g., signs 304, 306, 310, 316, 318), wherein each of these signs (e.g., signs 304, 306, 310, 316, 318) is displayed to include, for example, a "stick figure drawing" to indicate that the sign (e.g., signs 304, 306, 310, 316, 318) points to (or corresponds to) a pedestrian (e.g., pedestrians 300, 302, 308, 312, 314). Accordingly, reference is also made to Figure 8 When the status indicates that process 10 is tracking a vehicle near autonomous vehicle 10, visual display system 218 can present various signs (e.g., sign 320), including, for example, a "car image," to indicate that the sign (e.g., sign 320) points to (or corresponds to) a vehicle (e.g., third-party vehicle 322).

[0090] Status indication process (for AV status)

[0091] As described above, an interface system (e.g., interface system 202) can be configured to receive movable vehicle status information (e.g., movable vehicle status information 220), wherein the movable vehicle status information (e.g., movable vehicle status information 220) can identify one or more of the following:

[0092] Whether the movable vehicle is decelerating;

[0093] Whether the movable vehicle is accelerating;

[0094] · whether the movable vehicle is stationary; and

[0095] Whether the movable vehicle is turning.

[0096] A processing system (e.g., processing system 214) can be configured to process movable vehicle status information (e.g., movable vehicle status information 220) to generate vehicle status display information (e.g., vehicle status display information 222), wherein a visual display system (e.g., visual display system 218) can be configured to present the vehicle status display information (e.g., vehicle status display information 222).

[0097] Therefore, if Figure 9A As shown, nearby object portion 350 of visual display system 218 can be configured to track pedestrians near an autonomous vehicle (e.g., autonomous vehicle 10). Additionally, status portion 352 of visual display system 218 can be configured to present vehicle status display information 222. For example, status portion 352 can include:

[0098] "Left Turn" section 354, which may illuminate when autonomous vehicle 10 plans to turn left;

[0099] "Right Turn" section 356, which may illuminate when autonomous vehicle 10 plans to turn right;

[0100] Status section 358, which may:

[0101] i. By displaying a solid white color, for example, to indicate that the autonomous vehicle 10 is cruising in a stable state,

[0102] ii. indicating that the autonomous vehicle is accelerating by, for example, displaying a color change from white to dark green,

[0103] iii. indicating that the autonomous vehicle is slowing down by, for example, displaying a color change from white to rich red, and

[0104] iv. Indicate that the autonomous vehicle is stationary by, for example, displaying a solid red color.

[0105] In addition, if Figure 9B As shown, nearby object portion 350 of visual display system 218 can be configured to track third-party vehicles near an autonomous vehicle (e.g., autonomous vehicle 10). Similarly, status portion 352 of visual display system 218 can be configured to present vehicle status display information 222 (as described above).

[0106] summary

[0107] As will be appreciated by those skilled in the art, the present disclosure may be embodied as a method, system, or computer program product. Thus, the present disclosure may take the form of an entirely hardware implementation, an entirely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, all of which may be collectively referred to herein as a "circuit," "module," or "system." Furthermore, the present disclosure may take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied therein.

[0108] Any suitable computer-usable or computer-readable medium can be utilized. A computer-usable or computer-readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, apparatus, or propagation medium. More specific examples (non-exhaustive list) of computer-readable media can include the following: an electrical connector having one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a transmission medium such as one that supports the Internet or an intranet, or a magnetic storage device. A computer-usable or computer-readable medium can also be paper or another suitable medium on which the program is printed, because the program can be electronically captured, compiled, interpreted, or processed in a suitable manner as necessary, and then stored in a computer memory, for example, by optically scanning the paper or other medium, and then stored in a computer memory. In the context of this document, a computer-usable or computer-readable medium can be any medium that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer usable medium may include a data signal transmitted in baseband or as part of a carrier wave, with the computer usable program code embodied in the transmitted data signal. The computer usable program code may be transmitted using any appropriate medium, including but not limited to the Internet, wired, optical cable, radio frequency, etc.

[0109] The computer program code for performing the operation of the present disclosure can be written in an object-oriented programming language, such as Java, Smalltalk, C++ or similar languages. However, the computer program code for performing the operation of the present disclosure can also be written in a traditional procedural programming language, such as "C" programming language or similar programming languages. This program code can be performed completely on the user's computer as an independent software package, partly on the user's computer, and the independent software package is partly on the user's computer and partly on a remote computer or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via a local area network / wide area network / internet (e.g., network 14).

[0110] The present disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer / special-purpose computer / other programmable data processing device, thereby creating a means for implementing the functions / behaviors specified by one or more blocks in the flowchart and / or block diagram through instructions executed by the processor of the computer or other programmable data processing device.

[0111] These computer program instructions may also be stored in a computer-readable memory, which may instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture, which includes instruction means for implementing the functions / behaviors specified in the flowcharts and / or block diagrams.

[0112] Computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide steps for implementing the functions / behaviors specified in the flowcharts and / or block diagrams.

[0113] The flowcharts and block diagrams in the figures may illustrate the structure, functionality, and operation of possible implementations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may not appear in the order indicated in the figures. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart illustrations, and combinations of blocks in the block diagram and / or flowchart illustrations, may be implemented by a hardware-based special-purpose system that performs the specified function or behavior, or a combination of special-purpose hardware and computer instructions.

[0114] The terms used herein are intended only to describe specific embodiments and are not intended to limit the disclosure. As used herein, the singular forms "a," "an," and "the" also include the plural forms unless the context clearly indicates otherwise. It should be further understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of the recited features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0115] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other specifically claimed elements. The description of the present disclosure is presented for purposes of illustration and description and is not intended to be exhaustive or limited to the forms disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The embodiments are selected and described in order to best explain the principles and practical applications of the disclosure and to enable those of ordinary skill in the art to understand the various embodiments of the disclosure and to make various modifications thereto as appropriate for the particular use contemplated.

[0116] Having described the present disclosure in detail and by reference to embodiments thereof, it will be apparent that modifications and variations are possible without departing from the scope of the disclosure as defined in the appended claims.

Claims

1. A computer-implemented method for an autonomous vehicle, executed on a computing device, comprising: monitoring one or more machine vision sensors of the autonomous vehicle to obtain perception information about one or more third-party vehicles in the vicinity of the autonomous vehicle; identifying one or more vehicle shapes within the perception information, thereby defining one or more detected vehicle shapes; generating nearby object display information that positions the one or more detected vehicle shapes relative to the autonomous vehicle, wherein the nearby object display information includes dynamic nearby object display information that changes as the positions of the one or more detected vehicle shapes relative to the autonomous vehicle change; and presenting the nearby object display information on a visual display system of the autonomous vehicle to thereby confirm the autonomous vehicle's perception of the one or more third-party vehicles to the one or more third-party vehicles in the vicinity of the autonomous vehicle, wherein the dynamic nearby object display information is presented on the visual display system to dynamically confirm the autonomous vehicle's perception of the one or more third-party vehicles; wherein the presented dynamic nearby object display information includes a vehicle image representing the one or more third-party vehicles, and the vehicle image tracks the one or more third-party vehicles in the vicinity of the autonomous vehicle.

2. The computer-implemented method of claim 1 , wherein: Identifying one or more vehicle shapes within the perception information includes comparing the one or more defined vehicle shapes to one or more unidentified objects within the perception information to identify the one or more vehicle shapes within the perception information.

3. The computer-implemented method of claim 1 , wherein: The one or more machine vision sensors include a lidar system.

4. The computer-implemented method of claim 1 , wherein: The visual display system is configured to be mounted on a roof of an autonomous vehicle.

5. The computer-implemented method of claim 1 , wherein: The visual display system is a cylindrical visual display system.

6. The computer-implemented method of claim 5, wherein: The cylindrical visual display system includes an illuminated portion and an unilluminated portion between the illuminated portion and a roof of the autonomous vehicle.

7. The computer-implemented method of claim 1 , wherein: The visual display system is a 360-degree visual display system.

8. The computer-implemented method of claim 1 , wherein: Visual display systems are integrated into autonomous vehicles.

9. A computer program product residing on a computer-readable medium for use in an autonomous vehicle, the computer program product residing on the computer-readable medium having a plurality of instructions stored thereon, the plurality of instructions, when executed by a processor, causing the processor to perform operations comprising: monitoring one or more machine vision sensors of the autonomous vehicle to obtain perception information about one or more third-party vehicles in the vicinity of the autonomous vehicle; identifying one or more vehicle shapes within the perception information, thereby defining one or more detected vehicle shapes; generating nearby object display information that positions the one or more detected vehicle shapes relative to the autonomous vehicle, wherein the nearby object display information includes dynamic nearby object display information that changes as the positions of the one or more detected vehicle shapes relative to the autonomous vehicle change; and presenting the nearby object display information on a visual display system of the autonomous vehicle to thereby confirm the autonomous vehicle's perception of the one or more third-party vehicles to the one or more third-party vehicles in the vicinity of the autonomous vehicle, wherein the dynamic nearby object display information is presented on the visual display system to dynamically confirm the autonomous vehicle's perception of the one or more third-party vehicles; wherein the presented dynamic nearby object display information includes a vehicle image representing the one or more third-party vehicles, and the vehicle image tracks the one or more third-party vehicles in the vicinity of the autonomous vehicle.

10. The computer program product of claim 9, wherein: Identifying one or more vehicle shapes within the perception information includes comparing the one or more defined vehicle shapes to one or more unidentified objects within the perception information to identify the one or more vehicle shapes within the perception information.

11. The computer program product of claim 9, wherein: The one or more machine vision sensors include a lidar system.

12. The computer program product of claim 9, wherein: The visual display system is configured to be mounted on a roof of an autonomous vehicle.

13. The computer program product of claim 9, wherein: The visual display system is a cylindrical visual display system.

14. The computer program product of claim 13, wherein: The cylindrical visual display system includes: an illuminated portion; and an unilluminated portion located between the illuminated portion and a roof of the autonomous vehicle.

15. The computer program product of claim 9, wherein: The visual display system is a 360-degree visual display system.

16. The computer program product of claim 9, wherein: Visual display systems are integrated into autonomous vehicles.

17. A computing system for use in an autonomous vehicle, the computing system comprising a processor and a memory, the processor and the memory being configured to perform operations comprising: monitoring one or more machine vision sensors of the autonomous vehicle to obtain perception information about one or more third-party vehicles in the vicinity of the autonomous vehicle; Identifying one or more vehicle shapes within the perception information, thereby defining one or more detected vehicle shapes, wherein the nearby object display information includes dynamic nearby object display information that changes as the position of the one or more detected vehicle shapes relative to the autonomous vehicle changes; generating nearby object display information that positions the one or more detected vehicle shapes relative to the autonomous vehicle; and presenting the nearby object display information on a visual display system of the autonomous vehicle, thereby confirming the autonomous vehicle's perception of the one or more third-party vehicles to the one or more third-party vehicles in the vicinity of the autonomous vehicle, wherein the dynamic nearby object display information is presented on the visual display system, thereby dynamically confirming the autonomous vehicle's perception of the one or more third-party vehicles; wherein the presented dynamic nearby object display information includes a vehicle image representing the one or more third-party vehicles, and the vehicle image tracks the one or more third-party vehicles in the vicinity of the autonomous vehicle.

18. The computing system of claim 17, wherein: Identifying one or more vehicle shapes within the perception information includes comparing one or more defined vehicle shapes to one or more unidentified objects within the perception information to identify the one or more vehicle shapes within the perception information.

19. The computing system of claim 17, wherein: The one or more machine vision sensors include a lidar system.

20. The computing system of claim 17, wherein: The visual display system is configured to be mounted on a roof of an autonomous vehicle.

21. The computing system of claim 17, wherein: The visual display system is a cylindrical visual display system.

22. The computing system of claim 21, wherein: The cylindrical visual display system includes: an illuminated portion; and an unilluminated portion located between the illuminated portion and a roof of the autonomous vehicle.

23. The computing system of claim 17, wherein: The visual display system is a 360-degree visual display system.

24. The computing system of claim 17, wherein: Visual display systems are integrated into autonomous vehicles.

Images