Display system and method

Abstract

A method, a computer program product, and a computing system for: monitoring one or more machine vision sensors to obtain perception information about one or more pedestrians near an autonomous vehicle; identifying one or more humanoid shapes within the perception information, thereby defining one or more detected humanoid shapes; generating nearby object display information that localizes the one or more detected humanoid shapes relative to the autonomous vehicle; and presenting the nearby object display information on a visual display system to confirm the autonomous vehicle's perception of the one or more pedestrians.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 028,953, filed on May 22, 2020, the entire content of which is incorporated herein by reference. Technical Field

[0003] This application relates to display systems, and more particularly to display systems used in autonomous vehicles. Background Art

[0004] As the transportation industry moves towards autonomous (i.e., driverless) vehicles, the manufacturers and designers of these autonomous vehicles must address issues that traditional vehicles do not concern themselves with. Unfortunately, autonomous vehicles are still rare on the streets, and people are still unsure of how they operate.

[0005] For example, when a person is waiting to cross the road at a crosswalk and a vehicle is approaching the same crosswalk, the person typically waits until there is a signal indicating that the driver of the vehicle sees them. For example, the person might wait until the driver of another vehicle stares at them... or waves at them... or flashes their headlights at them.

[0006] However, autonomous vehicles do not have a driver. Thus, there is no one in the vehicle to provide a signal to the person waiting to cross the road at the crosswalk. Summary of the Invention

[0007] Concept 2

[0008] In one embodiment, a computer - implemented method is executed on a computing device and includes: monitoring one or more machine vision sensors to obtain perception information about one or more pedestrians near an autonomous vehicle; identifying one or more human - like shapes within the perception information, thereby defining one or more detected human - like shapes; generating nearby object display information that positions the one or more detected human - like shapes relative to the autonomous vehicle; and presenting the nearby object display information on a visual display system to confirm the autonomous vehicle's perception of the one or more pedestrians.

[0009] may include one or more of the following features. Identifying one or more humanoid shapes within the sensed information may include: comparing one or more defined humanoid shapes with one or more unrecognized objects within the sensed information to identify one or more humanoid shapes within the sensed information. Nearby object display information may include dynamic nearby object display information that varies as the position of the one or more detected humanoid shapes relative to the autonomous vehicle changes. Presenting the nearby object display information on a visual display system may include: presenting the dynamic nearby object display information on the visual display system to dynamically confirm the autonomous vehicle's perception of the one or more pedestrians. The one or more machine vision sensors may include a lidar system. The visual display system may be configured to be mounted on the roof of the autonomous vehicle. The visual display system may be a cylindrical visual display system. The cylindrical visual display system may include: an illuminated portion; and an unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle. The visual display system may be a 360-degree visual display system. The visual display system may be integrated into the autonomous vehicle.

[0010] In another embodiment, a computer program product resides on a computer-readable medium and has a plurality of instructions stored on the computer-readable medium. When executed by a processor, the instructions cause the processor to perform operations that include: monitoring one or more machine vision sensors to obtain sensed information regarding one or more pedestrians near the autonomous vehicle; identifying one or more humanoid shapes within the sensed information to define one or more detected humanoid shapes; generating nearby object display information that positions the one or more detected humanoid shapes relative to the autonomous vehicle; and presenting the nearby object display information on a visual display system to confirm the autonomous vehicle's perception of the one or more pedestrians.

[0011] It may include one or more of the following features. Identifying one or more humanoid shapes in the sensed information may include: comparing one or more defined humanoid shapes with one or more unrecognized objects in the sensed information to identify one or more humanoid shapes in the sensed information. The nearby object display information may include dynamic nearby object display information, and the dynamic nearby object display information changes as the position of the one or more detected humanoid shapes relative to the autonomous vehicle changes. Presenting the nearby object display information on the visual display system may include: presenting the dynamic nearby object display information on the visual display system to dynamically confirm the autonomous vehicle's perception of the one or more pedestrians. The one or more machine vision sensors may include a lidar system. The visual display system may be configured to be mounted on the roof of the autonomous vehicle. The visual display system may be a cylindrical visual display system. The cylindrical visual display system may include: an illuminated portion; and an unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle. The visual display system may be a 360-degree visual display system. The visual display system may be integrated into the autonomous vehicle.

[0012] In another embodiment, the computing system includes a processor and a memory, and the processor and the memory are configured to perform operations, and the operations include: monitoring one or more machine vision sensors to obtain sensed information about one or more pedestrians near the autonomous vehicle; identifying one or more humanoid shapes in the sensed information to define one or more detected humanoid shapes; generating nearby object display information that locates the one or more detected humanoid shapes relative to the autonomous vehicle; and presenting the nearby object display information on the visual display system to confirm the autonomous vehicle's perception of the one or more pedestrians.

[0013] may include one or more of the following features. Identifying one or more humanoid shapes within the sensed information may include: comparing one or more defined humanoid shapes with one or more unrecognized objects within the sensed information to identify one or more humanoid shapes within the sensed information. The nearby object display information may include dynamic nearby object display information that varies as the position of the one or more detected humanoid shapes relative to the autonomous vehicle changes. Presenting the nearby object display information on the visual display system may include: presenting the dynamic nearby object display information on the visual display system to dynamically confirm the autonomous vehicle's perception of the one or more pedestrians. The one or more machine vision sensors may include a lidar system. The visual display system may be configured to be mounted on the roof of the autonomous vehicle. The visual display system may be a cylindrical visual display system. The cylindrical visual display system may include: an illuminated portion; and an unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle. The visual display system may be a 360-degree visual display system. The visual display system may be integrated into the autonomous vehicle.

[0014] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of an autonomous vehicle according to an embodiment of the present disclosure.

[0016] Figure 2A is included in an embodiment of the present disclosure Figure 1 a schematic diagram of an embodiment of various systems within the autonomous vehicle.

[0017] Figure 2B is included in an embodiment of the present disclosure Figure 1 a schematic diagram of another embodiment of various systems within the autonomous vehicle.

[0018] Figure 3 is included in an embodiment of the present disclosure Figure 1 a schematic diagram of another embodiment of various systems within the autonomous vehicle.

[0019] Figures 4A - 4C is an embodiment of the present disclosure Figure 1 a schematic diagram of the status indication system of the autonomous vehicle.

[0020] Figure 5 is an embodiment of a status indication process performed on one or more systems of an autonomous vehicle according to an embodiment of the present disclosure Figure 1 a flowchart of an embodiment of the status indication process.

[0021] Figures 6A - 6C is of an embodiment of the present disclosure Figure 1 schematic diagram of a status indication system for an autonomous vehicle.

[0022] Figure 7 is of an embodiment of the present disclosure in Figure 1 flowchart of another embodiment of a status indication process executed on one or more systems of an autonomous vehicle.

[0023] Figure 8 is of an embodiment of the present disclosure Figure 1 schematic diagram of a status indication system for an autonomous vehicle; and

[0024] Figure 9A FIG. - 9B is of an embodiment of the present disclosure Figure 1 schematic diagram of a status indication system for an autonomous vehicle.

[0025] Like reference numerals in the figures represent like elements. Detailed Description

[0026] Overview of Autonomous Vehicle

[0027] Referring to Figure 1 , an autonomous vehicle 10 is shown. As is known in the art, an autonomous vehicle (e.g., autonomous vehicle 10) is a vehicle capable of sensing its environment and moving with little or no human input. An autonomous vehicle (e.g., autonomous vehicle 10) may incorporate various sensor systems to sense its surrounding environment, examples of which may include but are not limited to radar, computer vision, light detection and ranging (LIDAR), global positioning system (GPS), odometer, temperature, and inertia, wherein these sensor systems may be configured to interpret lanes and markings on the road, road signs, traffic lights, pedestrians, other vehicles, roadside objects, hazards, etc.

[0028] Autonomous vehicle 10 may include a plurality of sensors (e.g., sensor 12), a plurality of electronic control units (e.g., ECU 14), and a plurality of actuators (e.g., actuator 16). Thus, the sensors 12 within the autonomous vehicle 10 can monitor the environment in which the autonomous vehicle 10 is located, where the sensors 12 can provide sensor data 18 to the ECU 14. The ECU 14 can process the sensor data 18 to determine how the autonomous vehicle 10 should move. Then, the ECU 14 can provide control data 20 to the actuator 16 so that the autonomous vehicle 10 can move in the manner determined by the ECU 14. For example, a machine vision sensor included within the sensors 12 can "read" a speed limit sign indicating that the speed limit on the road on which the autonomous vehicle 10 is traveling is now 35 miles per hour. This machine vision sensor included within the sensors 12 can provide the sensor data 18 to the ECU 14 to indicate that the speed limit on the road on which the autonomous vehicle 10 is traveling is now 35 miles per hour (mph). When receiving the sensor data 18, the ECU 14 can process the sensor data 18 and can determine that the autonomous vehicle 10 (currently traveling at 45 miles per hour) is traveling too fast and needs to slow down. Thus, the ECU 14 can provide the control data 20 to the actuator 16, where the control data 20 can, for example, apply the brakes of the autonomous vehicle 10 or cancel any actuation signal currently applied to the accelerator (thereby allowing the autonomous vehicle 10 to coast until the speed of the autonomous vehicle 10 decreases to 35 miles per hour).

[0029] System redundancy

[0030] It is conceivable that since the autonomous vehicle 10 is controlled by various electronic systems (e.g., sensors 12, ECU 14, and actuator 16) included therein, when designing the autonomous vehicle 10, one or more possible failures of these systems should be considered, and appropriate contingency plans can be adopted.

[0031] For example, also referring to Figure 2A , the various ECUs (e.g., ECU 14) included within the autonomous vehicle 10 can be partitioned so that the responsibilities of the various ECUs (e.g., ECU 14) can be logically grouped. For example, the ECU 14 can include an autonomous control unit 50 that can receive the sensor data 18 from the sensors 12.

[0032] The autonomous control unit 50 can be configured to perform various functions. For example, the autonomous control unit 50 can receive and process external sensor data (e.g., sensor data 18), can estimate the position of the autonomous vehicle 10 within its operating environment, can calculate a representation of the surrounding environment of the autonomous vehicle 10, can calculate a safe trajectory for the autonomous vehicle 10, and can command other ECUs (notably the vehicle control unit) to cause the autonomous vehicle 10 to perform the required maneuvers.

[0033] The autonomous control unit 50 can include powerful computing capabilities, persistent memory, and memory.

[0034] Thus, the autonomous control unit 50 can process the sensor data 18 to determine how the autonomous vehicle 10 should operate. Then, the autonomous control unit 50 can provide vehicle control data 52 to the vehicle control unit 54, whereupon the vehicle control unit 54 can process the vehicle control data 52 to determine how the respective control systems (e.g., the powertrain system 56, the braking system 58, and the steering system 60) should respond to achieve the trajectory defined within the vehicle control data 52 by the autonomous control unit 50.

[0035] The vehicle control unit 54 can be configured to control other ECUs included within the autonomous vehicle 10. For example, the vehicle control unit 54 can control the steering, powertrain, and braking controller units. For example, the vehicle control unit 54 can: provide a powertrain control signal 62 to the powertrain control unit 64; provide a braking control signal 66 to the braking control unit 68; and provide a steering control signal 70 to the steering control unit 72.

[0036] The powertrain control unit 64 can process the powertrain control signal 62 so that appropriate control data (commonly represented by control data 20) can be provided to the powertrain system 56. Additionally, the braking control unit 68 can process the braking control signal 66 so that appropriate control data (commonly represented by control data 20) can be provided to the braking system 58. Additionally, the steering control unit 72 can process the steering control signal 70 so that appropriate control data (commonly represented by control data 20) can be provided to the steering system 60.

[0037] The powertrain control unit 64 can be configured to control the transmission (not shown) and the engine / traction motor (not shown) within the autonomous vehicle 10; while the braking control unit 68 can be configured to control the mechanical / regenerative braking system (not shown) within the autonomous vehicle 10; and the steering control unit 72 can be configured to control the steering column / bogie (not shown) within the autonomous vehicle 10.

[0038] The autonomous control unit 50 can be a highly complex computing system that can provide a wide range of processing capabilities (e.g., a workstation-class computing system with a multi-core processor, discrete co-processing units, thousands of megabytes of memory, and persistent storage). In contrast, the vehicle control unit 54 can be a much simpler device that can provide processing capabilities comparable to other ECUs included within the autonomous vehicle 10 (e.g., a computing system with a modest microprocessor (CPU frequency below 200 megahertz), less than 1 megabyte of system memory, and no persistent storage). Due to these simpler designs, the vehicle control unit 54 may have greater reliability and durability than the autonomous control unit 50.

[0039] To further increase redundancy and reliability, one or more ECUs (ECU 14) included within the autonomous vehicle 10 can be configured in a redundant manner. For example, also referring to Figure 2B , an embodiment of the ECU 14 is shown, in which multiple vehicle control units are utilized. For example, this particular embodiment is shown as including two vehicle control units, namely a first vehicle control unit (e.g., vehicle control unit 54) and a second vehicle control unit (e.g., vehicle control unit 74).

[0040] In this particular configuration, the two vehicle control units (e.g., vehicle control units 54, 74) can be configured in various ways. For example, the two vehicle control units (e.g., vehicle control units 54, 74) can be configured in an active-passive configuration, where, for example, the vehicle control unit 54 performs the active role of processing vehicle control data 52, while the vehicle control unit 74 assumes a passive role and is essentially in a standby mode. In the event of a failure of the vehicle control unit 54, the vehicle control unit 74 can transition from the passive role to the active role and assume the role of processing vehicle control data 52. Alternatively, the two vehicle control units (e.g., vehicle control units 54, 74) can be configured in an active-active configuration, where, for example, both the vehicle control unit 52 and the vehicle control unit 74 perform the active role of processing vehicle control data 54 (e.g., sharing the workload), where, in the event of a failure of either the vehicle control unit 54 or the vehicle control unit 74, the surviving vehicle control unit can process all of the vehicle control data 52.

[0041] Although Figure 2BAn example of a way in which various ECUs (e.g., ECU 14) included within the autonomous vehicle 10 can be configured in a redundant manner is illustrated, but this is for illustrative purposes only and is not intended to be a limitation of the present disclosure, as other configurations are possible and are considered to be within the scope of the present disclosure. For example, the autonomous control unit 50 can be configured in a redundant manner, wherein a second autonomous control unit (not shown) is included within the autonomous vehicle 10 and is configured in an active - passive or active - active manner. Additionally, it is foreseeable that one or more sensors (e.g., sensor 12) and / or one or more actuators (e.g., actuator 16) can be configured in a redundant manner. Thus, it can be understood that, with respect to the autonomous vehicle 10, the degree of redundancy achievable may be limited only by the design criteria and budget constraints of the autonomous vehicle 10.

[0042] Autonomous computing subsystem

[0043] Also refer to Figure 3 , the various ECUs of the autonomous vehicle 10 can be grouped / arranged / configured to achieve various functions.

[0044] For example, one or more of the ECUs 14 can be configured to implement / form a perception subsystem 100, wherein the perception subsystem 100 can be configured to process data from on - vehicle sensors (e.g., sensor data 18) to compute a concise representation of objects of interest near the autonomous vehicle 10 (examples of which may include but are not limited to other vehicles, pedestrians, traffic signals, traffic signs, road markers, hazards, etc.), and to identify environmental features that may help to determine the position of the autonomous vehicle 10. Additionally, one or more of the ECUs 14 can be configured to implement / form a state estimation subsystem 102, wherein the state estimation subsystem 102 can be configured to process data from on - vehicle sensors (e.g., sensor data 18) to estimate the position, orientation, and speed of the autonomous vehicle 10 within its operating environment. Additionally, one or more of the ECUs 14 can be configured to implement / form a planning subsystem 104, wherein the planning subsystem 104 can be configured to compute a desired vehicle trajectory (using the perception output 106 and the state estimation output 108). Additionally, one or more of the ECUs 14 can be configured to implement / form a trajectory control subsystem 110, wherein the trajectory control subsystem 110 uses the planning output 112 and the state estimation output 108 (in combination with feedback and / or feed - forward control techniques) to compute actuator commands (e.g., control data 20) that can cause the autonomous vehicle 10 to execute its intended trajectory within its operating environment.

[0045] For redundancy purposes, the above-described subsystems may be distributed across various devices (e.g., the autonomous control unit 50 and the vehicle control units 54, 74). Additionally / alternatively, due to the increasing computational requirements, the perception subsystem 100 and the planning subsystem 104 may be located almost entirely within the autonomous control unit 50, which (as described above) has more computational horsepower than the vehicle control units 54, 74. Conversely, due to the lower computational requirements of the state estimation subsystem 102 and the trajectory control subsystem 110, they may be located entirely on the vehicle control units 54, 74 if the vehicle control units 54, 74 have the necessary computational capabilities; and / or may be partially located on the vehicle control units 54, 74 and partially located on the autonomous control unit 50. However, the location of the state estimation subsystem 102 and the trajectory control subsystem 110 may be crucial in the design of any contingency plan architecture, as the location of these subsystems may determine how the contingency plan is computed, transmitted, and / or executed.

[0046] Status Indication System

[0047] Also refer to Figure 4A 、 4B 、4C, which shows an external view of the autonomous vehicle 10, where the autonomous vehicle 10 may include a status indication system 200 for communicating status information about the movable vehicle (e.g., the autonomous vehicle 10).

[0048] The status indication system 200 may include an interface system (e.g., the interface system 202) configured to receive perception information (e.g., the perception information 204) about one or more objects (e.g., the objects 206, 208, 210, 212) near the movable vehicle (e.g., the autonomous vehicle 10). One or more objects (e.g., the objects 206, 208, 210, 212) near the movable vehicle (e.g., the autonomous vehicle 10) may include one or more of the following:

[0049] · Pedestrians (e.g., the objects 208, 210, 212) near the movable vehicle (e.g., the autonomous vehicle 10), e.g., a person walking / passing by the autonomous vehicle 10;

[0050] · Third-party vehicles (e.g., the object 206) near the movable vehicle (e.g., the autonomous vehicle 10), e.g., other vehicles traveling / stopping near the autonomous vehicle 10.

[0051] As described above, the sensors 12 within the autonomous vehicle 10 can monitor the environment in which the autonomous vehicle 10 is located. Among them, the sensors 12 can provide sensor data 18 to the ECU 14, and the sensor data can be processed to determine how the autonomous vehicle 10 should operate. The interface system 202 can be configured to interface with the sensors 12 generally and one or more machine vision sensors specifically included within a movable vehicle (e.g., the autonomous vehicle 10). Examples of such machine vision sensors can include, but are not limited to, lidar systems.

[0052] As is known in the art, lidar is a method of determining range (variable distance) by aiming a laser at an object and measuring the time it takes for the reflected light to return to the receiver. Due to the differences in laser return times and by changing the laser wavelength, lidar can also be used to create a digital 3-D representation of areas on the Earth's surface and the ocean floor. It has terrestrial, aerial, and mobile applications. Lidar is an acronym for "light detection and ranging" or "laser imaging, detection, and ranging". Lidar is sometimes referred to as 3-D laser scanning and is a special combination of 3-D scanning and laser scanning. Lidar is commonly used to create high-resolution maps and is applied in surveying, geodesy, geography, geology, geomorphology, seismology, forestry, atmospheric physics, laser guidance, airborne laser scanning mapping (ALSM), and laser altimetry. This technology is also used for the control and navigation of some autonomous vehicles.

[0053] The status indication system 200 can include a processing system (e.g., the processing system 214), which is configured to process perception information (e.g., the perception information 204) to generate nearby object display information (e.g., the nearby object display information 216). The processing system 214 can be configured in various ways. One example of the processing system 214 can include a stand-alone processing system, which includes one or more processors (not shown) and one or more memory architectures (not shown). Another example of the processing system 214 can include a part of the ECU 14. The processing system 214 can be coupled to a storage device (e.g., the storage device 114). Examples of the storage device 114 can include, but are not limited to: hard disk drives; RAID devices; random access memory (RAM); read-only memory (ROM); and all forms of flash storage devices.

[0054] The status indication system 200 can include a visual display system (e.g., the visual display system 218), which is configured to present the nearby object display information (e.g., the nearby object display information 216). As will be discussed in more detail below, the visual display system (e.g., the visual display system 218) can be configured to visually convey status information about the autonomous vehicle 10 in a manner that is easily understandable to people near the autonomous vehicle 10.

[0055] In addition, an interface system (e.g., interface system 202) can be configured to receive movable vehicle status information (e.g., movable vehicle status information 220). As described above, sensors 12 within autonomous vehicle 10 can monitor the environment in which autonomous vehicle 10 is located, where sensors 12 can provide sensor data 18 to ECU 14. Thus, the interface system (e.g., interface system 202) can obtain such movable vehicle status information (e.g., movable vehicle status information 220) from ECU 14.

[0056] Accordingly, the movable vehicle status information (e.g., movable vehicle status information 220) can identify one or more of the following:

[0057] · Whether the movable vehicle is decelerating;

[0058] · Whether the movable vehicle is accelerating;

[0059] · Whether the movable vehicle is stationary; and

[0060] · Whether the movable vehicle is turning.

[0061] A processing system (e.g., processing system 214) can be configured to process the movable vehicle status information (e.g., movable vehicle status information 220) to generate vehicle status display information (e.g., vehicle status display information 222), where a visual display system (e.g., visual display system 218) can be configured to present the vehicle status display information (e.g., vehicle status display information 222). As will be discussed in more detail below, the visual display system (e.g., visual display system 218) can be configured to visually convey status information about autonomous vehicle 10 in a manner that is easily understandable to a person near autonomous vehicle 10.

[0062] The visual display system (e.g., visual display system 218) can be configured to be mounted on the roof (e.g., roof 224) of a movable vehicle (e.g., autonomous vehicle 10). The visual display system (e.g., visual display system 218) can be configured in various ways, all of which are considered to be within the scope of the present disclosure.

[0063] Columnar: The visual display system (e.g., visual display system 218) can be a columnar visual display system as Figures 4A - 4BAs shown). When the visual display system 218 is configured in a columnar manner, the visual display system 218 may include: an illuminated portion (e.g., illuminated portion 226); and an unilluminated portion (e.g., unilluminated portion 228) located between the illuminated portion (e.g., illuminated portion 226) and the roof (e.g., roof 224) of the movable vehicle (e.g., autonomous vehicle 10), thereby providing a "floating" appearance relative to the roof (e.g., roof 224) of the movable vehicle (e.g., autonomous vehicle 10).

[0064] Disk-shaped: The visual display system (e.g., visual display system 218) may be a disk-shaped visual display system as Figure 4C shown). When the visual display system 218 is configured in a disk-shaped manner, the visual display system 218 may be generally illuminated at a location on the roof (e.g., roof 224) near the movable vehicle (e.g., autonomous vehicle 10), thereby providing a lower profile, which may be desirable when used for vehicles with a higher profile (e.g., SUVs and vans).

[0065] Regardless of the configuration, the visual display system (e.g., visual display system 218) may be a 360-degree visual display system, thereby visually conveying status information about the autonomous vehicle 10 regardless of whether people are located in a position near the autonomous vehicle 10.

[0066] Although the visual display system (e.g., visual display system 218) has been described above as being mounted on the roof (e.g., roof 224) of the movable vehicle (e.g., autonomous vehicle 10), this is for illustrative purposes only and is not intended to be a limitation of the present disclosure, as other configurations are possible and are considered to be within the scope of the present disclosure. For example, the visual display system (e.g., visual display system 218) may be integrated into the movable vehicle (e.g., autonomous vehicle 10) in various ways. Thus, the visual display system (e.g., visual display system 218) may be part of (or incorporated into) a window (e.g., window 230) of the movable vehicle (e.g., autonomous vehicle 10).

[0067] Status indication process (for pedestrians)

[0068] As described above, the visual display system (e.g., visual display system 218) may be configured to visually convey status information about the autonomous vehicle 10 in a visual manner that is easily understandable to people near the autonomous vehicle 10.

[0069] Also refer to Figure 5 , to convey such status information in this visual manner, the processing system 214 may execute a status indication process 250. As described above, the processing system 214 may be configured in various ways, examples of which may include but are not limited to:

[0070] · An independent processing system, which includes one or more processors (not shown) and one or more memory architectures (not shown); and / or

[0071] · A part of ECU 14.

[0072] The instruction set and subroutines of the status indication process 250 can be stored on a storage device 114 coupled to ECU 14 and can be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within ECU 14. Examples of the storage device 114 can include but are not limited to: hard disk drives; RAID devices; random access memory (RAM); read-only memory (ROM); and all forms of flash storage devices.

[0073] The status indication process 250 can be executed on a single ECU or can be executed collaboratively across multiple ECUs. For example, the status indication process 250 can be executed only by the autonomous control unit 50, the vehicle control unit 54, or the vehicle control unit 74. Alternatively, the status indication process 250 can be executed collaboratively across a combination of the autonomous control unit 50, the vehicle control unit 54, and the vehicle control unit 74. Thus, in the latter configuration, in the event of a failure in one of the autonomous control unit 50, the vehicle control unit 54, or the vehicle control unit 74, the surviving control units can continue to execute the status indication process 250.

[0074] The status indication process 250 can monitor 252 one or more machine vision sensors (e.g., general sensor 12 and specific lidar sensors) to obtain perception information (e.g., perception information 204) about one or more pedestrians (e.g., objects 208, 210, 212) near the autonomous vehicle (e.g., autonomous vehicle 10). The perception information 204 can be a three-dimensional image to generally identify all objects in the geographical vicinity of the autonomous vehicle 10.

[0075] The status indication process 250 can identify 254 one or more humanoid shapes within the perception information (e.g., perception information 204), thereby defining one or more detected humanoid shapes. When identifying 254 one or more humanoid shapes within the perception information (e.g., perception information 204), the status indication process 250 can compare 256 one or more defined humanoid shapes (e.g., defined humanoid shape 232) with one or more unidentified objects within the perception information (e.g., perception information 204) to identify one or more humanoid shapes within the perception information (e.g., perception information 204).

[0076] The defined humanoid shape 232 can be manually defined (e.g., by the designer / programmer of the status indication process 250) and / or automatically defined (e.g., by artificial intelligence / machine learning (AI / ML) in a manner similar to how AI / ML can recognize a human face within a photo). Generally, the perception information 204 can be a three-dimensional image to generally identify multiple unrecognized objects in the geographical vicinity of the autonomous vehicle 10. Thus, the status indication process 250 can compare 256 the defined humanoid shape 232 with one or more unrecognized objects within the perception information (e.g., the perception information 204) to identify one or more humanoid shapes within the perception information (e.g., the perception information 204).

[0077] Once a humanoid shape 254 within the perception information 204 is recognized, the status indication process 250 can generate 258 nearby object display information (e.g., the nearby object display information 216) that positions the one or more detected humanoid shapes relative to the autonomous vehicle (e.g., the autonomous vehicle 10). Then, the status indication process 250 can present 260 the nearby object display information (e.g., the nearby object display information 216) on a visual display system (e.g., the visual display system 218), thereby confirming the perception of the autonomous vehicle (e.g., the autonomous vehicle 10) of one or more pedestrians (e.g., the objects 208, 210, 212).

[0078] For example, Figure 6A illustrates a situation where two pedestrians (e.g., pedestrians 300, 302) are standing still near the side of the autonomous vehicle 10. Thus, the nearby object display information 216 can position one or more detected humanoid shapes (e.g., pedestrians 300, 302) relative to the autonomous vehicle 10. Additionally, the visual display system 218 of the status indication system 200 can visually convey status information about the autonomous vehicle 10 in a manner that is easily understandable to people near the autonomous vehicle 10 (e.g., pedestrians 300, 302). Thus, the visual display system 218 can present a first sign (e.g., the sign 304) that points to (or corresponds to) the pedestrian 300. Since the pedestrian 300 is stationary, the sign 304 can be stationary within the visual display system 302. Additionally, the visual display system 218 can present a second sign (e.g., the sign 306) that points to (or corresponds to) the pedestrian 302. Since the pedestrian 302 is stationary, the sign 306 can be stationary within the visual display system 302.

[0079] In addition, Figure 6BIllustrates a situation where a pedestrian (e.g., pedestrian 300, 308) stands still near the front of the autonomous vehicle 10. Thus, the nearby object display information 216 can localize one or more detected humanoid shapes (e.g., pedestrian 308) relative to the autonomous vehicle 10. In addition, the visual display system 218 of the status indication system 200 can visually convey status information about the autonomous vehicle 10 in a visually understandable manner for a person near the autonomous vehicle 10 (e.g., pedestrian 308). Thus, the visual display system 218 can present a sign (e.g., sign 310) that points to (or corresponds to) the pedestrian 308. Since the pedestrian 308 is stationary, the indicator 310 can be stationary within the visual display system 302. In addition, since the pedestrian 310 is in the path of the autonomous vehicle 10 (i.e., obstructive), the sign 310 can provide an instruction to the pedestrian 310, in the form of, for example, a left-facing arrow, to request that the pedestrian 308 move left and out of the path of the autonomous vehicle 10.

[0080] The nearby object display information (e.g., nearby object display information 216) can include dynamic nearby object display information (e.g., nearby object display information 216) that changes as the position of one or more detected humanoid shapes changes relative to the autonomous vehicle (e.g., autonomous vehicle 10). Thus, when presenting 260 the nearby object display information (e.g., nearby object display information 216) on the visual display system (e.g., visual display system 218), the status indication process 250 can present 262 the dynamic nearby object display information (e.g., nearby object display information 216) on the visual display system (e.g., visual display system 218) to dynamically confirm the perception of the autonomous vehicle (e.g., autonomous vehicle 10) of the one or more pedestrians (e.g., objects 208, 210, 212).

[0081] For example, Figure 6CIllustrates a situation where two pedestrians (e.g., pedestrians 312, 314) are walking in front of the autonomous vehicle 10. Thus, the nearby object display information 216 can localize one or more detected humanoid shapes (e.g., pedestrians 312, 314) relative to the autonomous vehicle 10. In addition, the visual display system 218 of the status indication system 200 can visually convey status information about the autonomous vehicle 10 in a manner that is easily understandable to a person near the autonomous vehicle 10 (e.g., pedestrians 312, 314). Accordingly, the visual display system 218 can present a first sign (e.g., sign 316) that points to (or corresponds to) pedestrian 312. As pedestrian 312 moves from left to right, sign 316 can also move from left to right within the visual display system 302. In addition, the visual display system 218 can present a second sign (e.g., sign 318) that points to (or corresponds to) pedestrian 314. When pedestrian 314 moves from left to right, indicator 318 can also move from left to right within the visual display system 302.

[0082] Status indication process (for a vehicle)

[0083] In a manner similar to the status indication process 10 for tracking pedestrians near the autonomous vehicle (e.g., autonomous vehicle 10), the status indication process 10 can also track vehicles near the autonomous vehicle (e.g., autonomous vehicle 10).

[0084] For example, also referring to Figure 7 , the status indication process 250 can monitor 350 one or more machine vision sensors (e.g., general sensor 12 and specific lidar sensor) to obtain perception information (e.g., perception information 204) about one or more third-party vehicles (e.g., object 206) near the autonomous vehicle (e.g., autonomous vehicle 10).

[0085] The status indication process 250 can identify 352 one or more vehicle shapes within the perception information (e.g., perception information 204), thereby defining one or more detected vehicle shapes. When identifying 352 one or more vehicle shapes within the perception information (e.g., perception information 204), the status indication process 250 can compare 354 one or more defined vehicle shapes (e.g., defined vehicle shape 234) with one or more unrecognized objects within the perception information (e.g., perception information 204) to identify one or more vehicle shapes within the perception information (e.g., perception information 204).

[0086] The defined vehicle shape 234 can be manually defined (e.g., by the designer / programmer of the status indication process 250) and / or automatically defined (e.g., by artificial intelligence / machine learning (AI / ML) in a manner similar to how AI / ML can identify vehicles within a photo). Generally, the perception information 204 can be a three-dimensional image to commonly identify multiple unrecognized objects in the geographical vicinity of the autonomous vehicle 10. Thus, the status indication process 250 can compare 354 the defined vehicle shape 234 with one or more unrecognized objects within the perception information (e.g., the perception information 204) to identify one or more vehicle shapes within the perception information (e.g., the perception information 204).

[0087] Once a vehicle shape 352 within the perception information 204 is identified, the status indication process 250 can generate 356 nearby object display information (e.g., the nearby object display information 216) that positions one or more detected vehicle shapes relative to the autonomous vehicle (e.g., the autonomous vehicle 10). Then, the status indication process 250 can present 358 the nearby object display information (e.g., the nearby object display information 216) on a visual display system (e.g., the visual display system 218), thereby confirming the perception of the autonomous vehicle (e.g., the autonomous vehicle 10) of the one or more third-party vehicles (e.g., the object 206).

[0088] The nearby object display information (e.g., the nearby object display information 216) can include dynamic nearby object display information (e.g., the nearby object display information 216) that changes as the position of one or more detected vehicle shapes changes relative to the autonomous vehicle (e.g., the autonomous vehicle 10). Accordingly, when presenting 358 the nearby object display information (e.g., the nearby object display information 216) on the visual display system (e.g., the visual display system 218), the status indication process 250 can present 360 the dynamic nearby object display information (e.g., the nearby object display information 216) on the visual display system (e.g., the visual display system 218), thereby dynamically confirming the perception of the autonomous vehicle (e.g., the autonomous vehicle 10) of the one or more third-party vehicles (e.g., the object 206).

[0089] As Figure 6A 、 Figure 6B 、 Figure 6CAs shown, the visual display system 218 can present various signs (e.g., signs 304, 306, 310, 316, 318), where each of these signs (e.g., signs 304, 306, 310, 316, 318) is shown to include, for example, a "stick figure" to indicate that the sign (e.g., signs 304, 306, 310, 316, 318) points to (or corresponds to) a pedestrian (e.g., pedestrians 300, 302, 308, 312, 314). Accordingly, also referring to Figure 8 , when the status indication process 10 is tracking a vehicle near the autonomous vehicle 10, the visual display system 218 can present various signs (e.g., sign 320), including, for example, a "car diagram", to indicate that the sign (e.g., sign 320) points to (or corresponds to) a vehicle (e.g., a third-party vehicle 322).

[0090] Status indication process (for AV status)

[0091] As described above, the interface system (e.g., interface system 202) can be configured to receive movable vehicle status information (e.g., movable vehicle status information 220), where this movable vehicle status information (e.g., movable vehicle status information 220) can identify one or more of the following:

[0092] · Whether the movable vehicle is decelerating;

[0093] · Whether the movable vehicle is accelerating;

[0094] · Whether the movable vehicle is stationary; and

[0095] · Whether the movable vehicle is turning.

[0096] The processing system (e.g., processing system 214) can be configured to process the movable vehicle status information (e.g., movable vehicle status information 220) to generate vehicle status display information (e.g., vehicle status display information 222), where the visual display system (e.g., visual display system 218) can be configured to present the vehicle status display information (e.g., vehicle status display information 222).

[0097] Thus, as Figure 9A shown, the nearby object portion 350 of the visual display system 218 can be configured to track pedestrians near the autonomous vehicle (e.g., autonomous vehicle 10). In addition, the status portion 352 of the visual display system 218 can be configured to present the vehicle status display information 222. For example, the status portion 352 can include:

[0098] · A "left turn" section 354, which can be illuminated when the autonomous vehicle 10 plans to turn left;

[0099] · A "right turn" section 356, which can be illuminated when the autonomous vehicle 10 plans to turn right;

[0100] · A status section 358, which can:

[0101] i. Indicate that the autonomous vehicle 10 is cruising in a stable state by, for example, displaying pure white,

[0102] ii. Indicate that the autonomous vehicle is accelerating by, for example, displaying a color that changes from white to dark green,

[0103] iii. Indicate that the autonomous vehicle is decelerating by, for example, displaying a color that changes from white to dark red, and

[0104] iv. Indicate that the autonomous vehicle is stationary by, for example, displaying pure red.

[0105] In addition, as Figure 9B shown, the nearby object portion 350 of the visual display system 218 can be configured to track a third-party vehicle near the autonomous vehicle (e.g., the autonomous vehicle 10). Similarly, the status portion 352 of the visual display system 218 can be configured to present vehicle status display information 222 (as described above).

[0106] Summary

[0107] As will be understood by those skilled in the art, the present disclosure may be embodied as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may generally be referred to herein as a "circuit", "module", or "system". In addition, the present disclosure may take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied in the medium.

[0108] Any suitable computer-usable or computer-readable medium can be utilized. A computer-usable or computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium can include the following: an electrical connector having one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a transmission medium such as supporting the Internet or an intranet, or a magnetic storage device. The computer-usable or computer-readable medium can also be paper or another suitable medium, on which the program is printed, as the program can be electronically captured via, for example, optical scanning of the paper or other medium, then compiled, interpreted, or processed in a suitable manner if necessary, and then stored in a computer memory. In the context of this article, a computer-usable or computer-readable medium can be any medium that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer-usable medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the computer-usable program code is embodied. The computer-usable program code can be propagated using any appropriate medium, including but not limited to the Internet, wireline, optical fiber cable, radio frequency, etc.

[0109] The computer program code for performing the operations of the present disclosure can be written in an object-oriented programming language, such as Java, Smalltalk, C++, or similar languages. However, the computer program code for performing the operations of the present disclosure can also be written in a conventional procedural programming language, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer as a stand-alone software package, partially on the user's computer, with the stand-alone software package partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through a local area network / wide area network / Internet (e.g., network 14).

[0110] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer / special purpose computer / other programmable data processing apparatus to create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram by instructions executed by the processor of the computer or other programmable data processing apparatus.

[0111] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means for implementing the functions / acts specified in the flowchart and / or block diagram.

[0112] The computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that the instructions executed on the computer or other programmable device provide steps for implementing the functions / acts specified in the flowchart and / or block diagram.

[0113] The flowchart and block diagrams in the figures may illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose systems based on hardware for performing the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0114] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" also include the plural forms unless the context clearly dictates otherwise. It should be further understood that when the terms "comprises" and / or "comprising" are used in this specification, they specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0115] All structural, material, acts, and equivalents of the means or step plus function elements in the following claims are intended to include any structure, material, or act that performs the function in combination with other claimed elements. The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the forms disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present disclosure. The embodiments were chosen and described in order to best explain the principles of the present disclosure and its practical application, and to enable others of ordinary skill in the art to understand the present disclosure for various embodiments and with the various modifications that are suited to the particular use contemplated.

[0116] Some embodiments have been described. After the detailed description of the disclosure of the present application and reference to its embodiments, it will be apparent that modifications and variations can be made without departing from the scope of the disclosure as defined in the appended claims.

Claims

1. A computer-implemented method for an autonomous vehicle, the computer-implemented method being configured to be executed on a computing device and comprising: Monitoring one or more machine vision sensors included in the autonomous vehicle to obtain perception information about one or more pedestrians near the autonomous vehicle; Identifying one or more humanoid shapes within the perception information, thereby defining one or more detected humanoid shapes; Generating nearby object display information that locates the one or more detected humanoid shapes relative to the autonomous vehicle; And Presenting the nearby object display information on a visual display system of the autonomous vehicle to confirm the autonomous vehicle's perception of the one or more pedestrians, Wherein the presenting includes presenting a first marker pointing to or corresponding to a first pedestrian, and when the first pedestrian moves from right to left, the first marker also moves from right to left within the visual display system, and further, the presenting includes presenting a second marker pointing to or corresponding to a second pedestrian, and when the second pedestrian moves from left to right, the second marker also moves from left to right within the visual display system; Wherein the presenting includes presenting a third marker pointing to a stationary pedestrian, and the third marker remains stationary within the visual display system; Wherein the presenting includes presenting a fourth marker that, when a pedestrian is detected in the path of the autonomous vehicle, provides an instruction to the pedestrian in the form of an arrow to move out of the path of the autonomous vehicle.

2. The computer-implemented method according to claim 1, wherein, Identifying one or more humanoid shapes within the perception information includes: Comparing one or more defined humanoid shapes with one or more unrecognized objects within the perception information to identify one or more humanoid shapes within the perception information.

3. The computer-implemented method according to claim 1, wherein, The one or more machine vision sensors include a lidar system.

4. The computer-implemented method according to claim 1, wherein, The visual display system is configured to be mounted on the roof of the autonomous vehicle.

5. The computer-implemented method according to claim 1, wherein, The visual display system is a cylindrical visual display system.

6. The computer-implemented method according to claim 5, wherein, The cylindrical visual display system includes: An illuminated portion; and An unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle.

7. The computer-implemented method according to claim 1, wherein The visual display system is a 360-degree visual display system.

8. The computer-implemented method according to claim 1, wherein, The visual display system is integrated into the autonomous vehicle.

9. A computer program product residing on a computer-readable medium, having a plurality of instructions stored on the computer-readable medium, which when executed by a processor cause the processor to perform operations, the operations including: Monitoring one or more machine vision sensors included in the autonomous vehicle to obtain perception information about one or more pedestrians near the autonomous vehicle; Identifying one or more humanoid shapes within the perception information, thereby defining one or more detected humanoid shapes; Generating nearby object display information that locates the one or more detected humanoid shapes relative to the autonomous vehicle; And Presenting the nearby object display information on a visual display system of the autonomous vehicle to confirm the autonomous vehicle's perception of the one or more pedestrians, Wherein, the presentation includes presenting a first sign that points to or corresponds to a first pedestrian, and when the first pedestrian moves from right to left, the first sign also moves from right to left within the visual display system, and further, the presentation includes presenting a second sign that points to or corresponds to a second pedestrian, and when the second pedestrian moves from left to right, the second sign also moves from left to right within the visual display system; Wherein, the presentation includes presenting a third sign that points to a stationary pedestrian, and the third sign remains stationary within the visual display system; Wherein, the presentation includes presenting a fourth sign, and when a pedestrian is detected in the path of the autonomous vehicle, the fourth sign provides an instruction to the pedestrian in the form of an arrow to request the pedestrian to move out of the path of the autonomous vehicle.

10. The computer program product according to claim 9, wherein, Identifying one or more humanoid shapes in the perception information includes: Comparing one or more defined humanoid shapes with one or more unidentified objects in the perception information to identify one or more humanoid shapes in the perception information.

11. The computer program product according to claim 9, wherein, The one or more machine vision sensors include a lidar system.

12. The computer program product according to claim 9, wherein, The visual display system is configured to be mounted on the roof of the autonomous vehicle.

13. The computer program product according to claim 9, wherein, The visual display system is a cylindrical visual display system.

14. The computer program product according to claim 13, wherein, The cylindrical visual display system includes: An illuminated portion; and An unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle.

15. The computer program product according to claim 9, wherein, The visual display system is a 360-degree visual display system.

16. The computer program product according to claim 9, wherein, The visual display system is integrated into the autonomous vehicle.

17. A computing system including a processor and a memory, the processor and the memory being configured to perform operations, the operations including: Monitoring one or more machine vision sensors included in the autonomous vehicle to obtain perception information about one or more pedestrians near the autonomous vehicle; Identifying one or more humanoid shapes in the perception information to define one or more detected humanoid shapes; Generating nearby object display information that locates the one or more detected humanoid shapes relative to the autonomous vehicle; And Presenting the nearby object display information on the visual display system of the autonomous vehicle to confirm the autonomous vehicle's perception of the one or more pedestrians, Wherein, the presentation includes presenting a first sign that points to or corresponds to a first pedestrian, and when the first pedestrian moves from right to left, the first sign also moves from right to left within the visual display system, and further, the presentation includes presenting a second sign that points to or corresponds to a second pedestrian, and when the second pedestrian moves from left to right, the second sign also moves from left to right within the visual display system; Wherein, the presentation includes presenting a third sign that points to a stationary pedestrian, and the third sign remains stationary within the visual display system; Wherein, the presentation includes presenting a fourth sign, and when a pedestrian is detected in the path of the autonomous vehicle, the fourth sign provides an instruction to the pedestrian in the form of an arrow to request the pedestrian to move out of the path of the autonomous vehicle.

18. The computing system according to claim 17, wherein, Identifying one or more humanoid shapes in the perception information includes: Comparing one or more defined humanoid shapes with one or more unidentified objects in the perception information to identify the one or more humanoid shapes in the perception information.

19. The computing system according to claim 17, wherein, The one or more machine vision sensors include a lidar system.

20. The computing system according to claim 17, wherein, The visual display system is configured to be mounted on the roof of an autonomous vehicle.

21. The computing system according to claim 17, wherein, The visual display system is a cylindrical visual display system.

22. The computing system according to claim 21, wherein, The cylindrical visual display system includes: An illuminated portion; and An unilluminated portion located between the illuminated portion and the roof of the autonomous vehicle.

23. The computing system according to claim 17, wherein, The visual display system is a 360-degree visual display system.

24. The computing system according to claim 17, wherein, The visual display system is integrated into the autonomous vehicle.

Images