Cognitive-based parking assistance for autonomous machine systems and applications

Abstract

To provide perception-based parking assistance for autonomous and semi-autonomous machine applications.SOLUTION: In various examples, perception-based parking assistance systems and methods for an ego-machine are presented. Example embodiments may determine a location of a real-world parking strip relative to the ego-machine and an associated parking rule for the parking strip. A virtual parking strip and one or more virtual parking signs may be generated based at least in part on one or more detected features in an environment of the ego-machine and a tracked motion of the ego machine, and the virtual parking strip may be used to track parking strip locations and associated parking rules. The virtual parking strips and associated rules may be relied upon by an ego-machine to determine parking locations and / or to navigate into a suitable parking spot.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Autonomous or semi-autonomous machines operating on public roads and highways are expected to faithfully adhere to parking rules. For example, an autonomous delivery truck would be expected to determine where it is permitted to stop in order to find parking spots for unloading and picking up cargo, and to maintain compliance with parking regulations and avoid obstructing the operation of other machines and vehicles. Designated roadside parking lanes are a common feature of streets in urban areas where passenger cars, trucks, and other vehicles or machines may temporarily exit the flow of traffic to stop or park. Depending on how the parking area is marked and / or where the parking area is located relative to other features in its environment, a vehicle operator can determine whether parking or stopping their vehicle at a particular location is prohibited or permitted, and, if permitted, under what circumstances. Corresponding parking rules are often indicated by signs around or near the parking location, or implied by other features such as the distance to an intersection or the presence of a fire hydrant and / or painted border. The same applies to parking spaces within a parking lot where signs or other features are present to indicate which areas of the parking lot are valid parking spaces and which are prohibited parking spaces.

[0002] A location-based traffic management database represents one existing technology currently used for vehicle navigation assistance. For example, modern global positioning system (GPS) receivers often display a graphic representation of approaching road signs to the driver, such as the current speed limit or the presence of an approaching intersection or highway, and the vehicle's current position can be used by an on-board system to correlate with a database of road sign positions and street maps for instructing the driver. However, the information in such databases is static and tends to become old (inaccurate) over time. Also, there can be a very large number of objects encountered along the street that convey parking rules information. Further, temporary changes to parking rules, such as when existing parking signs are temporarily applied by a local government, are not uncommon and will not be represented in the database. As a result, these conventional systems may rely on old information and can result in illegal vehicle parking and / or unspecified parking spots.

Summary of the Invention

Means for Solving the Problems

[0003] Embodiments of the present disclosure relate to perception-based parking assistance for autonomous and semi-autonomous machine applications. A system and method are disclosed that provide perception-based parking assistance for an ego machine using data captured by on-board sensors. The parsed sensor data includes detailed parking information used to confirm both the geometry of the detected parking zones and the parking rules corresponding to those parking zones.

[0004] For example, in contrast to conventional systems such as those mentioned above, the cognitive-based parking assistance described herein calculates the geometry of a virtual parking space based at least in part on the recognized features and / or tracked motion of the ego machine. Parking signs and other features observed by cameras or other image sensors mounted on the ego machine are used to establish the start and end boundaries of the virtual parking space. As an example, when a first feature defining the start of the virtual parking space is recognized, a virtual parking sign is generated at the corresponding location on the path. The tracked motion of the ego machine is used to extend the length of the virtual parking space along the path, starting from the location of the first feature, until one or more second features indicating the end of the parking space are recognized. The recognized features may be filtered so that the features used for cognitive-based parking assistance are relevant to the ego machine's current path. Parsed sensor data may also be used to associate parking rules with the virtual parking space. Parking rules may provide, for example, the following parking-related information: parking is permitted within the virtual parking space, stopping is permitted within the virtual parking space, parking is not permitted within the virtual parking space, or stopping is not permitted within the virtual parking space. Parking rules may also indicate the time, date, and / or other conditions under which the parking rule applies. The resulting parking assistance output includes the parking rule and an output indicating the relative position of one or more virtual parking spaces to the ego machine. This parking assistance output may be used by one or more downstream components of the ego machine to manually or autonomously direct the ego machine into the parking space according to the parking rule of the parking space.

[0005] The system and method for cognitive-based parking assistance for autonomous machine applications are described in detail below with reference to the attached drawings. [Brief explanation of the drawing]

[0006] [Figure 1] This is a data flow diagram illustrating an exemplary process for a cognitive-based parking assistance system according to some embodiments of the present disclosure. [Figure 2] Block diagram showing an exemplary parking lane processor according to some embodiments of the present disclosure. [Figure 3] This figure shows an exemplary graphic display of a virtual parking space according to some embodiments of the present disclosure. [Figure 4] This flowchart illustrates a method for cognitive-based parking assistance according to some embodiments of the present disclosure. [Figure 5A] Illustrations of exemplary autonomous vehicles according to some embodiments of the present disclosure. [Figure 5B] Figure 5A shows examples of camera positions and fields of view of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 5C] Figure 5A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 5D] This is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle shown in Figure 5A, according to some embodiments of the present disclosure. [Figure 6] This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 7] This is an exemplary data center block diagram suitable for use in implementing some embodiments of the present disclosure. [Modes for carrying out the invention]

[0007] Systems and methods relating to cognitive-based parking assistance for autonomous machine applications are disclosed. This disclosure may be described in relation to an exemplary autonomous vehicle 500 (or also referred to herein as “Vehicle 500” or “Ego Machine 500,” examples thereof which are described with respect to Figures 5A–5D), but this is not intended to be limiting. For example, the systems and methods described herein may be used by non-autonomous vehicles, semi-autonomous vehicles (for example, in one or more advanced driver assistance systems (ADAS)), manned and unmanned robots or robotics platforms, warehouse vehicles, off-road vehicles, vehicles attached to one or more trailers, flying ships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. In addition, while this disclosure may describe parking assistance for either operator-driven or autonomous vehicles, this is not intended to be an limitation, and the systems and methods described herein may be used in any other technological space where augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or cognition-based operation of ego machines may be used.

[0008] This disclosure relates to the generation of cognitive-based parking information for use by autonomous or semi-autonomous machines. More specifically, the systems and methods presented in this disclosure assist an autonomous machine in parsing information obtained from its environment to determine a location where the autonomous vehicle is permitted to stop or park. Embodiments of this disclosure relate, in particular, to cognitive-based parking assistance systems and corresponding methods for parsing data captured by onboard sensors. The parsed sensor data includes detailed parking information used to verify both the geometry of detected parking lanes and the parking rules corresponding to those lanes. Onboard sensors may include, but are not limited to, one or more of the following: cameras or other optical sensors, radar sensors, ultrasonic or other SONAR sensors, LiDAR sensors, wireless beacon receivers, GPS or other navigation receivers, and / or other sensors configured to detect the characteristics of features in proximity to the autonomous machine.

[0009] With regard to geometry calculation, virtual parking lanes are generated by a cognition-based parking assistance system based at least in part on recognized features and the tracked motion of the ego machine. For example, the geometry of a virtual parking lane may be generated based on parking signs observed by a camera or other image sensor mounted on the ego machine. Parking signs unrelated to the ego machine's planned route may, in non-limiting examples, be filtered out by estimating the distance between the sign and the route. In embodiments, parking signs or other detected features may be filtered out additionally or alternatively based on criteria such as the feature being outside the ego machine's trajectory manifold, located on a different route from the ego machine, or the distance between the ego machine and the feature being greater than a threshold distance. Features related to the planned route may be stored in memory as virtual parking signs. In embodiments, virtual parking signs do not have to be of the same type as detected signs or features. For example, a red-painted border may be used to generate a virtual parking sign corresponding to the start of a no-parking zone. Similarly, detected "3-hour parking" signs can be used to generate a virtual parking sign corresponding to the start of a parking lane.

[0010] The geometry of a virtual parking lane can then be calculated from a filtered set of features in parallel with the motion of the ego machine. When a first feature defining the start of the virtual parking lane is recognized, a virtual parking sign is generated at the corresponding location on the path. The tracked motion of the ego machine can be used to extend the length of the virtual parking lane along the path, starting from the location of the first feature. That is, the length of the virtual parking lane is extended along the path based on the tracked motion of the ego machine until one or more second features indicating the end of the parking lane are recognized. The locations of one or more second features can be used to generate a second virtual parking sign and a virtual parking lane end boundary on the path. It should be noted that both the first and second features do not need to be signs, nor do they even need to be of the same type. For example, a parking assist system could generate the start of a virtual parking lane based on the detection of a roadside parking sign, and, as a non-limiting example, could generate the end of its virtual parking lane in response to the detection of intersections, surface colors along the path, and / or symbols on surfaces along the path.

[0011] Furthermore, in other instances, a single feature, such as the presence of a bus stop sign or a roadside fire hydrant, may be used to define a virtual parking area. In such instances, the length of the virtual parking area may be predefined, for example, from 304.8 cm (10 feet) before to 304.8 cm (10 feet) after a single feature, and virtual parking signs may be generated at those locations. Moreover, virtual parking areas may be generated on either side of a path based on where the detected features are located, the sequence of their appearance, and / or the information they convey (for example, as determined using one or more text recognition algorithms, computer vision algorithms, object detection algorithms, machine learning models, neural networks, and / or similar).

[0012] When virtual parking signs are generated, indicating the start and / or end boundaries of a virtual parking lane, a cognitive-based parking assistance system may calculate the relative distance of those virtual parking signs from the ego machine (e.g., from the ego machine's reference point to the projection of the virtual parking signs onto its path). Then, as the ego machine moves along its path, the tracked motion of the ego machine is also used to update the relative distances of the generated virtual parking signs from and / or from each other. As such, when developing the geometry of a virtual parking lane, the cognitive-based parking assistance system may be ambiguous with respect to any curvature in the path the ego machine is traveling. That is, the relative order of virtual parking signs from features may be maintained in memory (at least for a given duration) along with their relative distances from the ego machine, under the assumption that the processed features are on locally straight trajectories, and the relative distances to virtual parking signs in front of or behind the ego machine are updated. Maintaining virtual parking zones in memory for a certain duration allows the ego machine to move forward and backward over the least restrictive distance without having to recalculate the virtual parking zones each time it passes the same set of features. In some embodiments, passed virtual parking markers are removed from memory when their relative distance from the ego machine exceeds a threshold distance. In some embodiments, the stored parking zones may be used to update more long-term memory of the environment with the location, geometry, and / or type of detected parking zones, for example, to update a GNSS map or a high-definition (HD) map.

[0013] In addition to generating virtual parking zone geometry using detected features, parsed sensor data can be used to associate parking rules with virtual parking zones. That is, parking rules can be determined at least partially from information obtained from detected features. Parking rules can provide, for example, the following parking-related information: parking is permitted within the virtual parking zone, stopping is permitted within the virtual parking zone, parking is not permitted within the virtual parking zone, or stopping is not permitted within the virtual parking zone. Parking rules may indicate the time, date, and / or other conditions under which the parking rule applies. If parking or stopping is permitted, parking rules may also indicate whether the permission is conditional on having a valid permit, e.g., a parking permit for persons with disabilities, or limited to permitted residents, faculty, or by usage, e.g., for delivery purposes only.

[0014] In some embodiments, cognitive-based parking assistance systems may also incorporate sensor data, such as signals from navigation beacons or other navigation aids, to generate parking lane geometry and / or determine relevant parking rules. For example, a cognitive-based parking assistance system may receive location information of the ego machine from a navigation receiver (e.g., a GPS receiver, satellite navigation system, or other navigation system). Since parking rules vary by country or even within different areas of a country, a cognitive-based parking assistance system may use the geographic location information of the ego machine to determine applicable local parking rules for detected features. For example, in one community, valid legal parking may be permitted up to 304.8 cm (10 feet) from an intersection, while in another community, valid legal parking may be permitted up to 457.2 cm (15 feet) from an intersection. Such information for interpreting parking rules from features may be stored in onboard memory or data structures.

[0015] In some embodiments, the cognitive-based parking assistance system includes or communicates with an onboard feature classifier that performs analysis of sensor data captured by onboard sensors to identify features that define where the start and end of a virtual parking lane are defined, and to identify which specific parking rule the features convey. In some embodiments, the feature classifier is implemented with an artificial intelligence (AI) inference engine or other neural network technology trained to recognize features that convey parking information and to parse the corresponding parking rules from the recognized features. The onboard feature classifier may also output a confidence level indicating the reliability of the accuracy of the parking rule it determines from the features. For example, if a detected feature is partially blocked or damaged, the feature classifier may output a parking rule, but may also output a confidence level indicating an estimate of the likelihood that the determined parking rule is correct. The cognitive-based parking assistance system may then apply a confidence threshold such that it indicates the existence of a valid parking lane only if the confidence level exceeds the confidence threshold. Alternatingly, when the confidence level is below the confidence threshold, the cognitive-based parking assistance system may select and apply a default parking rule (e.g., "Parking Not Allowed") to the parking lane.

[0016] The planned paths taken by the Ego Machine are not limited to any one type of path or surface, and may include paths such as highways, paved roads, unpaved roads, private roads, parts of parking lots, paths, railway lines, rivers, footpaths, demarcated portions of the environment, or air routes, or any other situations in which the Ego Machine is expected to operate to a designated area for parking or stopping. For example, in the case of a parking lot, a cognitive-based parking assistance system can correlate features detected by sensors with parking spaces (for example, based on the projection of detected features onto parking spaces to generate virtual parking signs) and syntactically parse the parking rules for those parking spaces from the features.

[0017] The output generated by the cognitive-based parking assistance system may include parking assistance outputs indicating parking rules and the relative position of a virtual parking space to the ego machine. This parking assistance output may be used by one or more downstream components of the ego machine—for example, a world model manager, a route planner, a control component, a localization component, an obstacle avoidance component, an actuation component, and / or similar—to perform one or more actions for controlling the ego machine through the environment. In some embodiments, communication between the cognitive-based parking assistance system and such downstream components of the ego machine is implemented by the cognitive-based parking assistance system via an application programming interface (API).

[0018] For a semi-autonomous ego-machine (or an ego-machine operating in semi-autonomous mode), the relative position of a virtual parking lane may be displayed to the ego-machine operator (e.g., on a head-up windshield display or other display screen) to assist the operator in determining when the ego-machine is in or near a position where it can stop or park. Such a display may include any or all of the following: a first virtual parking lane that the ego-machine has passed through, a second virtual parking lane that the ego-machine is advancing adjacent to, and / or a third virtual parking lane that the ego-machine is about to enter, or any combination of these on either side of the path the ego-machine is traveling. In some embodiments, the display of each virtual parking lane will also include virtual parking signs located at the beginning and end of each virtual parking lane, along with a graphic representation of the parking rules associated with that lane. In some embodiments, the display of each virtual parking sign will include a standardized replica of the features from which the virtual parking sign is generated, or another graphic or icon representing the features.

[0019] In other embodiments where the ego machine has a higher degree of autonomy, the output generated by the cognitive-based parking assistance system includes a set of data that is stored in memory or otherwise transmitted to another ego machine system implementing the automatic parking navigation function. In one such embodiment, the automatic parking navigation function takes as input the output generated by the cognitive-based parking assistance system, identifies a valid virtual parking zone for the ego machine to park or stop within, and then performs its function to navigate the ego machine to the physical location associated with the valid virtual parking zone. The valid parking zone may be one that is immediately adjacent to or along the route where the ego machine is currently located, a parking zone that a preceding ego machine on the route is attempting to enter, or a parking zone that the ego machine has recently passed through.

[0020] The cognitive-based parking assistance system and corresponding method may be at least partially executed by one or more processing units coupled to memory. The processing units are programmed to execute code for implementing one or more of the features and functions of the cognitive-based parking assistance system to calculate the geometry of the virtual parking zone and to associate parking rules with the virtual parking zone. The geometry and parking rules are determined by the processing units from features indicating the parking information identified by the feature classifier. In some embodiments, all processing is performed on board the ego machine, while in other embodiments, the features and functions of the cognitive-based parking assistance system may be distributed and executed by any combination of on-board processors and cloud computing resources and sensor data obtained from on-board sensors augmented with supplementary data obtained from a data center or other server. In such an implementation, the ego machine further includes at least one wireless communication interface for connecting the cognitive-based parking assistance system to a wireless communication network.

[0021] Referring to Figure 1, Figure 1 is an illustrative data flowchart showing the interconnections and flow of information or data of components of a cognitive-based parking assistance system 100 for an ego-machine (e.g., an autonomous vehicle 500 described later with respect to Figure 5A) according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are provided merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory (e.g., a tangible memory storage device having a non-temporary physical form). In some embodiments, the systems, methods, and processes described herein may be implemented using components, features, and / or functionalities similar to those of the exemplary autonomous vehicle 500 in Figures 5A–5D, the exemplary computing device 600 in Figure 6, and / or the exemplary data center 700 in Figure 7.

[0022] As used herein, the term "parking", in conjunction with "parking spaces", "parking signs", "parking information", etc., is intended to be inclusive and applicable to both parking and stopping scenarios. Generally, stopping may include an intentional rest in an ego machine that is not due to traffic conditions or traffic control orders (e.g., when following traffic lights, yield or stop signs, or the orders of a public safety officer). An ego machine is typically considered parked when the operator leaves the ego machine and / or the ego machine is stopped for a period longer than a certain time duration (e.g., 3 minutes). If the parking rules indicate that stopping is not permitted within the parking space, that indication further implies the parking rule that parking is not permitted within that parking space. If the parking rules indicate that parking is permitted at the parking space name, that indication further implies the parking rule that stopping is also permitted within that parking space. However, a parking rule that only indicates that parking is not permitted within the parking space may imply that stopping is permitted within that parking space as a temporary stop, such as for loading or unloading goods or passengers.

[0023] As shown in Figure 1, the cognitive-based parking assistance system performing process 100 includes a parking lane processor 102 that can take feature data 104 as input and extract parking information used by the parking lane processor 102 to determine the geometry of detected parking lanes and the corresponding parking rules for those lanes. In this example, feature data 104 may be derived from sensor data 108 captured by one or more onboard sensors 110 of an ego machine (e.g., ego machine 500 in Figures 5A-5D). In the embodiment, a feature classifier 106 may be coupled to the sensors 110 to perform an analysis of the sensor data 108 captured by the sensors 110 to identify features indicating where the start and end of virtual parking lanes are defined and to identify which particular parking rules the features convey. The feature classifier 106 may be implemented with an artificial intelligence (AI) inference engine or other neural network technology trained to recognize features that convey parking information and, in some embodiments, to syntactically parse corresponding parking rule information from the recognized features. The feature classifier 106 may also output a confidence level to the parking lane processor 102, indicating the confidence level in the accuracy of the parking rule information it determines from the sensor data 108. The parking lane processor 102 may receive tracked motion data 112 as input, which may include information about the motion of the ego machine ("ego motion")—for example, the speed, velocity, attitude, and position of the ego machine as it moves along its planned path. For example, speed and velocity information may be input from the controller 536 and / or sensor 544, as described later. After the parking lane processor 102 has recognized a first feature defining the start of a virtual parking lane from the feature data 104, the tracked motion of the ego machine, as indicated by the tracked motion data 112, may be used to extend the length of the virtual parking lane along the path starting from the position of the first feature. The length of the virtual parking lane may be extended along the path based on the ego machine's tracking motion data 112 until one or more second features are recognized from the feature data 104 indicating the end of the parking lane.In the embodiment, even if the end of the current parking lane is not detected, the parking lane can be terminated after some threshold distance (e.g., 20 meters, 30 meters, etc.), and a new parking lane can be started.

[0024] As shown in Figure 1, a cognitive-based parking assistance system may include a feature classifier 106 that takes feature data 104 as input and extracts parking information used by a parking lane processor 102 to verify the geometry of detected parking lanes, as well as parking rules corresponding to those parking lanes. The feature data 104 and the extracted parking information may also be used by the parking lane processor 102 to determine and associate parking rules corresponding to virtual parking lanes. Parking rules may be determined at least in part from information obtained from detected feature data 104. Parking rules may provide, for example, the following parking-related information: parking is permitted in the virtual parking lane, stopping is permitted in the virtual parking lane, parking is not permitted in the virtual parking lane, or stopping is not permitted in the virtual parking lane. As mentioned above, in some embodiments, the parking lane processor 102 may also receive location information from a navigation receiver 114 to determine what set of local laws or regulations apply when a particular feature is detected. The navigation receiver 114 may, in non-limiting examples, include a GNSS receiver (e.g., a GPS receiver), a satellite navigation system, or other navigation system. For example, if the parking lane processor 102 detects a feature that does not inherently convey parking information (e.g., an intersection, a private road, a sidewalk ramp, an unmarked railway line, or a fire hydrant), it may refer to an effective local parking rule policy relating to the ego machine's current location to determine a parking rule for the detected feature and, accordingly, associate that rule with a virtual parking lane. In another example, detecting that the ego machine is passing through a tunnel may convey a parking rule (e.g., no parking or no stopping) unless other detected features indicate a parking rule to the contrary. Other features that convey parking information and / or parking rules may include the color of a surface along the path, or symbols on a surface along the path.For example, red on the border or surface may indicate no stopping, not moving, or parking (buses may stop in red zones marked for buses); red on the border or surface may indicate no parking, but vehicles may be permitted to stop to pick up and drop off passengers at certain times; white on the border or surface may indicate that stopping is permitted for a sufficient length of time or other limited duration for deliveries (e.g., dropping off mail at a postbox); blue on the border or surface may indicate that parking is permitted only for disabled persons; and / or green on the border or surface may indicate that parking is permitted only for limited times as posted on the border or sign.

[0025] In some embodiments, the parking area processor 102 may be connected to a wireless network interface 118 so that when the ego machine enters (or attempts to enter) an area that does not have a local parking rule policy, the ego machine can query a cloud-based service (or server) to download a local parking rule policy.

[0026] Figure 2 is a block diagram of an exemplary parking lane processor 102 in one embodiment. As shown in Figure 2, the parking lane processor 102 includes a virtual parking lane generator 210 and a parking rule policy 212. The virtual parking lane generator 210 can take feature data 104 and tracking motion data 112 as previously described and calculate the geometry of a virtual parking lane. The virtual parking lane generator 210 can also apply the feature data 104 to the parking rule policy 212 in order to assign an appropriate parking rule to the virtual parking lane. In some embodiments, when the reliability of a parking rule as determined from the feature data 104 is below a reliability threshold, the parking rule policy 212 may indicate a default parking rule (e.g., "Parking Not Allowed") to apply to the virtual parking lane. The parking rule policy 212 may be enhanced with parking rule data 216, which includes one or more local parking rule policies stored in memory. When a virtual parking lane is generated by the virtual parking lane generator 210, information about the virtual parking lane may be stored in the virtual parking lane memory 218. For example, the virtual parking space memory 218 may contain a history of previously calculated virtual parking signs generated for each virtual parking space, along with their relative distances from and / or from each other. The virtual parking space generator 210 can further update such relative distances of virtual parking signs stored in the virtual parking space memory 218 based on tracking motion data 112.

[0027] In some embodiments, parking signs or other detected features present in feature data 104 unrelated to the planned route of the ego machine can be filtered out by one or more feature filters 220. In non-limiting examples, the feature filters 220 can estimate the distance between detected features, routes, and / or ego machines and filter out those with a distance greater than a specified distance threshold. In embodiments, parking signs or other detected features can be filtered out by the feature filters 220 additionally or alternatively based on criteria such as the feature being outside the ego machine's orbital manifold or being located on a different route from the ego machine.

[0028] The output from the parking lane processor 102 may include a parking assistance output indicating parking rules and the relative position of a virtual parking lane to the ego machine. This parking assistance output may be used by one or more downstream navigation components 124 of the ego machine, such as the controller 536 described later. The downstream navigation component 124 may, for example, implement an automatic parking navigation function that takes the parking assistance output from the parking lane processor 102 as input to identify a valid virtual parking lane for the ego machine to park or stop, and then perform its function of operating the propulsion system 550 and steering system 554 to navigate the ego machine to a physical position associated with the valid virtual parking lane.

[0029] For a semi-autonomous ego-machine (or an ego-machine operating in semi-autonomous mode), parking assistance outputs may be input via a human-machine interface (HMI) 118, which includes a display for the operator of the ego-machine (e.g., on a head-up windshield display or other display screen). In the embodiment, the relative position of a virtual parking space generated by the parking space processor 102 may be displayed on the HMI 120 to assist the operator in determining when the ego-machine is in or near a position where it can stop or park. Such a display may include any or all of the following: a first virtual parking space that the ego-machine has passed through, a second virtual parking space that the ego-machine is advancing adjacent to, and / or a third virtual parking space that the ego-machine is about to enter, or any combination of these on either side of the path the ego-machine is traveling.

[0030] Figure 3 shows an exemplary graphic display 300 (for example, a display of virtual parking spaces generated by the parking space processor 102 on the HMI display 120). As shown in Figure 3, the virtual parking space generator 210 presents on the HMI display 120 a graphic representation or icon of an ego machine (shown in 305) traveling along a path 302 having an indicated direction of travel 307. In this particular example, to the right of the ego machine icon 305, the virtual parking space 310 adjacent to which the ego machine is traveling is shown on the right side of path 302. In some embodiments, the virtual parking space generator 210 may generate on the display 300 a virtual parking space 314 to the right of path 302 that the ego machine has passed through and / or a virtual parking space 312 to the right of path 302 that the ego machine is about to enter. To the left of the ego machine icon 305, the virtual parking space 320 adjacent to which the ego machine is traveling is shown on the left side of path 302. In some embodiments, the virtual parking space generator 210 may generate on the display 300 a virtual parking space 324 to the left of the path 302 that the ego machine has passed through and / or a virtual parking space 322 to the left of the path 302 that the ego machine is about to enter. It should be understood that in various implementations, the display 300 may include any one or more of the virtual parking spaces 310, 312, 314, 329, 322, or 324, and any combination thereof.

[0031] In this exemplary display 300, each virtual parking lane is shown to include virtual parking signs indicating their respective start and end boundaries. These virtual parking signs are generated by the virtual parking lane generator 210 in response to parking information detected from feature data 104, as described herein. In some embodiments, the graphics used for displaying the virtual parking signs may be obtained by the virtual parking lane generator 210 from a virtual parking lane graphics library 222, which contains standardized representations of the virtual parking signs and other features associated with the corresponding parking rules. That is, after the virtual parking lane graphics library 222 has determined which parking rules apply to the virtual parking lane, it can refer to the correct graphics for display based on the combination of those parking rules and the characteristics of the detected features. It should be understood that the specific virtual parking signs and their associated rules discussed with respect to Figure 3 are used for non-restrictive illustrative purposes only.

[0032] As shown in Figure 3, the virtual parking lane 314 to the right, which the ego machine has just passed, is marked with a no-parking start virtual parking sign 342 at its start boundary and a no-parking end virtual parking sign 340 at its end boundary, indicating that parking is prohibited within the virtual parking lane 314. For the virtual parking lane 310, which the ego machine is currently adjacent to, the first virtual parking sign 338 at its start boundary indicates that parking is permitted within the virtual parking lane 310. It should be noted that in some embodiments, the generation of virtual parking signs 338 and 340 may be triggered together by a common detected feature. For example, the detection of a parking permit sign from feature data 104 would imply that the previous no-parking rule no longer applies, so that both the no-parking end virtual parking sign 340 and the parking permit start virtual parking sign 338 may be generated. Similarly, the parking permit termination virtual parking sign 336 may be generated based on the actual detection of a parking permit termination sign, or implicitly based on a change in parking rules indicated by a feature that triggered the generation of the no-parking virtual parking sign 334 at the start of the next virtual parking lane 312. The virtual parking lane 312 itself is terminated by the detection of an intersection as represented by the virtual parking sign 332 (along with the applicability of the start of the no-parking rule of the virtual parking sign 334).

[0033] To the left of the Ego Machine's passage, virtual parking lane 324 is marked with a no-stop virtual parking sign 362 at its beginning boundary and a no-stop virtual parking sign 360 at its end boundary, indicating that stopping or parking is prohibited within virtual parking lane 324. Virtual parking lane 320 provides an example of a virtual parking lane 324 that can be defined (in both length and type) from parking rules generated from a single detected feature. For example, detection of a feature including a bus stop sign may trigger the virtual parking lane generator 210 to generate virtual parking lane 320 that extends 15.24 m (50 feet) after the detected feature, starts 15.24 m (50 feet) before the detected feature, and displays virtual parking signs 356 and 358 at the start and end of virtual parking lane 320 indicating that this parking lane is reserved for buses. Virtual parking lane 322 starts with a no-stop virtual parking sign 354 at its beginning boundary. In some embodiments, the no-stop virtual parking sign 354 may be generated based on the detection of actual features that indicate this parking rule, or based on keeping the no-stop rule in effect from the virtual parking area 324 before the bus stop. The virtual parking area 322 is terminated by the detection of an intersection as represented by the virtual parking sign 352 (along with the applicability of the start of the no-stop rule of the virtual parking sign 354).

[0034] It should be noted that in various embodiments, virtual parking lanes may be generated as described herein for routes carrying either one-way or two-way traffic. In some embodiments, one or more of the virtual parking lanes 310, 312, 314, 329, 322, or 324 may be generated from features facing the egomachine 305 as the egomachine 305 moves in the indicated direction of travel 307 and may be captured by front-view or surround sensors for detection. One or more of the virtual parking lanes 310, 312, 314, 329, 322, or 324 may also be generated from features facing the egomachine 305 and traffic traveling in the opposite direction to the route 302 and may be captured by rear-view or surround sensors for detection. For example, if the route 302 carries two-way traffic with right-hand traffic, one or more of the right-hand virtual parking lanes 310, 312, 314 may be generated by the parking lane processor 102 from features facing the ego machine 305 as the ego machine 305 moves forward and may be captured by the front view or surround sensors, and one or more of the left-hand virtual parking lanes 320, 322, 324 may be generated from features facing away from the ego machine 305 as the ego machine 305 moves forward and may be captured by the rear view or surround sensors. The parking lane processor 102 generates one or more left-side virtual parking lanes 320, 322, 324 from features facing the ego machine 305 as it moves, and one or more right-side virtual parking lanes 310, 312, 314 from features facing away from the ego machine 305 as it moves, similarly implementing the reverse when the route 302 carries bidirectional traffic with left-hand traffic. In further embodiments, the parking lane processor 102 may generate and display one or more virtual parking lanes for adjacent routes other than the specific route the ego machine is traveling on, using front view, rear view, and / or surround sensors. For example, in the case of a diverging highway or boulevard, the parking lane processor 102 may generate and display virtual parking lanes for adjacent routes based on captured features applied to the adjacent routes.Similarly, the parking lane processor 102 can generate and display virtual parking lanes for adjacent service roads in the same manner.

[0035] Figure 4 is a flowchart illustrating Method 400 for cognitive-based parking assistance according to several embodiments of this disclosure. Each block of Method 400 as described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 400 may also be performed as computer-available instructions stored in a computer storage medium. Method 400 may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 400 is described, as an example, with respect to a cognitive-based parking assistance system included in Process 100 of Figure 1. However, Method 400 may be performed by any one system or any combination of systems, including but not limited to those described herein, in addition or alternatively. Therefore, it should be understood that the features and elements described herein with respect to Method 400 in Figure 4 may be used in combination with, or in place of, any other embodiment described herein, or vice versa. Furthermore, it should be understood that the function, structure, and other descriptions of the elements in the embodiment shown in Figure 4 may apply to similar or similarly named or described elements in any of the figures and / or embodiments described herein, and vice versa.

[0036] Generally, Method 400 includes determining the location of a real-world parking lane and the associated parking rules for a parking lane for an ego machine using a virtual parking lane and one or more virtual parking signs generated at least in part based on one or more detected features in the environment of the ego machine.

[0037] Method 400 includes detecting, in block B402, one or more features indicating parking information relating to at least a portion of the planned route of the ego machine, based at least in part on sensor data generated using one or more sensors of the ego machine. In non-limiting examples, one or more detected features indicating parking information may include signs, intersections, surface colors along the route, specific objects, or symbols on surfaces along the route, or any other features that imply parking information, as discussed herein.

[0038] Method 400 includes, in block B404, calculating the geometry of a virtual parking space based at least in part on one or more features and the tracked motion of the ego machine. For example, the geometry of a virtual parking space may be calculated from a filtered set of features in parallel with the motion of the ego machine, as described herein. The tracked motion of the ego machine is used to extend the length of the virtual parking space along the path, starting from the location of the first feature, until one or more second features indicating the end of the parking space are recognized. The locations of one or more second features may be used to generate a second virtual parking sign and a virtual parking space end boundary on the path.

[0039] Method 400 includes associating a parking rule with a virtual parking space in block B406 based at least in part on one or more features. That is, a parking rule can be determined at least in part from information obtained from the detected features. A parking rule may provide, for example, the following parking-related information: parking is permitted in the virtual parking space, stopping is permitted in the virtual parking space, parking is not permitted in the virtual parking space, or stopping is not permitted in the virtual parking space. A parking rule may indicate the time, date, and / or other conditions under which the parking rule applies. If parking or stopping is permitted, a parking rule may also indicate that the permission is conditional on having a valid permit, e.g., a parking permit for persons with disabilities, or limited to residents, faculty and staff, or by use, e.g., for delivery purposes only.

[0040] Method 400 includes generating a parking assistance output in block B408 that indicates parking rules and the relative position of a virtual parking space to the ego machine. This parking assistance output may be used by one or more downstream components of the ego machine as input to perform one or more automated actions to assist the operator in the navigation of the ego machine or to control the ego machine through the environment.

[0041] The systems and methods described herein may, but are not limited to, be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotics platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying ships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. Furthermore, the systems and methods described herein may be used for a variety of purposes, for example and without limitation, for machine control, machine movement, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable applications.

[0042] The disclosed embodiments may include a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, cognitive systems for autonomous or semi-autonomous machines), systems implemented using robots, aerospace systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems implemented using edge devices, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems at least partially implemented in data centers, systems for performing conversational AI operations, systems for performing optical transport simulations, systems for performing collaborative content creation of 3D assets, systems at least partially implemented using cloud computing resources, and / or other types of systems.

[0043] Exemplary autonomous vehicle Figure 5A is an illustration of exemplary implementations of an Egomachine autonomous vehicle 500, including a cognitive-based parking assistance system 100, according to some embodiments of the present disclosure. The autonomous vehicle 500 (or, as referred to herein as "Vehicle 500") may include, but is not limited to, passenger vehicles, such as passenger cars, trucks, buses, first responder vehicles, shuttles, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, trailer-mounted vehicles, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). A mobile vehicle 500 may have the capability to function at one or more levels of autonomous driving from Level 3 to Level 5. A vehicle 500 may have the capability to function at one or more levels of autonomous driving from Level 1 to Level 5. For example, the vehicle 500 may have the capability of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment. In this specification, the term “autonomous” may include any and / or all types of autonomy of the vehicle 500 or any other machine, such as being fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing auxiliary autonomy, semi-autonomous, primarily autonomous, or other designations.

[0044] The mobile vehicle 500 may include components such as the chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. The mobile vehicle 500 may include a propulsion system 550, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 550 may be connected to the drivetrain of the mobile vehicle 500, which may include a transmission, in order to enable propulsion for the mobile vehicle 500. The propulsion system 550 may be controlled in response to receiving a signal from a throttle / accelerator 552.

[0045] A steering system 554, which may include a steering wheel, may be used to steer the vehicle 500 (for example, along a desired (planned) course or route) when the propulsion system 550 is operating (for example, when the vehicle is moving). The steering system 554 may receive signals from a steering actuator 556. The steering wheel may also be an option for fully automated (level 5) functionality.

[0046] The brake sensor system 546 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 548 and / or the brake sensor.

[0047] The controller 536, which may include one or more system-on-a-chip (SoC) 504 (Figure 5C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 500. For example, the controller can send signals to actuate the vehicle brakes via one or more brake actuators 548, actuate the steering system 554 via one or more steering actuators 556, and actuate the propulsion system 550 via one or more throttle / accelerators 552. The controller 536 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist the driver in driving the vehicle 500. The controller 536 may include a first controller 536 for autonomous driving functions, a second controller 536 for functional safety functions, a third controller 536 for artificial intelligence functions (e.g., computer vision), a fourth controller 536 for infotainment functions, a fifth controller 536 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 536 may handle two or more of the aforementioned functions, and two or more controllers 536 may handle a single function, and / or any combination thereof.

[0048] The controller 536 can provide signals for controlling one or more components and / or systems of the mobile vehicle 500 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, and without limitation, global navigation satellite system sensors 558 (e.g., global positioning system sensors), RADAR sensors 560, ultrasonic sensors 562, LIDAR sensors 564, inertial measurement unit (IMU) sensors 566 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 596, stereo cameras 568, wide-view cameras 570 (e.g., fisheye cameras), infrared cameras 572, surround cameras 574 (e.g., 360-degree cameras), long-range and / or medium-range cameras 598, speed sensors 544 (e.g., for measuring the speed of a moving vehicle 500), vibration sensors 542, steering sensors 540, brake sensors (e.g., as part of a brake sensor system 546), and / or other sensor types. Furthermore, the controller 536 may receive parking assistance output from the parking lane processor 102, which indicates parking rules and the relative position of the virtual parking lane to the ego machine. With the parking assistance output, the controller 536 can operate the vehicle 500 to navigate to a valid parking or stopping lane, while avoiding the parking lane when parking and / or stopping is not permitted.

[0049] One or more of the controllers 536 may receive inputs (represented, for example, by input data) from the instrument cluster 532 of the mobile vehicle 500 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 534, an audible annunciator, a loudspeaker, and / or other components of the mobile vehicle 500. The outputs may include information such as mobile vehicle velocity, speed, time, map data (e.g., HD map 522 in Figure 5C), location data (e.g., the location of the mobile vehicle 500, such as on the map), direction, the locations of other mobile vehicles (e.g., occupied grids), and information about objects and the status of objects as grasped by the controller 536. For example, the HMI display 534 can display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the moving vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting exit 34B within 3.22 km (2 miles), etc.). Furthermore, the HMI display 534 can display virtual parking lanes generated by the parking lane processor 102, as illustrated and described above with respect to Figure 3. The virtual parking lane generator 210 presents on the HMI display 534 graphic representations or icons of one or more virtual parking lanes as detected in the proximity of the vehicle 500 to the ego machine 305 traveling along the path 302 having an indicated direction of travel 307 and the corresponding virtual parking signs.

[0050] The mobile vehicle 500 further includes a network interface 524 that can communicate over one or more networks using one or more wireless antennas 526 and / or a modem. For example, the network interface 524 may have the capability to communicate over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antennas 526 can also use local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWAN) such as LoRaWAN, SigFox to enable communication between objects in the environment (e.g., mobile vehicles, mobile devices, etc.). In some embodiments, the parking zone processor 102 periodically or based on the location of the vehicle 500 downloads parking rule data 216 via the network interface 524, for example, to update its knowledge of local parking rules.

[0051] Figure 5B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 500 of Figure 5A according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different positions on the mobile vehicle 500. Any one or more of such cameras, as shown in Figure 5B, may be used as a sensor 110 for capturing sensor data 108 which is ultimately input to the parking lane processor 102 as feature data 104.

[0052] The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the mobile vehicle 500. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0053] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0054] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0055] A camera having a field of view that includes a portion of the environment in front of the moving vehicle 500 (e.g., a forward-facing camera) may be used for surround view to help identify the forward path and obstacles and, with the help of one or more controllers 536 and / or control SoCs, to help provide information essential for generating an occupied grid and / or determining a preferred moving vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as lane departure warning (LDW), autonomous cruise control (ACC), and / or traffic sign recognition.

[0056] Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 570, which can be used to capture objects entering the view from the periphery (e.g., pedestrians, intersecting traffic, or bicycles). Although only one wide-view camera is shown in Figure 5B, any number of wide-view cameras 570 may be present in the mobile vehicle 500. In addition, long-range cameras 598 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 598 may also be used for object detection and classification, as well as basic object tracking.

[0057] One or more stereo cameras 568 may also be included in the forward-facing configuration. The stereo camera 568 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the moving vehicle's environment, including distance estimates of all points in the image. An alternative stereo camera 568 may include a compact stereo vision sensor that includes two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the moving vehicle to an object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 568 may be used in addition to or instead of those described herein.

[0058] A camera having a field of view including a portion of the environment on the sides of the mobile vehicle 500 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 574 (e.g., four surround cameras 574 as shown in Figure 5B) may be positioned on the mobile vehicle 500. The surround cameras 574 may include wide-view cameras 570, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and on the sides of the mobile vehicle. In an alternative configuration, the mobile vehicle may use three surround cameras 574 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera.

[0059] A camera having a field of view that includes a portion of the environment behind the moving vehicle 500 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 598, stereo cameras 568), infrared cameras 572, etc., as described herein.

[0060] Figure 5C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 500 of Figure 5A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0061] Each component, feature, and system of the mobile vehicle 500 in Figure 5C is illustrated as being connected via a bus 502. Bus 502 may include a Controller Area Network (CAN) data interface (or referred to as the "CAN bus"). CAN may also be a network within the mobile vehicle 500 used to help control various features and functions of the mobile vehicle 500, such as the operation of brakes, acceleration, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other mobile vehicle status indicators. The CAN bus may be ASIL B compliant.

[0062] Bus 502 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or as an alternative to, a CAN bus. In addition, a single line is used to represent bus 502, but this is not intended to limit it. There may be any number of buses 502, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 502 may be used to perform different functions and / or for redundancy. For example, a first bus 502 may be used for collision avoidance and a second bus 502 may be used for operation control. In any example, each bus 502 may communicate with any of the components of the mobile vehicle 500, and two or more buses 502 may communicate with the same component. In some examples, each SoC 504, each controller 536, and / or each computer within the mobile vehicle may have access to the same input data (e.g., input from sensors in the mobile vehicle 500) and may be connected to a common bus such as a CAN bus.

[0063] The mobile vehicle 500 may include one or more controllers 536, such as those described herein with respect to Figure 5A. The controllers 536 may be used for a variety of functions. The controllers 536 may be coupled to any of the various other components and systems of the mobile vehicle 500 and may be used for the control of the mobile vehicle 500, the artificial intelligence of the mobile vehicle 500, infotainment for the mobile vehicle 500, and / or similar. For example, the features and functions of one or both of the parking area processor 102 or the feature classifier 106 may be performed at least in part by one or more controllers 536.

[0064] The mobile vehicle 500 may include a system-on-a-chip (SoC) 504. The SoC 504 may include a CPU 506, a GPU 508, a processor 510, a cache 512, an accelerator 514, a data store 516, and / or other components and features not shown. The SoC 504 may be used to control the mobile vehicle 500 in various platforms and systems. For example, the SoC 504 may be coupled in a system (e.g., the system of the mobile vehicle 500) that has an HD map 522 that can obtain map refreshes and / or updates via a network interface 524 from one or more servers (e.g., server 578 in Figure 5D).

[0065] The CPU 506 may include a CPU cluster or CPU complex (also referred to as "CCPLEX"). The CPU 506 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 506 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 506 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 506 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the CPU 506 clusters to be active at any given time.

[0066] The CPU506 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU506 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode.

[0067] The GPU508 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU508 may be programmable and efficient for parallel workloads. In some embodiments, the GPU508 may be able to use an enhanced tensor instruction set. The GPU508 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, the GPU508 may include at least eight streaming microprocessors. The GPU508 may be able to use a Computation Application Programming Interface (API). In addition, the GPU508 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0068] The GPU508 can be power-optimized for optimal performance in automotive and embedded use cases. For example, the GPU508 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be a limitation, and the GPU508 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance.

[0069] In some examples, the GPU508 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0070] The GPU508 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support may be used to allow the GPU508 to directly access the CPU506 page table. In such examples, when the GPU508 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU506. In response, the CPU506 can examine its page table for virtual-to-real-address mapping and send the translation back to the GPU508. As such, unified memory technology can enable a single, unified virtual address space for both the CPU506 and GPU508 memory, thereby simplifying GPU508 programming and porting of applications to the GPU508.

[0071] In addition, the GPU508 may include access counters that can record how often the GPU508 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0072] The SoC504 may include any number of caches 512, including those described herein. For example, cache 512 may include an L3 cache available to both the CPU 506 and the GPU 508 (e.g., connected to both the CPU 506 and the GPU 508). Cache 512 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, although a smaller cache size may be used.

[0073] The SoC504 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 500 (for example, a processing DNN). In addition, the SoC504 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU506 and / or GPU508.

[0074] The SoC 504 may include one or more accelerators 514 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC 504 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. For example, one or more embodiments of a feature classifier 106 may utilize the accelerator 514 to generate feature data 104 that are input to the parking processor 102. The hardware acceleration cluster may be used to complement the GPU 508 and to offload some of the tasks of the GPU 508 (e.g., to free up more cycles of the GPU 508 to perform other tasks). As an example, the accelerator 514 may be used for a target workload that is sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural network (CNN), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0075] The accelerator 514 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA can provide more performance per millisecond than a general-purpose GPU and significantly exceed the performance of a CPU. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0076] DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and mobile vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0077] DLA can perform any function of GPU508, and by using inference accelerators, for example, a designer can target either DLA or GPU508 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU508 and / or other accelerators 514.

[0078] The accelerator 514 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may also be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may include, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0079] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may contain any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0080] DMA can enable PVA components to access system memory independent of the CPU506. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0081] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0082] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can run the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or even run different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0083] The accelerator 514 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 514. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, and not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB).

[0084] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0085] In some embodiments, the SoC504 may include a real-time ray tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0086] The accelerator 514 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA may also be a programmable vision accelerator that can be used in critical processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are suitable for areas of algorithms that require predictable processing at low power and low latency. In other words, the PVA works well in semi-high density or high density normal computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, since the PVA is efficient in operation in object detection and integer computation, in relation to a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms.

[0087] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. While semi-global matching-based algorithms may be used in some examples, this is not intended to be a limitation. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0088] In some applications, PVA can be used to perform high-density optical flow by processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR data. In other applications, PVA is used for flight depth processing, for example, by processing raw flight data to provide processed flight data.

[0089] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing a relative "weight" for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positive detections. In an embodiment, such confidence values ​​can be used by the parking lane processor 102 in determining the degree to which parking rules derived from recognized feature data 104 are reliable enough to be associated with virtual parking lanes. Parking rules with high confidence values ​​(e.g., at least meeting the confidence threshold) can be assigned to virtual parking lanes. Parking rules with low confidence values ​​(e.g., not meeting the confidence threshold) may be ignored, and a default parking rule is assigned to the virtual parking lane.

[0090] In an automatic emergency braking (AEB) system, a false positive detection would cause the moving vehicle to automatically apply the emergency brakes, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for the AEB. A DLA can run a neural network that regresses confidence values. The neural network can take as its input at least a subset of parameters, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), vehicle orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation of an object acquired from the neural network and / or other sensors (e.g., LIDAR sensor 564 or RADAR sensor 560), and others.

[0091] The SoC504 may include a data store 516 (for example, memory). The data store 516 may also be the on-chip memory of the SoC504 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 516 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 512 may include an L2 or L3 cache 512. References to the data store 516 may include references to memory associated with the PVA, DLA, and / or other accelerators 514, as described herein.

[0092] The SoC504 may include one or more processors 510 (e.g., integrated processors). The processors 510 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC504 boot sequence and can provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC504 thermal and temperature sensors, and / or management of SoC504 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC504 may use the ring oscillators to detect the temperatures of the CPU 506, GPU 508, and / or accelerator 514. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine, placing the SoC504 into a lower power state and / or putting the mobile vehicle 500 into chauffeur safe shutdown mode (for example, safely shutting down the mobile vehicle 500).

[0093] The processor 510 may further include a set of integrated processors that can perform the functions of an audio processing engine. The audio processing engine may also be an audio subsystem that enables full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0094] The processor 510 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0095] The processor 510 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations.

[0096] The processor 510 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0097] Processor 510 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0098] The processor 510 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce a final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 570, the surround camera 574, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and appropriately respond to in-cabin events. The in-cabin system can perform lip-reading to activate cellular services and make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0099] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0100] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU508 is not required to continuously render new surfaces. Even when the GPU508 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU508 to improve performance and responsiveness.

[0101] The SoC504 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC504 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0102] The SoC504 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC504 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., Lidar sensor 564, Radar sensor 560, etc., which may be connected via Ethernet®), data from bus 502 (e.g., speed of vehicle 500, steering wheel position, etc.), and data from GNSS sensor 558 (connected, for example, via Ethernet® or CAN bus). The SoC504 may further include its own DMA engine and a dedicated high-performance mass storage controller which may be used to free up CPU 506 from routine data management tasks.

[0103] The SoC504 may also be an inter-terminal platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and, together with deep learning tools, provides a platform for a flexible, reliable driving software stack. The SoC504 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 514 is coupled with the CPU 506, the GPU 508, and the data store 516 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0104] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0105] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU520) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs and passing that semantic understanding to a route planning module running on the CPU complex.

[0106] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network, informing the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU508, for example.

[0107] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of the vehicle 500. An always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and, in security mode, to stop the vehicle when the owner leaves it. In this way, the SoC504 provides security against theft and / or vehicle hijacking.

[0108] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 596. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 504 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In a preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensor 558. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, a control program may be used, with the assistance of ultrasonic sensor 562, to perform emergency vehicle safety routines such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle until the emergency vehicle has passed.

[0109] The vehicle may include a CPU 518 (e.g., a separate CPU, or dCPU) which can be connected to the SoC 504 via a high-speed interconnect (e.g., PCIe). The CPU 518 may include, for example, an x86 processor. The CPU 518 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 504, and / or monitoring the status and condition of the controller 536 and / or the infotainment SoC 530.

[0110] The mobile vehicle 500 may include a GPU 520 (e.g., a separate GPU, or dGPU) which can be connected to the SoC 504 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 520 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from the mobile vehicle 500's sensors (e.g., sensor data).

[0111] The mobile vehicle 500 may further include a network interface 524 which may include one or more wireless antennas 526 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth® antennas). The network interface 524 may be used to enable wireless connectivity to a cloud over the Internet (e.g., with a server 578 and / or other network devices), to other mobile vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other mobile vehicles, a direct link may be established between two mobile vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a mobile vehicle-to-mobile communication link. The mobile vehicle-to-mobile communication link can provide mobile vehicle 500 information about mobile vehicles in close proximity to mobile vehicle 500 (e.g., mobile vehicles in front of, beside, and / or behind mobile vehicle 500). This functionality may also be part of the mobile vehicle 500's joint adaptive cruise control functionality.

[0112] The network interface 524 may include an SoC that provides modulation and demodulation functions and enables the controller 536 to communicate over a wireless network. The network interface 524 may include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. Frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth®, Bluetooth® LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0113] The mobile unit 500 may further include a data store 528 which may include storage outside the chip (for example, outside the SoC 504). The data store 528 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0114] The vehicle 500 may further include a GNSS sensor 558. The GNSS sensor 558 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) assists in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 558 may be used, including, but not limited to, a GPS using a USB connector with Ethernet® to a serial (RS-232) bridge. In the embodiment, the tracked motion data 112 may be generated at least partially from the output of the GNSS sensor 558.

[0115] The mobile vehicle 500 may further include a RADAR sensor 560. The RADAR sensor 560 may be used by the mobile vehicle 500 for long-range mobile vehicle detection, even in darkness and / or harsh weather conditions. In some embodiments, the RADAR sensor 560 may serve as a sensor 110 for capturing sensor data 108 from which feature data 104 is derived. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 560 may use CAN and / or bus 502 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 560) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 560 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0116] The RADAR sensor 560 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 560 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include monostatic multimodal RADARs having multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record around the moving vehicle 500 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of moving vehicles entering or leaving the lane of the moving vehicle 500.

[0117] As an example, a medium-range RADAR system may include a range of up to 560m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 550 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the moving vehicle.

[0118] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0119] The mobile vehicle 500 may further include ultrasonic sensors 562. Positioned on the front, rear, and / or sides of the mobile vehicle 500, the ultrasonic sensors 562 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 562 may be used, and different ultrasonic sensors 562 may be used for detection at different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 562 may operate at a functional safety level of ASIL B.

[0120] The mobile vehicle 500 may include a LiDAR sensor 564. The LiDAR sensor 564 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 564 may also have a functional safety level of ASIL B. In some examples, the mobile vehicle 500 may include multiple LiDAR sensors 564 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0121] In some examples, the LIDAR sensor 564 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 564 may have an advertised range of approximately 500m, for example, with an accuracy of 2cm to 3cm and support for a 500Mbps Ethernet® connection. In some examples, one or more non-protruding LIDAR sensors 564 may be used. In such examples, the LIDAR sensor 564 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a mobile vehicle 500. In such examples, the LIDAR sensor 564 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 564 may be configured for a horizontal field of view between 45 and 135 degrees.

[0122] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around a moving vehicle up to approximately 200m. The flash LiDAR unit includes a receptor that records the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the moving vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the moving vehicle. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 564 may be less susceptible to motion blur, vibration, and / or shock.

[0123] The mobile vehicle may further include an IMU sensor 566. In some examples, the IMU sensor 566 may be positioned in the center of the rear axle of the mobile vehicle 500. The IMU sensor 566 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 566 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 566 may include an accelerometer, a gyroscope, and a magnetometer. In the embodiment, the tracked motion data 112 may be generated at least partially from the output of the IMU sensor 566.

[0124] In some embodiments, the IMU sensor 566 may be implemented as a miniature, high-performance GPS-aided inertial navigation system (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 566 may enable the moving vehicle 500 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating changes in velocity from the GPS to the IMU sensor 566. In some embodiments, the IMU sensor 566 and the GNSS sensor 558 may be combined in a single integrated unit.

[0125] The mobile vehicle may include a microphone 596 placed inside and / or around the mobile vehicle 500. The microphone 596 may, among other things, be used for emergency vehicle detection and identification.

[0126] The mobile vehicle may further include any number of camera types, including a stereo camera 568, a wide-view camera 570, an infrared camera 572, a surround camera 574, a long-range and / or medium-range camera 598, and / or other camera types. The cameras may be used to capture image data around the entire exterior surface of the mobile vehicle 500. The type of camera used will depend on the embodiment and requirements of the mobile vehicle 500, and any combination of camera types may be used to achieve the required coverage around the mobile vehicle 500. In addition, the number of cameras may vary depending on the embodiment. For example, the mobile vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 5A and 5B.

[0127] The mobile vehicle 500 may further include a vibration sensor 542. The vibration sensor 542 can measure vibrations of components of the mobile vehicle, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 542 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0128] The mobile vehicle 500 may include an ADAS system 538. In some examples, the ADAS system 538 may include a System of Control (SoC). The ADAS system 538 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0129] The ACC system may utilize a radar sensor 560, a lithium-ion sensor 564, and / or a camera. The ACC system may include longitudinal ACC and / or transverse ACC. Longitudinal ACC monitors and controls the distance of vehicle 500 to the vehicle immediately in front of it and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Transverse ACC performs distance maintenance and advises vehicle 500 to change lanes when necessary. Transverse ACC is related to other ADAS applications such as LCA and CWS.

[0130] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through a network interface 524 and / or a wireless antenna 526, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., a vehicle in the same lane as vehicle 500, immediately in front of vehicle 500), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about vehicles ahead of vehicle 500, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0131] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 560, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual, vibration, and / or quick brake pulses.

[0132] An AEB system can detect an imminent forward collision with another moving vehicle or other object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or impending collision braking.

[0133] The LDW system warns the driver when the vehicle 500 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0134] The LKA system is a modified version of the LDW system. The LKA system provides steering input or braking to correct the vehicle 500 if it begins to drift out of its lane.

[0135] The BSW system detects and warns the driver of a moving vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or radar sensor 560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0136] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle 500 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rearward-facing RADAR sensors 560, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0137] Conventional ADAS systems warn the driver and allow the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have sometimes tended to produce misjudgments that, while not usually catastrophic, can be troubling and distracting to the driver. In the autonomous vehicle 500, however, if the results are contradictory, the vehicle 500 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 536 or the second controller 536). For example, in some embodiments, the ADAS system 538 may also be a backup and / or secondary computer to provide perceptual information to a backup computer rationality module. The backup computer rationality monitor can run a variety of redundant software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 538 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0138] In some implementations, a primary computer may be configured to provide a supervising MCU with a reliability score indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0139] The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer provides a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based FCW system, the neural network in the supervisory MCU can learn when the FCW identifies metal objects that are not actually dangerous, such as sewer grates or manhole covers that trigger an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments involving a neural network running on the supervisory MCU, the supervisory MCU may include at least one DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC504.

[0140] In other examples, ADAS system 538 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error.

[0141] In some examples, the output of the ADAS system 538 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 538 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0142] The mobile vehicle 500 may further include an infotainment SoC 530 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more separate components. The infotainment SoC 530 may include a combination of hardware and software that can be used to provide the mobile vehicle 500 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total distance traveled, brake fuel level, oil level, door open / close, air filter information, and other mobile vehicle-related information). For example, the infotainment SoC 530 may also be a wireless, disc player, navigation system, video player, USB and Bluetooth® connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 534, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 530 may be further used to provide information (for example, visual and / or audible) to the user of the vehicle, such as information from the ADAS system 538, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0143] The infotainment SoC 530 may include GPU functionality. The infotainment SoC 530 can communicate with other devices, systems, and / or components of the vehicle 500 via bus 502 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 530 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 536 (e.g., the vehicle 500's primary and / or backup computer) failure. In such examples, the infotainment SoC 530 can put the vehicle 500 into a chauffeur-safe stop mode as described herein.

[0144] The mobile vehicle 500 may further include an instrument cluster 532 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 532 may include a controller and / or a supercomputer (e.g., a separate controller or supercomputer). The instrument cluster 532 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 530 and the instrument cluster 532. In other words, the instrument cluster 532 may be included as part of the infotainment SoC 530, and vice versa.

[0145] Figure 5D is a system diagram of communication between the cloud-based server of Figure 5A and an exemplary autonomous vehicle 500, according to some embodiments of the present disclosure. System 576 may include a server 578, a network 590, and a mobile vehicle including the mobile vehicle 500. Server 578 may include a plurality of GPUs 584(A) to 584(H) (collectively referred to herein as GPU 584), PCIe switches 582(A) to 582(H) (collectively referred to herein as PCIe switch 582), and / or CPUs 580(A) to 580(B) (collectively referred to herein as CPU 580). The GPUs 584, CPUs 580, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 588 and / or PCIe connections 586 developed by NVIDIA. In some examples, the GPU584 is connected via NVLink and / or NVSwitch SoCs, and the GPU584 and PCIe switch 582 are connected via PCIe interconnects. Eight GPU584s, two CPU580s, and two PCIe switches are illustrated, but this is not intended to be an limitation. Depending on the embodiment, each server 578 may contain any number of GPU584s, CPU580s, and / or PCIe switches. For example, each server 578 may contain eight, sixteen, thirty-two, and / or more GPU584s.

[0146] Server 578 can receive image data from mobile vehicles via network 590, representing images showing unexpected or altered road conditions, such as recently commenced road construction. Server 578 can transmit map information 594, including information about traffic and road conditions, to mobile vehicles via network 590, including information about the neural network 592, updated neural network 592, and / or map information 594. Updates to map information 594 may include updates to HD map 522, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some instances, the neural network 592, updated neural network 592, and / or map information 594 may have arisen from new training and / or experience represented in data received from any number of mobile vehicles in the environment, and / or based on training performed in a data center (for example, using server 578 and / or other servers).

[0147] Server 578 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a vehicle and / or synthetically generated in a simulation (e.g., using a game engine) and / or as multidimensional (e.g., 2D or 3D) assets for a collaborative content creation platform for heterogeneous content creation applications. In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, the machine learning model may be used by the mobile vehicle (e.g., transmitted to the mobile vehicle via network 590), and / or the machine learning model may be used by server 578 to remotely monitor the mobile vehicle.

[0148] In some examples, Server 578 can receive data from a mobile vehicle and apply it to a state-of-the-art real-time neural network for real-time intelligent inference. Server 578 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 584, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 578 may include deep learning infrastructure that uses only CPU-powered data centers.

[0149] The deep learning infrastructure of server 578 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware in mobile vehicle 500. For example, the deep learning infrastructure can receive periodic updates from mobile vehicle 500, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by mobile vehicle 500, and if the results do not match and the infrastructure concludes that the AI ​​in mobile vehicle 500 is not functioning properly, server 578 can send a signal to mobile vehicle 500 instructing the failsafe computer in mobile vehicle 500 to infer control, notify passengers, and complete a safe parking operation.

[0150] For inference, server 578 may include GPU 584 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when high performance is not required, a server powered by a CPU, FPGA, and other processors may be used for inference.

[0151] Exemplary computing devices Figure 6 is a block diagram of an exemplary computing device 600 suitable for use in implementations of some embodiments of the present disclosure, including, but not limited to, a parking area processor 102 and / or a feature classifier 106. The computing device 600 may include an interconnection system 602 that indirectly or directly connects the following devices: memory 604, one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, a communication interface 610, input / output (I / O) ports 612, input / output components 614, a power supply unit 616, one or more presentation components 618 (e.g., a display), and one or more logic units 620. In at least one embodiment, the computing device 600 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 608 may include one or more vGPUs, one or more of the CPUs 606 may include one or more vCPUs, and / or one or more of the logical units 620 may include one or more virtual logical units. As such, the computing device 600 may include individual components (e.g., an entire GPU dedicated to the computing device 600), virtual components (e.g., a portion of a GPU dedicated to the computing device 600), or a combination thereof.

[0152] The various blocks in Figure 6 are shown connected by lines via the interconnection system 602, but this is not intended to be restrictive and is simply for clarity. For example, in some embodiments, a presentation component 618, such as a display device, could be considered an I / O component 614 (for example, if the display is a touchscreen). As another example, the CPU 606 and / or GPU 608 may include memory (for example, memory 604 may represent a storage device in addition to the memory of the GPU 608, CPU 606, and / or other components). In other words, the computing devices in Figure 6 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 6 and are therefore not distinguished.

[0153] The interconnection system 602 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 602 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 606 may be directly connected to the memory 604. Furthermore, the CPU 606 may be directly connected to the GPU 608. Where direct or point-to-point connections exist between components, the interconnection system 602 may include PCIe links to implement the connections. In these examples, the PCI bus does not need to be included in the computing device 600.

[0154] Memory 604 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 600. The computer-readable media may include both volatile and non-volatile media, and removable and non-removable media. For example, but not limited to, the computer-readable media may include computer storage media and communication media.

[0155] Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 604 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 600. In this specification, computer storage media does not include signals themselves.

[0156] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. For example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0157] The CPU 606 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 600 to execute one or more of the methods and / or processes described herein. For example, one or more of the parking area processor 102 and / or feature classifier 106 may be implemented as code executed by the CPU 606. The CPU 606 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads concurrently. The CPU 606 may include any type of processor, and depending on the type of computing device 600 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 600, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 600 may include one or more CPUs 606 within one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0158] In addition to or instead of the CPU 606, the GPU 608 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 600 to execute one or more of the methods and / or processes described herein. One or more of the GPU608 may be an integrated GPU (for example, one or more of the CPU606), and / or one or more of the GPU608 may be a discrete GPU. In some embodiments, one or more of the GPU608 may be a coprocessor of one or more of the CPU606. The GPU608 may be used by the computing device 600 to render graphics (for example, 3D graphics), a visual display of parking assistance output from the parking lane processor 102 to the HMI display 120, or to perform general-purpose computing. The GPU608 may be used by the computing device 600 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPU608 may be used for GPU-based general-purpose computing (GPGPU). It can be used for a GPU. The GPU608 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU608 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU606 received via the host interface). The GPU608 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 604. The GPU608 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or to the GPUs via a switch (for example, using NVSwitch).When combined, each GPU 608 can generate pixel data or GPGPU data for different parts of the output or for different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs. In some embodiments, the GPU 608 may render elements of virtual parking spaces and virtual parking signs to the HMI display 120 based on images from the virtual parking space graphics library 222.

[0159] In addition to or instead of the CPU 606 and / or GPU 608, the logic unit 620 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 600 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 606, GPU 608, and / or logic unit 620 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logic units 620 may be part of and / or integrated with one or more of the CPU 606 and / or GPU 608, and / or one or more of the logic units 620 may be discrete components of the CPU 606 and / or GPU 608 or otherwise external to them. In embodiments, one or more of the logic units 620 may be coprocessors of one or more of the CPU 606 and / or one or more of the GPU 608.

[0160] Examples of logical unit 620 include one or more processing cores and / or components thereof, such as a Data Processing Unit (DPU), Tensor Core (TC), Tensor Processing Unit (TPU), Pixel Visual Core (PVC), Vision Processing Unit (VPU), Graphics Processing Cluster (GPC), Texture Processing Cluster (TPC), Streaming Multiprocessor (SM), Tree Traversal Unit (TTU), Artificial Intelligence Accelerator (AIA), and Deep Learning Accelerator (DLA). This includes an Accelerator, a Logical Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating-Point Unit (FPU), input / output (I / O) elements, a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) element, and / or similar.

[0161] The communication interface 610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 600 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 610 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 620 and / or the communication interface 610 may include one or more data processing units (DPUs) to transmit data received via the network and / or through the interconnection system 602 directly to one or more GPUs 608 (e.g., their memory).

[0162] I / O port 612 can enable the computing device 600 to be logically connected to other devices, including I / O components 614, presentation components 618, and / or other components, some of which can be built into (e.g., integrated into) the computing device 600. Exemplary I / O components 614 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 614 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 600 (as described in more detail below). The computing device 600 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 600 may include accelerometers or gyroscopes that enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 600 to render immersive augmented reality or virtual reality.

[0163] The power supply device 616 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 616 can provide power to the computing device 600 to enable the components of the computing device 600 to operate.

[0164] The presentation component 618 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 618 can receive data from other components (e.g., GPU 608, CPU 606, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0165] Exemplary data center Figure 7 shows an exemplary data center 700 that may be used in at least one embodiment of the present disclosure. The data center 700 may include a data center infrastructure layer 710, a framework layer 720, a software layer 730, and / or an application layer 740. The data center 700 may provide one or more services to a cognitive-based parking assistance system 100 (which can access the data center 700 via a wireless network interface 118). For example, the cognitive-based parking assistance system 100 may query the data center 700 to receive local parking rule information based on the location of the ego machine, and / or updates to other information used to interpret and / or display parking information from detected features in firmware, software, or feature data 104, or other capability upgrades.

[0166] As shown in Figure 7, the data center infrastructure layer 710 may include a resource orchestrator 712, grouped computing resources 714, and node computing resources ("node CRs") 716(1) to 716(N), where "N" represents any integer or natural number. In at least one embodiment, the node CRs 716(1) to 716(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules. In some embodiments, one or more nodes CR716(1) to 716(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR716(1) to 7161(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more nodes CR716(1) to 716(N) may correspond to a virtual machine (VM).

[0167] In at least one embodiment, the grouped computing resources 714 may include a separate group of nodes CR716 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CR716 within the grouped computing resources 714 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR716, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0168] The resource orchestrator 712 can configure or otherwise control one or more nodes CR716(1) to 716(N) and / or grouped computing resources 714. In at least one embodiment, the resource orchestrator 712 may include a software design infrastructure ("SDI") management entity for the data center 700. The resource orchestrator 712 may include hardware, software, or any combination thereof.

[0169] In at least one embodiment, as shown in Figure 7, the framework layer 720 may include a job scheduler 733, a configuration manager 734, a resource manager 736, and / or a distributed file system 738. The framework layer 720 may include a framework to support software 732 of the software layer 730 and / or one or more applications 742 of the application layer 740. The software 732 or application 742 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. The framework layer 720 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 738 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 733 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 700. The configuration manager 734 may have the ability to configure different layers, for example, a software layer 730 and a framework layer 720 that includes Spark and a distributed file system 738 to support large-scale data processing. The resource manager 736 may have the ability to manage clustered or grouped computing resources that are mapped or allocated for support of the distributed file system 738 and the job scheduler 733. In at least one embodiment, the clustered or grouped computing resources may include computing resources 714 grouped in the data center infrastructure layer 710. The resource manager 736 can coordinate with the resource orchestrator 712 to manage these mapped or allocated computing resources.

[0170] In at least one embodiment, the software 732 included in the software layer 730 may include software used by at least a portion of nodes CR716(1) to 716(N), grouped computing resources 714, and / or the distributed file system 738 of the framework layer 720. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0171] In at least one embodiment, the application 742 included in the application layer 740 may include one or more types of applications used by at least a portion of the nodes CR716(1) to 716(N), the grouped computing resources 714, and / or the distributed file system 738 of the framework layer 720. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0172] In at least one embodiment, any of the configuration manager 734, resource manager 736, and resource orchestrator 712 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 700 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center.

[0173] Data Center 700 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to Data Center 700. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to Data Center 700 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0174] In at least one embodiment, the data center 700 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) for performing training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0175] Exemplary network environment A network environment suitable for use in implementing the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 600 in Figure 6, for example, each device may include similar components, features, and / or functionalities of the computing device 600. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 700, examples of which are further detailed herein with respect to Figure 7.

[0176] Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may include multiple networks, or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity.

[0177] Compatible network environments may include one or more peer-to-peer network environments (in which case servers may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices.

[0178] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In the embodiment, one or more of the client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”).

[0179] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may delegate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0180] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 600 described herein with respect to Figure 6. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0181] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0182] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0183] The subject matter of this disclosure is described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the claimed subject matter may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, expressed.

Claims

1. Based at least partially on sensor data generated using one or more sensors of the ego machine, one or more features indicating parking information related to at least a portion of the ego machine's planned route are detected. The geometry of the virtual parking area is calculated based at least partially on one or more of the aforementioned features and the tracked movement of the ego machine. Associating parking rules with the virtual parking area based at least in part on one or more of the above features, Generates a parking assistance output that shows the parking rules and the relative position of the virtual parking area to the ego machine. A processor comprising one or more circuits, The calculation of the geometry of the virtual parking area is To generate the starting boundary line of the virtual parking area based at least partially on the first position of the first feature among the one or more features indicating the starting point of the virtual parking area, The length of the virtual parking area is extended along the planned path based on the tracked movement of the ego machine until the second position of the second of the one or more features indicates the endpoint of the virtual parking area, and To generate the end boundary line of the virtual parking area based at least partially on the second location. A processor that includes this.

2. The aforementioned parking rules Parking is permitted within the aforementioned virtual parking area. Stopping is permitted within the aforementioned virtual parking area. Parking is not permitted within the aforementioned virtual parking area. Stopping is not permitted within the aforementioned virtual parking area, or Parking is permitted within the aforementioned virtual parking area, subject to the requirement of a permit. The processor according to claim 1, wherein at least one of the following is represented.

3. The processor according to claim 1, wherein one or more of the features include a sign, an intersection, a surface color along the planned route, or a symbol on the surface along the planned route.

4. The detection of one or more of the above features is Removing at least one of the aforementioned one or more features by filtering to generate a filtered set of features. The filtering process includes, The at least one of the aforementioned features lies outside the orbital manifold of the egomachine. The fact that at least one of the aforementioned features is located on a different path from the ego machine, or The distance between the ego machine and the at least one feature is greater than the threshold distance. Based at least in part on at least one of the following, The calculation of the geometry of the virtual parking area is performed using a filtered set of features. The processor according to claim 1.

5. The calculation of the geometry of the virtual parking area is further performed The geometry of the virtual parking area is calculated at least partially based on the relative positions of the first and second positions with respect to the ego machine. The processor according to claim 1, including the following:

6. The aforementioned one or more circuits may further A first virtual marker corresponding to the starting boundary line of the virtual parking area is generated. A second virtual marker corresponding to the end boundary line of the virtual parking area is generated. The semantic information corresponding to the first position and the first virtual sign, the second position, and the semantic information corresponding to the second virtual sign are stored in the memory of the ego machine. The processor according to claim 1, including an electrical circuit.

7. The processor according to claim 1, wherein the generation of the parking assistance output includes causing a display on the ego machine's display of a first graphic for a first virtual sign corresponding to the start boundary of the virtual parking lane and a second graphic for a second virtual sign corresponding to the end boundary of the virtual parking lane.

8. The aforementioned one or more circuits may further Generate a display that includes at least one of the following: a first virtual parking space through which the ego machine has passed, a second virtual parking space adjacent to which the ego machine is moving, or a third virtual parking space into which the ego machine is about to enter. The processor according to claim 1, including an electrical circuit.

9. The aforementioned one or more circuits may further The reliability level of the parking rule associated with the virtual parking area is determined based at least partially on the aforementioned parking information. When the reliability level is below a predetermined threshold, the default parking rule is selected. The processor according to claim 1, including an electrical circuit.

10. The processor according to claim 1, wherein the parking rules are at least partially based on parking rules relating to the geographical location of the ego machine.

11. The processor according to claim 1, wherein the detection of one or more features includes determining that the parking information is related to at least a portion of the planned route, at least on the basis of estimating the distance between the one or more features and the planned route.

12. The aforementioned one or more circuits may further The history of previously calculated virtual parking spaces along the planned route is stored in the memory of the ego machine. The processor according to claim 1, including an electrical circuit.

13. The aforementioned planned route, Paved road, off road, highway, Private road, Part of the parking lot, path, line, rivers, sidewalk, The demarcated portion of the environment, or air route The processor according to claim 1, comprising at least one of the following.

14. The aforementioned processor, Control systems for autonomous or semi-autonomous machines, Cognitive systems for autonomous or semi-autonomous machines A system that performs simulation operations. A system that generates synthetic data using machine learning. A system that generates multidimensional assets using a collaborative content creation platform. A system that performs deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The processor according to claim 1, which is included in at least one of the following.

15. One or more sensors that generate sensor data, One or more processing units including processing electrical circuits A system including the processing electrical circuit, One or more features indicating parking information related to at least a portion of the planned path of the ego machine are detected, at least partially based on the sensor data. The geometry of the virtual parking area is calculated based at least partially on one or more of the aforementioned features and the tracked movement of the ego machine. Associating parking rules with the virtual parking area based at least in part on one or more of the above features, Performing one or more actions based at least partially on the virtual parking area and the associated parking rules, It is a system, The calculation of the geometry of the virtual parking area is The starting boundary line of the virtual parking area is generated based at least partially on the first position of the first feature among the one or more features indicating the starting point of the virtual parking area, The length of the virtual parking area is extended along the planned path based on the tracked movement of the ego machine until the second position of the second of the one or more of the aforementioned features indicates the end of the virtual parking area. To generate the end boundary line of the virtual parking area based at least partially on the second location and A system that includes this.

16. The system according to claim 15, wherein one or more of the features include a sign, an intersection, a surface color along the planned route, or a symbol on the surface along the planned route.

17. The detection of one or more of the above features is Removing at least one of the aforementioned one or more features by filtering to generate a filtered set of features. The filtering process includes, The at least one of the aforementioned features lies outside the orbital manifold of the egomachine. The fact that at least one of the aforementioned features is located on a different path from the ego machine, or The distance between the ego machine and the at least one feature is greater than the threshold distance. Based at least in part on at least one of the following, The calculation of the geometry of the virtual parking area is performed using a filtered set of features. The system according to claim 15.

18. The calculation of the geometry of the virtual parking area is The geometry of the virtual parking area is calculated at least partially based on the relative positions of the first and second positions with respect to the ego machine. The system according to claim 15, further comprising:

19. The aforementioned processing electrical circuit further, A first virtual marker corresponding to the starting boundary line of the virtual parking area is generated. A second virtual marker is generated that corresponds to the end boundary line of the virtual parking area. The semantic information corresponding to the first position and the first virtual sign, the second position and the semantic information corresponding to the second virtual sign are stored in the memory of the ego machine. The system according to claim 15.

20. The system according to claim 15, wherein the detection of one or more features includes determining that the parking information is related to at least a portion of the planned route, at least on the basis of estimating the distance between the one or more features and the planned route.

21. The aforementioned system Control systems for autonomous or semi-autonomous machines, Cognitive systems for autonomous or semi-autonomous machines A system that performs simulation operations. A system that generates synthetic data using machine learning. A system that generates multidimensional assets using a collaborative content creation platform. A system that performs deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The system according to claim 15, which is included in at least one of the following.

22. A method performed by one or more circuits provided in a processor, A step of detecting one or more features indicating parking information related to at least a portion of the planned path of the ego machine, based at least in part on sensor data generated using one or more sensors of the ego machine, A step of calculating the geometry of a virtual parking area based at least partially on one or more of the aforementioned features and the tracked movement of the ego machine, A step of associating a parking rule with the virtual parking lane based at least in part on one or more of the aforementioned features, A step of generating a parking assistance output that shows the parking rules and the relative position of the virtual parking area to the ego machine. Includes, The step of calculating the geometry of the virtual parking area is: A step of generating a starting boundary line for the virtual parking area based at least partially on a first position of a first feature among the one or more features indicating the starting point of the virtual parking area, A step of extending the length of the virtual parking area along the planned path based on the tracked movement of the ego machine until the second position of the second of the one or more features indicates the end of the virtual parking area, The steps of generating the end boundary line of the virtual parking area based at least partially on the second location and Methods that include...

23. The method according to claim 22, wherein one or more of the features include a sign, an intersection, a surface color along a planned route, or a symbol on the surface along the planned route.

Images