Parallel processing for planning vehicle routes suitable for parking.

Abstract

To provide a system and a method for parallel processing of a vehicle path planning suitable for parking.SOLUTION: To determine a path through a pose configuration space, trajectories of poses may be evaluated in parallel based at least on translating the trajectories along at least one axis of the pose configuration space (e.g., an orientation axis). A trajectory may include at least a portion of a turn having a fixed turn radius. Turns or turn portions that have the same turn radius and initial orientation can be translatively shifted along and processed in parallel along the orientation axis as they are translated copies of each other, but with different starting points. Trajectories may be evaluated based at least on processing variables used to evaluate reachability as bit vectors with threads, effectively performing large vector operations in synchronization. A parallel reduction pattern may be used to account for dependencies that may exist between sections of a trajectory for evaluating reachability, allowing for the sections to be processed in parallel.SELECTED DRAWING: Figure 1A

Description

[Background technology]

[0001] To control a vehicle or other maneuverable object, a proposed path can be determined from the vehicle's current attitude (e.g., position and orientation) toward a target attitude for the vehicle (e.g., autonomous parking). Determining a proposed path for a vehicle may involve exploring potential paths in a given configuration space under non-holonomic constraints described by the vehicle's kinematics. Conventional methods for solving this problem involve graph search using heuristics that reduce the search space by incentivizing early exploration of promising paths (e.g., A * ) may be used. However, the best course of action may not be immediately apparent and therefore may not be specified in a scenario. Similarly, environments in which heuristics do not apply precisely as heuristics may not be applicable to all situations.

[0002] In addition, the algorithms used by conventional systems are sometimes only lightly parallelized. For example, to speed things up fourfold, A uses eight processing threads. * Parallel implementations of this have been developed. However, modern parallel processors can utilize thousands of processor cores with a much higher degree of parallelism. Due to the limited parallelization provided by conventional methods, the density of the configuration space investigated to determine the path in a computationally efficient way is often relatively sparse. In other words, if the configuration space is too dense, identifying the path using conventional methods can be extremely time-consuming or consume too many resources for practical purposes. Therefore, the number of possible paths that can be determined may be limited by the density of the configuration space. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Patent Application No. 16 / 101,232 [Overview of the Initiative] [Means for solving the problem]

[0004] Embodiments of this disclosure relate to large-scale parallel vehicle path planning suitable for parking. A system and method are disclosed that can determine the path of a vehicle through attitude configuration space in a highly parallel manner.

[0005] In contrast to the conventional methods described above, the disclosed method allows for the rapid investigation of paths (e.g., all paths) in a dense attitude configuration space in parallel with determining a path through the attitude configuration space. The attitude trajectory in the attitude configuration space can be evaluated in parallel based on translating the trajectory along at least one axis of the attitude configuration space (e.g., the θ-axis representing the orientation of the vehicle). In at least one embodiment, the trajectory may include at least a portion of turns having a constant turn radius. Turns having the same turn radius and initial orientation are translated copies of each other but have different starting points (x,y), and can therefore be translated and processed in parallel along the θ-axis. Further, the trajectory can be evaluated based on at least treating the variables used to evaluate reachability as bit vectors. Logical bitwise operations may be performed by each processing thread to control the propagation of reachability while avoiding conditional processing or branching, so that threads can synchronously and effectively perform large-scale vector operations. The disclosed method uses a parallel reduction pattern to consider any dependencies that may exist between sections of a trajectory in order to assess reachability, and to enable the sections to be processed in parallel.

[0006] The system and method for parallel processing of vehicle path planning suitable for parking will be described in detail below with reference to the attached diagrams. [Brief explanation of the drawing]

[0007] [Figure 1A] This figure includes examples of a route planner according to some embodiments of the present disclosure. [Figure 1B] This figure shows additional elements that may be included in the route planner of Figure 1A according to some embodiments of the present disclosure. [Figure 2] This figure shows examples of attitude configuration spaces that can be used to model vehicle attitudes according to some embodiments of the present disclosure. [Figure 3] This figure illustrates an example of a motion model that can be used to define the trajectory of a vehicle through an attitude configuration space, according to some embodiments of the present disclosure. [Figure 4] This figure shows an example of an occupancy space that captures the occupancy rate of a posture in a posture configuration space, according to some embodiments of the present disclosure. [Figure 5] This figure illustrates examples of different turn types that can be evaluated in posture configuration space, according to some embodiments of the present disclosure. [Figure 6A] This figure shows an example of a cork screw shape of a turn laid out in a posture configuration space, according to some embodiments of the present disclosure. [Figure 6B] Figure 6A shows an example of the corkscrew shape of a turn, with the corkscrew shape of another turn being a translated replica of the turn laid out in the attitude configuration space, according to some embodiments of the present disclosure. [Figure 6C] This figure shows an example of a shifted orbit formed by translating the cork screw shape in Figure 6B, according to some embodiments of the present disclosure. [Figure 7] This figure illustrates how a linear driving path can be conceptually mapped to attitude configuration space in some embodiments of the present disclosure. [Figure 8] The following are examples of computational flow graphs that may be used to process turns, where reachability is encoded using binary values, according to some embodiments of the present disclosure. [Figure 9]FIG. is an example of a computational flow graph that can be used to process turns, where reachability is encoded using non-binary values, according to some embodiments of the present disclosure. [Figure 10] FIG. is a flowchart showing a method for determining a shifted pose of a pose configuration space to determine a path through the pose configuration space, according to some embodiments of the present disclosure. [Figure 11] FIG. is a flowchart showing a method 1100 for translating a shifted trajectory of a pose in a pose configuration space to determine a path through the pose configuration space, according to some embodiments of the present disclosure. [Figure 12] FIG. is a flowchart showing a method 1200 for evaluating the reachability of sections of a trajectory in parallel, according to some embodiments of the present disclosure. [Figure 13A] FIG is an illustration of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 13B] FIG is an example of the camera position and field of view of the exemplary autonomous vehicle of FIG. 13A, according to some embodiments of the present disclosure. [Figure 13C] FIG is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle of FIG. 13A, according to some embodiments of the present disclosure. [Figure 13D] FIG is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle of FIG. 13A, according to some embodiments of the present disclosure. [Figure 14] FIG is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 15] FIG is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure.

MODE FOR CARRYING OUT THE INVENTION

[0008] A system and method are disclosed for parallel processing of vehicle path planning suitable for parking. This disclosure may be described in relation to an exemplary autonomous vehicle 1300 (or referred to herein as “Vehicle 1300” or “Ego Vehicle 1300,” examples thereof are described with respect to Figures 13A–13D), but this is not intended to be limiting. For example, the systems and methods described herein may be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotics platforms, warehouse vehicles, off-road vehicles, vehicles attached to one or more trailers, flying ships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. In addition, while this disclosure may describe, but is not intended to limit, the systems and methods described herein may be used for route planning in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technological space where route planning may be used.

[0009] In parking applications, the objective may be to maneuver a vehicle (e.g., from within an approximate line of sight) to a parking spot while avoiding collisions with obstacles (e.g., other vehicles, pillars, barriers, walls, parking structures, pedestrians, etc.) under non-holonomic constraints. Disclosed are systems and methods that can determine the vehicle's path from its current orientation to a target orientation in orientation configuration space (e.g., a multi-point turn suitable for parking, or maneuvering in a narrow space), in which case the orientation may be in free space or blocked by obstacles (e.g., obstacles recognized based on vehicle sensor data). That is, several different paths from the current orientation to the target orientation may exist, each path may consist of various combinations of turns (e.g., sharp left, gentle left, straight, gentle right, and sharp right) and directions (e.g., forward and reverse). Furthermore, one or more obstacles may be located along one or more of the paths. The disclosed method can be used to evaluate a course and identify a recommended course to go around an obstacle and towards a target orientation based on one or more criteria (e.g., shortest distance, fastest time, minimum number of turns).

[0010] The disclosed method allows for a parallel and rapid exploration of paths (e.g., all paths) in a dense attitude configuration space when determining a recommended path from the current attitude to a target attitude (e.g., from a set of attitude targets). An iterative method can be used in which the reachability of a set of trajectories can be evaluated in parallel in one iteration, and the results can serve as input for evaluating the reachability of that set of trajectories (or a different set of trajectories) in the next iteration (effectively extending reachable trajectories with additional trajectories). Disadvantageous path recommendations can be avoided by thoroughly exploring the probabilities of the paths. While work of this scale has traditionally been too large to be practically performed for dense attitude configuration spaces, the disclosed method takes into account large-scale parallelism that can leverage modern parallel processing architectures (e.g., with thousands of cores and / or threads). Therefore, it is possible to explore dense attitude configuration spaces much faster, more efficiently, and / or with a much finer granularity (e.g., with higher spatial and angular density) than was previously possible.

[0011] In at least one embodiment, the attitude configuration space may represent the attitude of a vehicle in a given environment using at least the x and y axes for the vehicle's position and the θ axis for the vehicle's orientation (e.g., azimuth). Large-scale parallelism may arise, at least in part, from evaluating the attitude trajectory in the attitude configuration space in parallel, based at least on translating the trajectory along at least one axis (e.g., the θ axis) of the attitude configuration space. Such a mechanism would allow the trajectory and / or parts thereof to be worked independently as parallel processing. For example, at least parts of the attitude of the trajectory could be shifted (e.g., on the fly or beforehand using a general translation function) to form parallel lines, one or more sections of these parallel lines could be processed independently in parallel to evaluate reachability.

[0012] In one or more embodiments, the track may include at least a portion of turns having a constant turn radius. When the track is a turn, it may follow a circular or linear (infinite turn radius) track. The disclosed technique may take advantage of the fact that each turn (or portion thereof) of a turn having the same turn radius and initial direction (same turn type) can be a translated copy of each of the other turns (or their corresponding portions), but have different starting points (x,y), due to the fact that the vehicle can behave the same regardless of where it is initially positioned. Thus, turns (or portions of turns) of turn types corresponding to different tracks can be translatedly shifted along the θ axis and processed in parallel along the θ axis.

[0013] Furthermore, the variables used to evaluate reachability may consist of bits representing binary values ​​(e.g., whether reachable, whether in free space), with each attitude corresponding to an individual bit of the variable. Thus, trajectories can be evaluated based on treating the variable as a bit vector, for example, one trajectory per bit of a bit vector can be evaluated in parallel. Logical bitwise operations may be performed by each thread, avoiding conditional processing or branching (which do not contribute to parallelism) to control the propagation of reachability, so that threads can perform large vector operations synchronously and effectively. Thus, this technique may be suitable for modern parallel processors that develop with large vector operations well-aligned in memory, in the manner of Simultaneous Instruction Multiple Threading (SIMT).

[0014] This disclosure also provides a technique that can be used to divide the processing work used to evaluate the trajectory into sections of the trajectory so that these sections can be evaluated in parallel rather than the entire trajectory. Since reachability may undulate throughout the entire trajectory (for example, both forward and backward in a periodic trajectory), the sections of the trajectory may not be independent of each other. This dependency may hinder the parallel processing of the sections. The disclosed technique can address this dependency using a parallel reduction pattern, which hierarchically collects results from sections (for example, computed in parallel), performs some small number of operations (for example, in parallel) using the collected results, and then returns the results again (for example, in parallel) as input to the individual sections for further processing.

[0015] This route planner can offer additional opportunities for parallelism. For example, when processing the reachability of each turn of each turn type, a turn can be divided into subsections, and each subsection of a turn can be processed semi-independently by a separate thread. In particular, the option to process subsections of a turn independently can spread the work to more threads (if available). In addition, since subsections of a turn tend to be shorter than a full turn, memory latency can be reduced because it becomes less likely for one thread to get stuck (for a long time) and wait for another thread to finish.

[0016] Referring to Figure 1A, Figure 1A is a diagram including an example of a route planner 110 according to several embodiments of the present disclosure. It should be understood that this and other configurations described herein are provided merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0017] In at least one embodiment, the route planner 110 may include a configuration space manager 136, a free space manager 120, a reachability manager 122, and a route evaluator 130. The configuration space manager 136 can manage the attitude configuration space 112, which represents the attitude of a vehicle (or other object) including its position and orientation in a given environment (e.g., a parking lot) (e.g., attitudes 114, 116, and 117).

[0018] The free space manager 120 and the reachability manager 122 can process the attitude configuration space 112 to determine one or more paths to maneuver in the attitude configuration space 112 from the current attitude C to a target attitude T (or generally between any two attitudes). For example, the free space manager 120 may conduct a collision test of the vehicle with an object 124 in the environment to determine an occupancy space 118 in which it can capture which attitudes may be blocked or occupied (e.g., attitudes 116 and 118) and which attitudes may be free or unoccupied (e.g., attitude 114). The reachability manager 122 may analyze at least some of the trajectories of attitudes the vehicle may travel in the attitude configuration space 112 (e.g., trajectories 126 and 128) to determine a reachability space (may also be called cost space) 102 in which it can capture which attitudes in the attitude configuration space 112 are reachable by trajectory and / or the cost to reach those attitudes. The route evaluator 130 can identify one or more proposed or potential routes for the vehicle, based at least on an assessment by the reachability manager 122. For example, the route evaluator 130 can identify and / or evaluate routes or multi-segment trajectories (e.g., routes 132 and 134) based on one or more criteria (e.g., distance, number of turns, number of gear changes, cost, reachability, etc.).

[0019] Next, referring to Figure 1B, which shows additional elements that may be included in the trajectory planner 110 of Figure 1A according to some embodiments of the present disclosure. The configuration space manager 136 may include a parameter controller 138, the reachability manager 122 may include an attitude translator 146 and a reachability evaluator 148, the free space manager 120 may include an object detector 140 and an occupancy evaluator 142, and the trajectory evaluator 130 may include a trajectory assessor 150 and a back tracer 152.

[0020] In at least one embodiment, the configuration space manager 136 can use the parameter controller 138 to facilitate operations with respect to the attitude configuration space 112 and set parameters for the attitude configuration space 112. The attitude configuration space 112 can represent vehicle attitudes using a space such as a multidimensional space (e.g., 3D space). For example, the vehicle may be in a driving environment (e.g., a parking lot). Each attitude in the attitude configuration space 112 may include at least one vehicle position in the driving environment and the orientation of the vehicle at that vehicle position. In at least one embodiment, the position in the environment can be represented in the attitude configuration space 112 using an XY grid (e.g., representing the base plane), as shown in Figure 1A. The orientation in the environment can be represented in the attitude configuration space 112 using an angular component θ (e.g., rotation around the x-axis). In at least one embodiment, the attitude configuration space 112 can be parameterized as attitude P=(x,y,θ).

[0021] Figure 2 shows examples of attitude configuration spaces 112 that can be used to model vehicle attitudes according to some embodiments of the present disclosure. Figure 2 illustrates examples of how a grid of attitudes 210 stored in the attitude configuration space 112 (each attitude P=(x,y,θ)) represents the position 212 of the vehicle in the driving environment. For example, Figure 2 illustrates that each θ slice 214 and 216 of the attitude configuration space 112, in combination with the individual angular direction θ, may represent a set of (x,y) values.

[0022] Figure 3 illustrates an example of a motion model 300 that can be used to define the trajectory of a vehicle through an attitude configuration space 112, according to some embodiments of the present disclosure. The attitude captured in the attitude configuration space 112 may correspond to a point on the trajectory captured by the motion model 300. In the examples shown, the motion model 300 includes an Ackermann model for the motion. Under the motion model 300, a turn can proceed as the motion 310 of a circular body, where the center of the turn 312 is axially aligned with the rear axle 314 of the vehicle at a distance 316 from the center of the rear axle 318. As shown in Figure 3, the center of the rear axle 318 may correspond to the x,y coordinates of the attitude in the attitude configuration space 112. The distance 316 corresponds to the radius of the turn, which may be determined by the angles of the vehicle's front wheels 320 and 322. As shown in Figure 3, the angles of the vehicle's front wheels 320 and 322 may correspond to the θ coordinates of the attitude in the attitude configuration space 112.

[0023] Various parameters, such as the size of the environment represented by the attitude configuration space 112, as well as spatial and angular cell sizes, can affect the attitude configuration space 112. Therefore, the configuration space manager 136 can use the parameter controller 138 to define these settings. For example, the parameter controller 138 may receive the dimensions of the environment being evaluated (e.g., 30m x 30m) and the cell size (e.g., 0.3m), which can affect the spatial density of the attitude configuration space 112 (e.g., the number of cells along the x and y axes). In addition, the angular cell size (e.g., in radians) or the number of angular cells may also be given to the parameter controller 138, which can affect the angular density of the attitude configuration space 112 (e.g., the number of layers along the θ axis). A posture configuration space with more cells in a given space (for example, due to smaller spatial cell sizes and / or angular cell sizes in a given area) can present a greater number of eligible postures and paths for evaluation, potentially allowing for greater precision when dealing with postures.

[0024] In at least one embodiment, the occupancy rate of the attitude configuration space 112 can be calculated, for example, using a free space manager 120, to determine a course and / or multi-point turns specific to a given environment. That is, the free space manager 120 can determine which attitudes in the attitude configuration space 112 are perceived as at least partially occupied by obstacles, and which attitudes are perceived as empty, i.e., free. The free space manager 120 can use an object detector 140 to detect and / or identify objects that may occupy or obstruct a position in the environment. The free space manager 120 may additionally or alternatively receive data representing objects from an object detector 140 operating outside the course planner 110. The free space manager 120 can use an occupancy rate evaluator 142 to perform collision tests of the vehicle body against an object. A margin can be provided using some bulge of the vehicle and / or the object's body.

[0025] During the crash test, the representations of the obstacle and the vehicle body used as input to the occupancy rate evaluator 142 may be polygonal or rasterized, and the output may be rasterized. The crash test may use the center of the vehicle's rear axle at (x,y), rotated θ with respect to the x-axis. Referring now to Figure 4, which is a diagram of an example of an occupancy space 412 that captures attitude occupancy in attitude configuration space according to some embodiments of the present disclosure. In particular, Figure 4 depicts at least a portion of the occupancy space 412 that can be parameterized (for example, using at least x,y,θ) in the same way as the attitude configuration space 112. In addition, Figure 4 depicts an example obstacle 408 that can be recognized, and depicts the position of the obstacle 408 relative to a set of (x,y) values ​​414 that can be consistent across all θ. Furthermore, Figure 4 illustrates example data values ​​for θ planes 416 and 418, each of which can correspond to individual vehicle angular directions.

[0026] As an example, and not by way of limitation, the occupancy space 412 can store 1 bit per cell, where 1 can represent that the corresponding pose is free and zero can represent that the corresponding pose is not free (e.g., occupied). As shown in FIG. 4, a portion of the pose 420 for the θ plane 416 is set to 1 and is free based on the angular direction of the vehicle. However, the corresponding pose 422 in the θ plane 418 is set to zero and is potentially blocked due to the different angle of the vehicle and potentially overlaps with the obstacle 408.

[0027] In at least one embodiment, the occupancy estimator 142 may calculate the occupancy of the poses in the pose configuration space 112 and write the results to the occupancy space 412 in parallel. For example, each pose may be tested separately (e.g., by a thread) and the results written to the occupancy space 412 in parallel (e.g., by that thread). The data in the occupancy space 412 may be used by the reachability manager 122 for subsequent processing as well as for path identification and evaluation.

[0028] Using data from the occupancy space obtained from the free space manager 120, the path planner 110 executes an algorithm to find a recommended path from the current pose (x c , y c , θ c ) to a set of target poses (x min , y min , θ min ), (x max , y max , θ max ) given the non-holonomic constraints of the vehicle. Both the current pose and the set of target poses may be provided to the path planner 110 as input from one or more other motion planners (e.g., when detecting a parking spot, a location where a robot is permitted to lift a pallet, etc.).

[0029] In at least one embodiment, the route planner 110 may be configured to evaluate one or more predetermined turn or trajectory types relative to the current or starting attitude in the attitude configuration space 112, in which case the turn or trajectory type may correspond to a given turn radius and direction (e.g., forward or backward). For example, the trajectory type may define multiple trajectories across the entire attitude configuration space 112, and the reachability manager 122 may use the reachability evaluator 148 to evaluate all or almost all of those attitudes relative to the initial or current attitude. While this specification primarily describes trajectories that are at least part of a turn as an example, the disclosed embodiments can be applied more generally to other types of trajectories that may be constructed from, for example, turn primitives.

[0030] In at least one embodiment, the reachability evaluator 148 can determine reachability, for example, whether a trajectory attitude can be reached from the vehicle's current attitude or starting attitude C (in some embodiments, a target attitude T or other attitude). An iterative method may be employed in which the reachability manager 122 evaluates the reachability of a set of trajectories (e.g., one or more turn types) and uses the result of that evaluation as input to evaluate the reachability of a set of trajectories (or different sets of trajectories) in the next iteration. For example, the trajectory of the next iteration may be reachable on the basis that it is reachable (e.g., connected to) at least one trajectory from the preceding iteration, and that the starting attitude is reachable by that at least one trajectory from the preceding iteration.

[0031] The reachability evaluator 148 may evaluate reachability for any number of trajectory runs (e.g., up to a maximum threshold number of trajectory runs and / or until the target attitude can be reached). For example, the reachability manager 122 may evaluate reachability if the vehicle performs a single turn or a multi-point turn in the attitude configuration space 112 (e.g., up to a maximum threshold number of turns, such as 8 turns). In at least one embodiment, the reachability evaluator 148 may also evaluate the reachability of a trajectory in terms of whether its attitude is free (e.g., using a determination from the free space manager 120). For example, obstacles in the trajectory may automatically eliminate the reachability of any subsequent attitudes in the trajectory.

[0032] In some embodiments, the reachability evaluator 148 may also determine and / or record the cost associated with reaching the attitude in the orbit, for example, while evaluating the reachability of the attitude. As described herein, the cost score may be used as an indicator of reachability, for example, using 1 or 0 to indicate whether the attitude is reachable, with the highest cost score indicating unreachability and the other cost score indicating reachability, and so on. In other embodiments, the reachability manager 122 may store the cost score separately from the reachability indicator. In various embodiments, one or more obstacles may introduce some cost that may be greater than the cost value in the absence of the obstacles, even though the attitude in the orbit may not necessarily be unreachable. In addition, different obstacles may have different cost values.

[0033] As described herein, one or more of the trajectories associated with the attitude in attitude configuration space 112 may be turns, each "turn" may include the vehicle moving forward or backward a distance while keeping the steering wheel fixed (thus maintaining the turn radius). For example, Figure 5 illustrates examples of various turn types that can be evaluated in attitude configuration space 112, including a sharp left 510, a gentle left 512, a straight 514 (θ=0), a gentle right 516, and a sharp right 518. Each of the various turn types may include individual turn radii, with left turns represented as negative (-) and right turns as positive (+). For example, the radius of a sharp left 510 may be -10m, and the radius of a gentle left 512 may be -20m. On the other hand, the radius of a sharp right 518 may be 10m, and the radius of a gentle right 516 may be 20m.

[0034] In at least one embodiment, while performing a certain type of turn, the vehicle follows an attitude trajectory in attitude configuration space 112, which may be defined using equation (1): P(θ)=(x0+qQ sinθ,y0+qQ(cosθ-1),θ), (1) It is parameterized by θ, where Q is the turning radius, q is the turning direction (for example, +1 is right when moving forward, -1 is left when moving forward), and (x0, y0) is the position when θ is zero.

[0035] The reachability evaluator 148 may process through all potential turn combinations, including any variation of turn type (e.g., forward, backward, backward). After all reachability evaluation iterations, the reachability manager 122 may update the reachability space 102 with an indication of whether each attitude in the attitude configuration space 112 is reachable in that particular iteration (e.g., reachable to return to the starting attitude and not blocked by obstacles), and / or with the cost of reaching the attitude. Traversing all trajectories in all iterations can represent a massive amount of processing, which in some cases can be greatly accelerated by parallel processing.

[0036] In at least one embodiment, to process trajectories in parallel to determine the reachability of attitudes in a trajectory, the reachability evaluator 148 can analyze disjoint trajectories within the attitude configuration space 112, enabling independent processing. According to the disclosed embodiment, disjoint trajectories may include a common trajectory or a set of trajectories of a turn type (e.g., turn radius and direction). When processing a trajectory or a section thereof (e.g., by individual threads, respectively), the attitude translator 146 of the reachability manager 122 may be configured to determine the shifted attitude of the attitude configuration space 112, based on translating the attitude of the attitude configuration space 112 along at least one axis of the attitude configuration space 112, in order to determine the shifted attitude of the disjoint trajectories. Disjoint trajectories (or sections thereof) may be evaluated using parallel processing (e.g., by individual threads) to enable the reachability evaluator 148 to quickly evaluate the reachability of the shifted attitude.

[0037] Using Figures 6A and 6B, examples of techniques that may be used by the attitude translator 146 to translate attitudes in attitude configuration space 112 are illustrated. Referring to Figure 6A, Figure 6A illustrates an example of a cork screw shape of a turn 610 laid out in attitude configuration space 112 according to some embodiments of the present disclosure. As shown, the turn 610 is conceptually mapped in a cork screw shape in attitude configuration space 112 by following the attitude trajectory defined by equation (1). That is, as the turn progresses through space (for example, following the attitude defined by equation (1)), the turn moves from the current coordinate (x,y) to the next coordinate (x,y), and at the same time can also move along the θ axis through a θ plane such as the θ plane 616. In other words, all x,y, and θ coordinates may change throughout the turn in the cork screw shape shown in Figure 6A.

[0038] In the attitude configuration space 112, due to the fact that vehicles can behave the same way regardless of their initial position, each turn having the same turn radius and initial direction (same turn type) can be a translated copy of another turn, but having a different starting point (x,y). For example, Figure 6A illustrates a turn 610 that starts at (x,y) coordinate 614 in the θ-plane 616, associated with a fixed turn radius. Each of the other turns having the same turn radius (but starting at a different (x,y) coordinate in the θ-plane 616) can be a translated copy of turn 610 in the attitude configuration space 112. Referring to Figure 6B, for example, Figure 6B is an example of the cork screw shape of turn 610 in Figure 6A, with a cork screw shape of turn 612, which is a translated copy of turn 618 laid out in the attitude configuration space 112, according to some embodiments of the present disclosure. Turns having the same turn radius may have translational parallelism, which can be used for parallel processing using the attitude translator 146. While turns have been described, other types of orbits and / or sections thereof may also have translational parallelism and can likewise be used for parallel processing as described herein.

[0039] In at least one embodiment, the attitude translator 146 can translationally shift the attitudes of the attitude configuration space 112 along at least one axis (e.g., the θ-axis), and the reachability evaluator 148 can process each of the attitudes in parallel along at least one axis to assess reachability (e.g., one thread per turn or per section). For example, turn-type parallelism can be revealed by the attitude translator 146, which performs a translational shift that transforms a set of corkscrew shapes in the attitude configuration space 112 corresponding to the same turn radius into a bundle of parallel lines along the θ-axis. The parallel lines traverse the attitude configuration space 112 in parallel as dissimilar trajectories, allowing for their independent processing.

[0040] In at least one embodiment, for a given turn radius and direction (turn type), the attitude translator 146 may perform a translational shift defined by equation (2): t(θ)=(t x (θ), t y (θ))=(qQ sinθ,qQ(cosθ-1)) (2) As a function of θ, equation (3): (x,y,θ)→(xt x (θ), yt y (θ),θ) (3) A deformation defined by (for example, using warp) is applied to the attitude configuration space 112. This may correspond to a translational shift of each constant θ plane in the attitude configuration space 112. A modified version of the turn-type turn trajectory may be defined by equation (4): P(θ)=(x0,y0,θ), (4) This is a set of lines parallel to each other and to the θ-axis. For example, referring to Figure 6C, Figure 6C is a diagram illustrating an example of a shifted trajectory formed by translating the corkscrew shape of Figure 6B, according to some embodiments of the present disclosure. In particular, Figure 6C conceptually illustrates the attitude configuration space 112 with turns 610 and 618 translated or shifted by the attitude translator 146. As shown, turns 610 and 618 (for example, having the same turn radius and direction) are now parallel lines along the θ-axis and can be treated as such. This deformation may be possible because the trajectory is an invariant translation with respect to translation along x and y in the sense that the action of traversing the curve of the turn is exchanged with the action of translating it. Turn-type trajectories can be thought of as translated copies of each other and may have the same shape regardless of the starting point (x0, y0) coordinates. Parallelism along the θ-axis contributes to the large-scale parallelism of the trajectory planner 110 in a manner suitable for modern parallel processing. Optionally, the data may be transposed to switch between the θ and x axes so that the trajectory is always parallel to the x axis (or θ axis).

[0041] In one or more embodiments, the attitude translator 146 may deform the attitude configuration space 112 for each turn type (e.g., same radius and direction) such that the curves of a certain turn type form parallel lines (e.g., virtually by access patterns, or actually by duplicating data around them). The reachability evaluator 148 and / or other system components can then process in parallel along each of the lines. The processing may be performed in parallel with respect to the θ axis along each of the parallel lines as selected by (x0, y0), while accessing the original attitude configuration space according to equation (5): (x0+t x (θ), y0+t y (θ),θ), (5) This may also be given by the inverse of the deformation defined above. This deformation can be easily inverted and is therefore bijective. When using a discretized implementation of the deformation, it may be desirable to maintain this bijective property. For example, having a bijective deformation allows for complete parallel separation between threads processing individual orbits (or sections thereof) without relying on atomic operations or worrying about read / write race conditions. This also improves the ability of each cell in the discretized attitude configuration space to be handled by one of the threads when processing turn types. The bijective property can be realized, for example, by quantizing t(θ) in the same way in each constant θ plane. The deformation of the attitude configuration space 112 performed by the attitude translator 246 may also be a translational shift of each constant θ plane, which can be considered as an image of an integer number of pixels. This may be ideally suited to parallel implementations because the shifted memory access occurs on the fly while processing is taking place, thereby reducing memory access that could be a limiting factor. In addition, differences in translational shifts in a continuous plane are reduced (for example, so that they do not differ by multiple pixels or some other threshold), thus avoiding large jumps associated with discretized trajectory processing. This can be achieved by having sufficient angular resolution in the attitude configuration space 112.

[0042] When assessing reachability, the course planner 110 may evaluate curved turns, as well as linear maneuvers (for example, turns with an infinite turning radius). When moving along a linear course in a trajectory of attitude, the attitude may be defined using equation (6): P(u)=(x0+u cosθ,y0+u sinθ,θ), (5) u is parameterized, where θ is the current constant azimuth angle, and (x0,y0) is the starting position when u is zero. In attitude configuration space 112, trajectories can be represented as lines at an angle θ in each θ plane. For example, Figure 7 illustrates trajectories 710 and 712, which start at individual (x,y) coordinates and are both oriented according to a θ associated with the θ plane 714. In addition, Figure 7 illustrates trajectories 716 and 718, which start at individual (x,y) coordinates and are both oriented according to different θ associated with the θ plane 720.

[0043] The linear transformation to parallel lines aligned to one of the coordinate axes (e.g., by attitude translator 146) can be achieved in various ways. For example, in at least one embodiment, the θ plane can be rotated by an angle of -θ, and as a continuous transformation, this technique is bijective. For example, a warp of work may be defined such that each warp independently rotates a constant θ plane by an angle of -θ around its center so that all lines being driven are parallel to the x-axis. This technique can be efficient on a GPU because the warps are of a constant θ plane, which are rotations of an image (e.g., to preserve integer integrity in the space being warped) that can be achieved using the interpolation method "nearest neighbor" (e.g., to preserve integer integrity in the space being warped). The rotation may also rotate some angles outside the original space, which can be described by using a padded version of the space. All cells outside the original space can be treated as never reachable and are not permitted, such as due to obstacles or high cost.

[0044] Discretized image rotations may not readily preserve the properties of bijective deformations, but bijective image rotations can be achieved by three shear coordinate transformations. In an alternative embodiment, a single shear transformation may be applied to each θ-plane, which may generate less discretization noise and can provide the ability to perform the transformation on the fly while being handled by simple translational shifts of memory-accessed vector operations. For example, if the direction of a line is closer to the x-axis than the y-axis (|tanθ|≦1), or close to some other threshold, the attitude translator 146 may shear the trajectory to be parallel to the x-axis, and vice versa. In the former case (e.g., close to the x-axis), the attitude translator 146 may set x0=0, and in the latter case (e.g., close to the y-axis), the attitude translator 146 may set y0=0 to further consider all lines. These operations can give a trajectory family defined by equation (6):

number

number

number

number

number

[0045] As described herein, an iterative method may be employed in which the reachability manager 122 evaluates the reachability of a set of trajectories (e.g., one or more turn types) and uses the result of that evaluation as input to evaluate the reachability of a set of trajectories (or different sets of trajectories) in the next iteration. In this sense, the reachability evaluator 148 can be used to determine whether a posture in the posture configuration space 112 (e.g., as deformed by the posture translator 146) is reachable from the current position by processing one or more trajectories in one iteration, then performing one or more subsequent iterations (e.g., up to a given maximum number of turns), and determining whether the posture will be reachable in the next iteration. For example, the reachability evaluator 148 may evaluate whether a posture is reachable based on whether the posture is blocked or occupied (e.g., using the occupancy space 412) and whether it is included in one or more trajectories (e.g., 510, 512, 514, 516, 518, etc.) in a given iteration.

[0046] In at least one embodiment, the reachability evaluator 148 determines the current or starting posture (x c ,y c ,θ cThe iteration may begin with a certain orientation, and only that orientation may be marked as reachable. In the first iteration, the reachability evaluator 148 may annotate the reachable space 102 for the first iteration to indicate (e.g., mark) whether each orientation is reachable from the current orientation using the trajectory in the first iteration (e.g., along at least one turn trajectory of the trajectory, not blocked from the current orientation by an obstacle). For example, the reachability evaluator 148 may evaluate reachability with respect to each turn type. This may result in a reachable space 102 after the first iteration showing a set of reachable orientations. In the second iteration, the reachability evaluator 148 may annotate a different reachable space 102 (or, in some embodiments, the same reachable space 120) for the second iteration to indicate (e.g., mark) whether each orientation is reachable from the current orientation using the trajectory in the second iteration (e.g., for each turn type and / or different turn types). The iterations may continue in the same manner until the reachability evaluator 148 reaches its maximum number of iterations (e.g., 8 turns) and discovers the target orientation, and / or until some other termination condition is met. The route evaluator 130 may use the annotated reachability space 102 to identify which route is recommended, if any, based on a cost function (e.g., minimum number of turns, shortest distance, or other cost assessment) or other method.

[0047] The reachability evaluator 148 can process the attitude configuration space 112 in various ways as the iteration progresses. In at least one embodiment, the reachability evaluator 148 may refer to an occupancy space 112 parameterized by (x,y,θ), and separate reachable spaces 102, also parameterized by (x,y,θ), for each trajectory type (e.g., reachable space 102 for abrupt left forward driving, reachable space 102 for abrupt left reverse driving, reachable space 102 for linear forward driving, etc.). In addition, each cell correlated to a single attitude (x,y,θ) may be updated in the corresponding reachable space 102 when the reachability evaluator 148 evaluates whether the individual attitude is free and reachable in its iteration.

[0048] Referring next to Figure 8, which illustrates an example of a computational flow graph 800 that may be used to process turns in which reachability is encoded using binary values, according to some embodiments of the present disclosure. The computational flow graph 800 may be suitable for embodiments in which binary values ​​are used to store whether a posture is reachable or not. In one or more embodiments, in each iteration, the reachability evaluator 148 may start in a shared reachability space and process each turn type in parallel to compute the corresponding reachability space. For example, in iteration 802A, the reachability evaluator 148 may start in reachable space 808A and process in parallel a steep left turn type to generate reachable space 810A, a steep left turn type to generate reachable space 810A, a gentle left turn type to generate reachable space 812A, a straight turn type to generate reachable space 814A, a gentle right turn type to generate reachable space 816A, and a steep right turn type to generate reachable space 818A. Although not shown, both forward and reverse turn types may be processed.

[0049] Each iteration may also include the reachability evaluator 148 merging reachable spaces for turn types to provide a shared reachable space for the next iteration. For example, iteration 802A may include the reachability evaluator 148 performing a merge 820A on reachable spaces 810A, 812a, 814A, 816A, and 818A to produce reachable space 808B as input to iteration 802B. In one or more embodiments, the merge 820A may include a logical OR operation. A logical OR may be suitable to reflect that the reachable spaces capture a variety of different ways to reach a particular cell, each of which may be valid. In at least one embodiment, the shared reachable space may be cached in shared memory (e.g., on a GPU) accessed by each thread.

[0050] In aspects of this disclosure, the reachability evaluator 148 propagates reachability along the entire turn trajectory, thereby marking all attitudes that are along the turn trajectory of a single reachable attitude (e.g., the shaded rectangle in Figure 9) and are not separated from the single reachable attitude by an obstacle as reachable. To determine which of one or more attitudes from a previous iteration is reachable, the route planner goes through all single turns with turn radius and direction, and this can be done in parallel as translational shifts are applied. At each attitude (x,y,θ), the route planner turns off reachability (e.g., based on occupancy space) if it finds an obstacle and turns on reachability if it finds reachability from a previous iteration.

[0051] Therefore, to process through iterations, the reachability evaluator 148 can make the reachable space from a previous iteration the reachable space after the iteration. In at least one embodiment, the reachability evaluator 148 may apply the rule that if the attitude on the trajectory being processed (e.g., a particular turn of a turn type) was previously reachable (as indicated by the reachable space), then all attitudes along the trajectory that were previously reachable and are not separated from any by obstacles are reachable. In this way, reachability can propagate along the trajectory until it is stopped by an obstacle.

[0052] Figure 8 includes pseudocode 830 to illustrate how a thread of the reachability evaluator 148 may process through M attitude cells of the trajectory using a core loop, where the core loop turns off the reachability represented by attitude cell i, local variable r, when the thread finds an obstacle such that it is represented by an element of the occupancy space input F, and turns on the reachability represented by attitude cell i, local variable r, when the thread finds reachability from a previous iteration such that it is represented by the reachability space input Ri, and then writes the reachability r to the reachability space output Ro. As described herein, the attitude may be shifted in advance or on the fly for processing using the attitude translator 146.

[0053] In one or more embodiments, all variables r, F, Ri, and Ro may be treated as bit vectors. For example, if the variables are declared as 32-bit unsigned integers, then 32 bits, or 32 parallel trajectories, may be treated in parallel as bit vectors. The treatment may not include conditional treatment or branching, and may only include a logical AND operation to stop propagation when free space is stopped, and a logical OR operation to start or restart propagation when reachability from a previous iteration is indicated in the input. In addition, the treatment of this core loop may flow in parallel and precisely synchronously among many threads, each responsible for one trajectory, thereby allowing the threads to effectively perform large vector operations synchronously.

[0054] Therefore, this approach, similar to but distinct from Simultaneous Instruction Multiple Data (SIMD), is a Simultaneous Instruction Multiple-Threading (SIMT) method that can be suited to modern parallel processors evolving with large-scale vector operations well-equipped in memory. For example, some parallel processing architectures may include basic units of parallel processing, such as warps or 32-thread wavefronts (for instance), which are executed concurrently in an effort to increase efficiency. Where referred to herein as warps, they may be more generally referred to as basic units of parallel processing. Each streaming multiprocessor can process one or more basic units concurrently, and there may be many multiprocessors. A single thread can simultaneously process 32 orbits (e.g., turns) as 1-bit vectors, so that each basic unit (e.g., warps) can process 32 × 32 = 1024 orbits in parallel. Furthermore, there may be eight or more multiprocessors, each processing many basic units concurrently, to achieve tens of thousands of parallelisms. Instead of applying heuristics with expectations to initiate a promising path, the path planner 110 can perform all tasks by allowing all trajectories (e.g., turns) to be evaluated in each iteration. Therefore, the path planner 110 can avoid recommendations resulting from inappropriate heuristics that attempt to limit calculations to the best possible work first.

[0055] The route planner 110 may proceed through one or more iterations, and the route evaluator 130 may use the results of one or more iterations to identify and / or recommend at least one route to use when steering the vehicle toward a target attitude. In this sense, the output reachable space from each iteration may be maintained to support back tracing as described herein. Since the reachable space requires only one bit per cell, this may mean that in the attitude configuration space, only N bits per cell are required. For a reasonable number of iterations, this is simply what is used by representing the space as an 8, 16, or 32-bit integer or floating-point number.

[0056] To identify and / or select a course, the course evaluator 130 may include a course assessor 150 that evaluates one or more costs associated with a trajectory (e.g., a turn), and there are many cost functions of varying complexity that can be applied to identify a recommended course. One simple cost function may consider the number of trajectories (e.g., turns) required to achieve a particular attitude and rank courses that reach that attitude with fewer trajectories higher. A more complex cost function may consider how many resources may be consumed to follow a certain course. For example, the course assessor 150 may apply a cost function that models the length of time it takes to travel a certain trajectory, which may include penalties for changing from forward to reverse or backward and assess any time spent traveling the trajectory depending on the distance. In even more complex examples, a state space may be used by adding turn curvature and sign (the underlying state may be the steering and wheel positions and gears) as well as velocity. This makes it possible to model the cost of having to slow down to a stop when changing to reverse or making a large change in steering position.

[0057] The trajectory assessor 150 can apply a variety of different cost models depending on the purpose and application of the route planner 110. For example, one cost model may remove the explicit velocity component and instead have a cost that depends on the distance traveled, at least for some turns, along with a penalty for changes in turn type (including changes to reverse), which depend on the types of turns before and after the switch. This approach approximates the understanding that turns are traveled at a somewhat constant low speed, which can be reached relatively quickly, and that changes between turn types incur a further penalty for having to slow down to move the steering wheel and / or change gears. This cost model can be handled by having one reachable space 102 in the form of a cost volume parameterized by (x,y,θ) for each turn type (including, for example, turn radius and gear (forward / reverse)), where each turn type, along with the corresponding turn type as the last turn, holds the minimum cost required to reach its attitude state. This model can essentially operate in a four-dimensional state space, in which the fourth dimension holds the possible turn radius multiplied by two gears.

[0058] The path planner 110 may perform one turn at a time and proceed through an update step, updating the state space from a state having all the paths for the maximum n turns under consideration to a state having n+1 paths under consideration. For each turn type used in the (n+1)th turn, the trajectory assessor 150 may consider starting from any of the other turn types and first paying the cost of that transition. The minimum probability of this can be considered the minimum cost at which it is ready to start this type of turn from this attitude, and the path planner 110 only needs to consider the most efficient one. The path planner 110 can then process through the turns and reach the minimum cost after completing the (n+1)th turn in this turn type. In one example, processing through N turns with a K turn radius may require processing 2NK turns (for example, assuming that a separate processing step is required for a reverse). Therefore, before each turn, the trajectory assessor 150 may find the minimum transition from a 2K cost volume to start the turn. In aspects of this disclosure, this step may be performed at once for all 2K turn types, simultaneously reading the 2K volume and (2K) 2 The transition is calculated, the 2K minimum value is found, and that volume is written back simultaneously. This process upgrades the cost from "after" the turn to "before" the next turn, which has a 4NK volume equivalent to the memory access. Each turn also reads the cost back, reads free space, and writes out the new cost, resulting in an additional volume of 6NK memory accesses. This technique can result in a total volume of 10NK accesses.

[0059] The cost function can be further simplified; for example, evaluating the transition cost between turn types may be evaluated as the same regardless of the type of turn involved in the transition. In addition, the cost model may assume that the cost of moving through a turn is negligible compared to the transition cost. In this approach, the cost may be assessed based on the number of turns. While this approach greatly simplifies the calculations and may not consider some situations (e.g., two short turns with relatively short transitions rather than one long turn, or transitions between two turns of similar radii in the same gear being faster than a large change in steering or a gear change), this cost model can still provide a reasonable heuristic. In this simple model, a three-dimensional state space may be used instead of a four-dimensional state space. Now, a single state volume can hold whether or not a corresponding attitude can be reached in n turns, which is identical or similar to the approach reflected by pseudocode 830. In this simple cost model, the processing of each turn may be read from a common previous cost volume, and since the transition penalty is always paid, there is no need to remember which turn type was the last one. The occupancy space is read, a new cost is written for the first turn type, and then combined with the previous result using logical OR for subsequent turn types, which involves both reading and writing. This represents a total of 8NK-2N volume memory accesses. In addition, the cost volume may be represented by a single bit per cell (reachable or not) rather than 8, 16, or 32 bits, depending on the resolution of the cost function. The processing speed of many modern kernels is largely determined by the amount of memory access, and this method can have a small amount of memory access.

[0060] Referring next to Figure 9, Figure 9 illustrates an example of a computational flow graph 900 that may be used to process turns, in which reachability is encoded using a non-binary value, according to some embodiments of the present disclosure. The computational flow graph 900 may be suitable when the reachability evaluator 148 uses a more general cost function than the computational flow graph 800 of Figure 8.

[0061] Similar to computational flow graph 800, computational flow graph 900 can progress through N iterations, such as iterations 902A and 902B, considering K trajectory types each time (e.g., sharp left, gentle left, straight, gentle right, sharp right, etc.). Rather than using a shared reachable space, each iteration may include a cost update of the post-trajectory cost output from the previous iteration to generate the pre-trajectory cost for the iteration. For example, iteration 802B may include a cost update 820 of the post-trajectory cost output spaces 910A, 912A, 914A, 916A, and 918A from iteration 902A to generate the pre-trajectory cost output spaces 910B, 912B, 914B, 916B, and 918B for iteration 902B. In one or more embodiments, the post-trajectory cost may represent the minimum cost to reach an attitude ending in the corresponding trajectory type, and the pre-trajectory cost may represent the minimum cost to reach and be ready for the attitude without penalty for the corresponding turn trajectory. Next, the core turn processing of the iteration can convert the pre-orbit cost to the post-orbit cost, and the processing can be repeated. For example, iteration 902B may include converting the pre-orbit cost output spaces 910B, 912B, 914B, 916B, and 918B, as shown, to the corresponding post-orbit cost output spaces. In Figure 9, each vertical line represents an evaluation of a certain orbit type.

[0062] Figure 9 includes pseudocode 930 to show how the reachability evaluator 148 threads can be processed through M attitude cells of the trajectory using a core loop that operates with a more general cost function than pseudocode 830, where the cost c is involved. Here, the variable is no longer a bit vector in order to capture a non-binary cost value. Also, logical operations have been replaced with max / min operations. Obstacles can be represented by some maximum cost that cannot be changed, or they may be proportional to the size of the obstacle (for example, if the obstacle does not necessarily block the vehicle). In addition, the cost may increase by 1 for each step.

[0063] In one or more embodiments, iterations can process each trajectory in both forward and reverse directions. In the case of periodic iterations, which is an example of turns unless the trajectory is linear, this can also be processed in two periods. This is because knowing where to start the period is not a priori straightforward, and in the worst case, reachability from the last cell must propagate the entire second period, although certain heuristics can be developed to account for this. Thus, the trajectory can be processed in four sweeps using code that does not have long branching differences between threads to avoid thread divergence that hinders parallelism. However, most of this processing can be avoided, and much more parallelism can be demonstrated by dividing each trajectory into any number of sections that can be processed in parallel at the same time.

[0064] Using the disclosed technique for dividing a trajectory into parallel-processable sections makes it possible to avoid most of the costs incurred by performing the process through forward and backward periodic turns and over two cycles. In one or more embodiments, the reachability manager 122 divides the processing of one or more trajectories into independent parallel sections. The result is that the sections of turns may not be completely independent of each other, as reachability may undulate throughout the entire two-cycle trajectory, both forward and backward. The disclosed technique can address this dependency using a parallel reduction pattern that hierarchically collects the results, performs some minor processing on the collected results, and then returns the results back as input to the individual sections.

[0065] The results of one section may be independent of other sections, except for the reachability into the section at its beginning (or at its end, if processed backward). The disclosed techniques may calculate the forward and / or backward reachability into a section at its beginning or end, enabling independent processing of sections using such inputs. For small sections, the input occupancy space and reachability space may be loaded at once, and both forward and backward paths may be processed simultaneously, thereby halving global memory access. This is sometimes desirable because holding data in local registers or shared memory close to the processing core is usually much faster than accessing global memory.

[0066] To determine the forward and backward reachability into each section, a processing pass may be performed for each section (for example, in parallel) to compute the forward reachability (SRf) of starting within the section and leaving it in the forward direction, the backward reachability (SRb) of starting within the section and leaving it in the backward direction, and the section-free space (SF), which indicates whether the section as a whole is made up of free space.

[0067] The section-based forward and backward reachability resulting from this sweep is reachability that originates within the section. Because the section as a whole is made up of free space, it is not possible to locally detect reachability propagating from outside the section, such as reachability that enters the beginning of the section and passes through the entire section. This can be explained by computing the section-free space, thereby allowing such globally propagating effects to be determined in much smaller processing sweeps that operate on sections rather than individual cells.

[0068] The following is an example of pseudocode for a processing path, omitting backward paths that are mixed with forward paths: for(r=0,f=(~0),i=0;i <m;i++) { t = F[i]; / / Load free f &=t; / / Logical AND with free r &=t; / / Logical AND with free r |= Ri[i]; / / Logical OR with attainment level } SRf[s]=r;SF[s]=f; / / Store section output However, m represents the section length, and s is the index of a single section.

[0069] Similar to the processing paths within a section, another processing path may be used that operates within the section but instead uses section reachability and section-free space. The payload of this processing may loop through the section output twice (in the case of a periodic turn). The first loop can warm up any potential reachability to propagate. The second loop can complete the full propagation for the periodic turn and write the results back. The following is a pseudocode example of a processing path; only the forward path is shown, as the backward path is entirely similar: for(r=0,s=0;s <M / m;s++) { r &=SF[s]; / / Logical AND r |= SRf[s]; / / Logical OR } for(s=0;s <M / m;s++) { r &=SF[s]; / / Logical AND r |= SRf[s]; / / Logical OR SRf[s] = r; / / Store and return }.

[0070] This process passes the section output four times (for example, in parallel), but the total work in this step is only 4 / m times, and the processing decreases as the section gets longer (by selecting a larger m).

[0071] Another processing path for section processing can be nearly identical to the basic loop, with the modification of using section reachability as input instead of start reachability at zero. The following is a pseudocode example of the processing path, with the backward path omitted: for(r=SRf[(s-1)%(M / m)];i=0;i <m;i++) { r &=F[i]; / / Logical AND with free r |= Ri[i]; / / Logical OR with attainment level Ro[i] = r; / / Store and return }

[0072] The processing path that writes the result back may be changed to a logical OR with what is already there for all turn types except the first turn type. Section processing using a more general cost function works similarly, except that reachability r may correspond to the minimum cost of leaving or entering a section.

[0073] Using the parallel reduction pattern described herein may include 2 reads and 3 / m writes per cell in the attitude configuration space 112 for a processing path that includes calculating reachability starting within a section. A processing path that includes propagating the section output may include 8 / m reads and 2 / m writes. A processing path that uses section reachability as input may include 2 (or 3) reads and 1 write. Thus, the parallel reduction pattern can be achieved with 5+8 / m reads and 1+5 / m writes, or a total of 6+13 / m accesses. This can be compared to processing without sections, which may include 2 reads and 1 write × 3 passes and 3 reads and 1 write in the final pass, or a total of 9 reads, 4 writes, and 13 total memory accesses. In this way, section processing can save more than twice as much in memory access. In addition, by caching the free space or occupancy and the reachability input in shared memory between the first and third paths, and saving two read operations in the third path, resulting in 4+13 / m accesses, more than three times the memory access savings can be achieved, and further savings can be realized. The parallel reduction pattern technique also exhibits further parallelism because there is an M / m section in each turn that can function in parallel.

[0074] As described herein, the trajectory assessor 150 of the trajectory assessor 130 may use the reachable space annotated by the reachability assessor 148 over one or more iterations to identify and / or select a trajectory using the back tracer 152. In one or more embodiments, the back tracer 152 can back trace to find a trajectory realized in the minimum number of trajectories or turns (or, more generally, the lowest cost trajectory) to reach the objective. The back tracer 152 may be performed by the CPU and / or using parallel processing (e.g., using at least one GPU).

[0075] In at least one embodiment, the back tracer 152 may search for one or more cells from a set of targets for one or more attitudes, the final reachability output being set in reachable space. For example, the back tracer 152 may loop through the set of targets for attitudes, or use parallel reduction if parallelism is desired. If there are two or more cells with reachability sets, one or more attitudes may be selected by some preference function, such as based on proximity to several selected attitudes. If no such attitudes exist, this indicates that there is no N-turn route planning to reach the target set, in which case (N+1)-turn planning may be evaluated.

[0076] Once a cell is selected to begin backtracing, it can be assumed that after i iterations, the cell is reached and one of the turn types was reached from a different orientation reached in the previous iteration (in iteration i-1). Thus, the backtracer 152 can backtraverse all K turn types from that cell and find a cell somewhere along the way that is set to be reachable in the output reachable space from iteration i-1. Since turns may have repeatable coordinate definitions, turns can be backtraced exactly as they were evaluated. This backtracing of K turns can be done sequentially by the CPU or by parallel reduction. Since turns may have been used in the forward or backward direction, periodic ambiguity may exist. Ambiguity may be resolved by checking free space along the backtraced turn and stopping the backtracing if an obstacle is found (in embodiments where an obstacle can completely block the path).

[0077] In a more general cost function example, the criterion might be to find the cell with the minimum cost among cells whose cost reduction is equal to the cost of the turn in which the cost reduction is back-traceable. If there are two or more cells that satisfy the criterion, it is possible to select a cell using a heuristic, such as picking the cell that requires the shortest number of turns to reach. If the target set is reached in fewer than N turns, some of the initial back-trace steps find the reachable cell as the same cell from the previous iteration (because the steps for that turn were not required). The back-tracer 152 may determine that this criterion is satisfied and remove that turn from the path solution. Once a cell is found, the process can be repeated from that cell, after one iteration, until a reachable cell is found. The back-trace may end in the current orientation representing the reachable space before any turn, and may be performed in the current orientation to resolve periodic ambiguities.

[0078] Referring now to Figure 10, each block of Method 1000 and other methods described herein includes a computation process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. The methods may also be performed as computer-usable instructions stored on a computer storage medium. The methods may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, as an example, the methods relating to the route planner 110 in Figures 1A and 1B are described. However, these methods, including but not limited to those described herein, may be performed additionally or alternatively by any one system or within any combination of systems.

[0079] Figure 10 is a flowchart illustrating a method 1000 for determining a shifted attitude of an attitude configuration space to determine a path through the attitude configuration space, according to some embodiments of the present disclosure. Method 1000 includes, in block B1002, determining a shifted attitude based on translating the attitude of the attitude configuration space to generate a shifted trajectory that includes the shifted attitude. For example, the attitude translator 146 of the reachability manager 122 may determine a shifted attitude of an attitude configuration space 112 representing the attitude of an object (e.g., a vehicle 1300) in a given environment based on translating a set of attitudes in the attitude configuration space 112 corresponding to trajectories (e.g., turns 618 and 610 in Figure 6B) along at least one axis (e.g., the θ-axis) to generate a shifted trajectory that includes the shifted attitude (e.g., turns 618 and 610 in Figure 6C).

[0080] Method 1000 includes determining a path through attitude configuration space in block B1004, based at least on evaluating the reachability of a shifted attitude using parallel processing of shifted trajectories. For example, the path evaluator 130 may determine a path from the current attitude C in Figure 1A to the target attitude T in Figure 1A, based at least on an reachability evaluator 148 that evaluates the reachability of a shifted attitude from a first attitude using parallel processing of shifted trajectories (for example, according to pseudocode 830 or 930).

[0081] Referring now to Figure 11, which is a flowchart illustrating a method 1100 for translating a trajectory of an attitude in an attitude configuration space into a shifted trajectory in order to determine a path through the attitude configuration space, according to some embodiments of the present disclosure. Method 1100 includes, in block 1102, translating a trajectory of an attitude in an attitude configuration space into a shifted trajectory that includes one or more sections parallel to each other and to at least one axis of the attitude configuration space. For example, the attitude translator 146 may translate a trajectory formed by an attitude in an attitude configuration space 112 into a shifted trajectory that includes at least sections parallel to each other and to at least one axis of the attitude configuration space 112.

[0082] Method 1100 includes, in block B1104, processing at least the sections of the shifted trajectory in parallel along at least one axis to calculate an indicator of reachability associated with the shifted trajectory. For example, the reachability evaluator 148 may process at least the sections of the shifted trajectory in parallel along at least one axis to calculate an indicator of reachability associated with the shifted trajectory (for example, according to pseudocode 830 or 930).

[0083] Method 1100 includes determining a path through the attitude configuration space based on at least an indicator of reachability in block B1106. For example, the path evaluator 130 may determine a path through the attitude configuration space 112 based on at least an indicator of reachability.

[0084] Figure 12 is a flowchart illustrating a method 1200 for parallel evaluation of the reachability of trajectory sections according to some embodiments of the present disclosure. Method 1200 includes, in block B1202, calculating an indicator of reachability away from a first section of the trajectory sections of an attitude in attitude configuration space. For example, the reachability evaluator 148 may calculate an indicator of reachability away from a first section of the turn 618 section in Figures 6B and 6C, which is formed by the attitude in attitude configuration space 112.

[0085] Method 1200 includes, in block B1204, using an indicator of reachability to leave the first section to calculate an indicator of reachability to enter the second section of the section. For example, the reachability evaluator 148 may use an indicator of reachability to leave the first section to calculate an indicator of reachability to enter the second section of the section.

[0086] Method 1200 includes, in block B1206, calculating a reachability indicator for attitudes within the first section using reachability indicators that enter into the second section. For example, the reachability evaluator 148 may calculate a reachability indicator for attitudes within the first section (for example, with respect to the current attitude C in Figure 1A) using reachability indicators that enter into the second section. Method 1200 may be performed in parallel for multiple trajectories using parallel processing as part of a parallel reduction pattern.

[0087] Exemplary autonomous vehicle Figure 13A shows an exemplary autonomous vehicle 1300 according to some embodiments of the present disclosure. The autonomous vehicle 1300 (or referred to herein as "vehicle 1300") may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttles, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, trailer-mounted vehicles, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). The mobile vehicle 1300 may have the capability to function at one or more of the autonomous driving levels from Level 3 to Level 5. For example, depending on the embodiment, the vehicle 1300 may have the capability of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0088] The mobile vehicle 1300 may include components such as the mobile vehicle's chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. The mobile vehicle 1300 may include a propulsion system 1350, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 1350 may be connected to the drivetrain of the mobile vehicle 1300, which may include a transmission, in order to enable propulsion for the mobile vehicle 1300. The propulsion system 1350 may be controlled in response to receiving a signal from a throttle / accelerator 1352.

[0089] A steering system 1354, which may include a steering wheel, may be used to steer the vehicle 1300 (for example, along a desired course or route) when the propulsion system 1350 is operating (for example, when the vehicle is moving). The steering system 1354 may receive signals from the steering actuator 1356. The steering wheel may also be an option for fully automated (level 5) functionality.

[0090] The brake sensor system 1346 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1348 and / or the brake sensor.

[0091] The controller 1336, which may include one or more system-on-a-chip (SoC) 1304 (Figure 13C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 1300. For example, the controller can send signals to actuate the vehicle brakes via one or more brake actuators 1348, actuate the steering system 1354 via one or more steering actuators 1356, and actuate the propulsion system 1350 via one or more throttle / accelerators 1352. The controller 1336 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist the driver in driving the vehicle 1300. The controller 1336 may include a first controller 1336 for autonomous driving functions, a second controller 1336 for functional safety functions, a third controller 1336 for artificial intelligence functions (e.g., computer vision), a fourth controller 1336 for infotainment functions, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1336 may handle two or more of the aforementioned functions, and two or more controllers 1336 may handle a single function, and / or any combination thereof.

[0092] The controller 1336 can provide signals for controlling one or more components and / or systems of the mobile vehicle 1300 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example and without limitation, global navigation satellite system sensors 1358 (e.g., global positioning system sensors), RADAR sensors 1360, ultrasonic sensors 1362, LIDAR sensors 1364, inertial measurement unit (IMU) sensors 1366 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1396, stereo cameras 1368, wide-view cameras 1370 (e.g., fisheye cameras), infrared cameras 1372, surround cameras 1374 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1398, speed sensors 1344 (e.g., for measuring the speed of a moving vehicle 1300), vibration sensors 1342, steering sensors 1340, brake sensors (e.g., as part of a brake sensor system 1346), and / or other sensor types.

[0093] One or more of the controllers 1336 may receive inputs (represented, for example, by input data) from the instrument cluster 1332 of the mobile vehicle 1300 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 1334, an audible annunciator, a loudspeaker, and / or other components of the mobile vehicle 1300. The outputs may include information such as mobile vehicle velocity, speed, time, map data (e.g., HD map 1322 in Figure 13C), location data (e.g., the location of the mobile vehicle 1300, such as on the map), direction, the locations of other mobile vehicles (e.g., occupied grids), and information about objects and the status of objects as grasped by the controller 1336. For example, the HMI display 1334 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the moving vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting exit 34B within 3.22 km (2 miles), etc.).

[0094] The mobile vehicle 1300 further includes a network interface 1324 that can communicate over one or more networks using one or more wireless antennas 1326 and / or a modem. For example, the network interface 1324 may have the capability to communicate over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antennas 1326 can also enable communication between objects in the environment (e.g., mobile vehicles, mobile devices, etc.) using local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWANs) such as LoRaWAN, SigFox.

[0095] Figure 13B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 1300 of Figure 13A according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different positions on the mobile vehicle 1300.

[0096] The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the mobile vehicle 1300. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0097] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0098] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0099] A camera having a field of view that includes a portion of the environment in front of the mobile vehicle 1300 (e.g., a forward-facing camera) may be used for surround view to help identify the forward path and obstacles and, with the help of one or more controllers 1336 and / or control SoCs, to help provide information essential for generating an occupied grid and / or determining a preferred mobile vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as lane departure warning (LDW), autonomous cruise control (ACC), and / or traffic sign recognition.

[0100] Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 1370, which can be used to capture objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera is shown in Figure 13B, any number of wide-view cameras 1370 may be present in the mobile vehicle 1300. In addition, long-range cameras 1398 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 1398 may also be used for object detection and classification, as well as basic object tracking.

[0101] One or more stereo cameras 1368 may also be included in the forward-facing configuration. The stereo camera 1368 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the moving vehicle's environment, including distance estimates of all points in the image. An alternative stereo camera 1368 may include a compact stereo vision sensor that includes two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the moving vehicle to an object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 1368 may be used in addition to or instead of those described herein.

[0102] A camera having a field of view including a portion of the environment on the sides of the mobile vehicle 1300 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 1374 (e.g., four surround cameras 1374 as shown in Figure 13B) may be positioned on the mobile vehicle 1300. The surround cameras 1374 may include wide-view cameras 1370, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and on the sides of the mobile vehicle. In an alternative configuration, the mobile vehicle may use three surround cameras 1374 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera.

[0103] A camera having a field of view that includes a portion of the environment behind the mobile vehicle 1300 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 1398, stereo camera 1368), infrared cameras 1372, etc., as described herein.

[0104] Figure 13C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 1300 of Figure 13A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0105] Each component, feature, and system of the mobile vehicle 1300 in Figure 13C is illustrated as being connected via a bus 1302. Bus 1302 may include a Controller Area Network (CAN) data interface (or, referred to as the "CAN bus"). CAN may also be a network within the mobile vehicle 1300 used to assist in the control of various features and functions of the mobile vehicle 1300, such as the operation of brakes, acceleration, steering, and windshield wipers. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other mobile vehicle status indicators. The CAN bus may be ASIL B compliant.

[0106] Bus 1302 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or as an alternative to, a CAN bus. In addition, a single line is used to represent bus 1302, but this is not intended to limit it. There may be any number of buses 1302, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1302 may be used to perform different functions and / or for redundancy. For example, a first bus 1302 may be used for collision avoidance and a second bus 1302 may be used for operation control. In any example, each bus 1302 may communicate with any of the components of the mobile vehicle 1300, and two or more buses 1302 may communicate with the same component. In some examples, each SoC 1304, each controller 1336, and / or each computer within the mobile vehicle may have access to the same input data (e.g., input from sensors in the mobile vehicle 1300) and may be connected to a common bus such as a CAN bus.

[0107] The mobile vehicle 1300 may include one or more controllers 1336, such as those described herein with respect to Figure 13A. The controllers 1336 may be used for a variety of functions. The controllers 1336 may be connected to any of the various other components and systems of the mobile vehicle 1300 and may be used for the control of the mobile vehicle 1300, the artificial intelligence of the mobile vehicle 1300, infotainment for the mobile vehicle 1300, and / or the like.

[0108] The mobile vehicle 1300 may include a system-on-a-chip (SoC) 1304. The SoC 1304 may include a CPU 1306, a GPU 1308, a processor 1310, a cache 1312, an accelerator 1314, a data store 1316, and / or other components and features not shown. The SoC 1304 may be used to control the mobile vehicle 1300 in various platforms and systems. For example, the SoC 1304 may be coupled in a system (e.g., the system of the mobile vehicle 1300) having an HD map 1322 that can obtain map refreshes and / or updates from one or more servers (e.g., server 1378 in Figure 13D) via a network interface 1324.

[0109] The CPU 1306 may include a CPU cluster or CPU complex (also referred to as "CCPLEX"). The CPU 1306 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1306 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1306 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 1306 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the CPU 1306 clusters to be active at any given time.

[0110] The CPU1306 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU1306 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode.

[0111] The GPU1308 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU1308 may be programmable and efficient for parallel workloads. In some embodiments, the GPU1308 may be able to use an enhanced tensor instruction set. The GPU1308 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU1308 may include at least eight streaming microprocessors. The GPU1308 may be able to use a Computation Application Programming Interface (API). In addition, the GPU1308 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0112] The GPU1308 can be power-optimized for optimal performance in automotive and embedded use cases. For example, the GPU1308 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be a limitation, and the GPU1308 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance.

[0113] In some examples, the GPU1308 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0114] The GPU1308 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support can be used to allow the GPU1308 to directly access the CPU1306 page table. In such examples, when the GPU1308 memory management unit (MMU) experiences a miss, an address translation request can be sent to the CPU1306. In response, the CPU1306 can examine its page table for virtual-to-real-address mapping and send the translation back to the GPU1308. As such, unified memory technology can enable a single, unified virtual address space for both the CPU1306 and GPU1308 memory, thereby simplifying GPU1308 programming and porting of applications to the GPU1308.

[0115] In addition, the GPU1308 may include an access counter that can record how often the GPU1308 accesses the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0116] The SoC1304 may include any number of caches 1312, including those described herein. For example, cache 1312 may include an L3 cache available to both the CPU 1306 and the GPU 1308 (e.g., connected to both the CPU 1306 and the GPU 1308). Cache 1312 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, although a smaller cache size may be used.

[0117] The SoC1304 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 1300 (for example, a processing DNN). In addition, the SoC1304 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU1306 and / or GPU1308.

[0118] The SoC1304 may include one or more accelerators 1314 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC1304 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU1308 and to offload some of the tasks of the GPU1308 (e.g., to free up more cycles of the GPU1308 to perform other tasks). As an example, accelerator 1314 may be used for target workloads that are sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0119] Accelerator 1314 (e.g., hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA can provide more performance per millisecond than a general-purpose GPU and significantly exceed the performance of a CPU. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0120] DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and mobile vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0121] DLA can perform any function of GPU1308, and by using inference accelerators, for example, a designer can target either DLA or GPU1308 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU1308 and / or other accelerators 1314.

[0122] The accelerator 1314 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may include, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0123] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may contain any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0124] DMA can enable PVA components to access system memory independent of the CPU 1306. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0125] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0126] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can run the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or even run different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0127] The accelerator 1314 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 1314. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, and not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB).

[0128] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0129] In some embodiments, the SoC1304 may include a real-time ray tracing hardware accelerator, such as the one described in Patent Document 1, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0130] The accelerator 1314 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA may also be a programmable vision accelerator that can be used in critical processing stages in ADAS and autonomous vehicles. The PVA's capabilities are suitable for areas of algorithms requiring predictable processing at low power and low latency. In other words, the PVA performs well in semi-high density or high density typical computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, because the PVA is efficient in object detection and integer computation, in relation to a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms.

[0131] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. While semi-global matching-based algorithms may be used in some examples, this is not intended to be a limitation. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0132] In some applications, PVA can be used to perform high-density optical flow by processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR data. In other applications, PVA is used for flight depth processing, for example, by processing raw flight data to provide processed flight data.

[0133] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the moving vehicle to automatically apply the emergency brakes, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for the AEB. DLA can run a neural network that devolves the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), object orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation acquired from the neural network and / or other sensors (e.g., LIDAR sensor 1364 or RADAR sensor 1360), and others.

[0134] The SoC1304 may include a data store 1316 (for example, memory). The data store 1316 may also be the on-chip memory of the SoC1304 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 1316 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 1312 may comprise an L2 or L3 cache 1312. References to the data store 1316 may include references to memory associated with the PVA, DLA, and / or other accelerators 1314, as described herein.

[0135] The SoC1304 may include one or more processors 1310 (e.g., embedded processors). The processors 1310 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC1304 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, assistance with system low-power state transitions, management of SoC1304 thermal and temperature sensors, and / or management of SoC1304 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC1304 may use the ring oscillators to detect the temperatures of the CPU 1306, GPU 1308, and / or accelerator 1314. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine and place the SoC1304 into a lower power state and / or put the mobile vehicle 1300 into chauffeur safe shutdown mode (for example, safely shut down the mobile vehicle 1300).

[0136] The processor 1310 may further include a set of integrated processors capable of performing the functions of an audio processing engine. The audio processing engine may also be an audio subsystem enabling full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0137] The processor 1310 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0138] The processor 1310 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations.

[0139] The processor 1310 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0140] The processor 1310 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0141] The processor 1310 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 1370, the surround camera 1374, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and appropriately respond to in-cabin events. The in-cabin system can perform lip-reading to activate cellular services and make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0142] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0143] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU1308 is not required to continuously render new surfaces. Even when the GPU1308 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU1308 to improve performance and responsiveness.

[0144] The SoC1304 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC1304 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0145] The SoC1304 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC1304 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., LiDAR sensor 1364, RADAR sensor 1360, etc., which may be connected via Ethernet®), data from bus 1302 (e.g., speed of vehicle 1300, steering wheel position, etc.), and data from GNSS sensor 1358 (connected, for example, via Ethernet® or CAN bus). The SoC1304 may further include a dedicated high-performance mass storage controller which may include its own DMA engine and which may be used to free up CPU 1306 from routine data management tasks.

[0146] The SoC1304 may also be an inter-terminal platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC1304 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 1314 is coupled with the CPU 1306, the GPU 1308, and the data store 1316 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0147] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0148] In contrast to conventional systems, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU1320) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs and passing that semantic understanding to a route planning module running on the CPU complex.

[0149] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network, informing the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU1308, for example.

[0150] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of the vehicle 1300. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and, in security mode, to stop the vehicle when the owner leaves the vehicle. In this way, the SoC 1304 provides security against theft and / or vehicle hijacking.

[0151] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 1396. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 1304 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In a preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the moving vehicle is operating, as identified by GNSS sensor 1358. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, the control program may be used, with the assistance of the ultrasonic sensor 1362, to perform an emergency vehicle safety routine, such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle, until the emergency vehicle has passed.

[0152] The vehicle may include a CPU 1318 (e.g., a separate CPU, or dCPU) which can be connected to the SoC 1304 via a high-speed interconnect (e.g., PCIe). The CPU 1318 may include, for example, an x86 processor. The CPU 1318 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 1304, and / or monitoring the status and condition of the controller 1336 and / or the infotainment SoC 1330.

[0153] The mobile vehicle 1300 may include a GPU 1320 (e.g., a separate GPU, or dGPU) which can be connected to the SoC 1304 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 1320 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from the mobile vehicle 1300's sensors (e.g., sensor data).

[0154] Mobile vehicle 1300 may further include a network interface 1324 which may include one or more wireless antennas 1326 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth® antennas). The network interface 1324 may be used to enable wireless connectivity to a cloud over the Internet (e.g., with server 1378 and / or other network devices), to other mobile vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other mobile vehicles, a direct link may be established between two mobile vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a mobile vehicle-to-mobile communication link. The mobile vehicle-to-mobile communication link may provide mobile vehicle 1300 information about mobile vehicles in close proximity to mobile vehicle 1300 (e.g., mobile vehicles in front of, beside, and / or behind mobile vehicle 1300). This function may also be part of the joint adaptive cruise control function of the mobile vehicle 1300.

[0155] The network interface 1324 may include an SoC that provides modulation and demodulation functions and enables the controller 1336 to communicate over a wireless network. The network interface 1324 may include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. Frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth®, Bluetooth® LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0156] The mobile unit 1300 may further include a data store 1328 which may include storage outside the chip (for example, outside the SoC 1304). The data store 1328 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0157] The vehicle 1300 may further include a GNSS sensor 1358. The GNSS sensor 1358 (e.g., GPS, an assisted GPS sensor, a differential GPS (DGPS) sensor, etc.) assists in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 1358 may be used, including, but not limited to, a GPS using a USB connector with Ethernet® to a serial (RS-232) bridge.

[0158] The mobile vehicle 1300 may further include a RADAR sensor 1360. The RADAR sensor 1360 may be used by the mobile vehicle 1300 for long-range mobile vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 1360 may use CAN and / or bus 1302 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 1360) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 1360 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used.

[0159] The RADAR sensor 1360 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 1360 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include monostatic multimodal RADARs having multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record around the moving vehicle 1300 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of moving vehicles entering or leaving the lane of the moving vehicle 1300.

[0160] As an example, a medium-range RADAR system may include a range of up to 1360 m (forward) or 80 m (rear) and a field of view of up to 42 degrees (forward) or 1350 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the moving vehicle.

[0161] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0162] The mobile vehicle 1300 may further include ultrasonic sensors 1362. Positioned on the front, rear, and / or sides of the mobile vehicle 1300, the ultrasonic sensors 1362 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 1362 may be used, and different ultrasonic sensors 1362 may be used for detection of different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 1362 may operate at a functional safety level of ASIL B.

[0163] The mobile vehicle 1300 may include a LiDAR sensor 1364. The LiDAR sensor 1364 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 1364 may also have a functional safety level of ASIL B. In some examples, the mobile vehicle 1300 may include multiple LiDAR sensors 1364 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0164] In some examples, the LIDAR sensor 1364 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 1364 may have an advertised range of approximately 1300m, for example, with an accuracy of 2cm to 3cm and support for 1300Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 1364 may be used. In such examples, the LIDAR sensor 1364 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a mobile vehicle 1300. In such examples, the LIDAR sensor 1364 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 1364 may be configured for a horizontal field of view between 45 and 135 degrees.

[0165] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around a moving vehicle up to approximately 200m. The flash LiDAR unit includes receptors that record the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the moving vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the moving vehicle 1300. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1364 can be made less susceptible to motion blur, vibration, and / or shock.

[0166] The mobile vehicle may further include an IMU sensor 1366. In some examples, the IMU sensor 1366 may be positioned in the center of the rear axle of the mobile vehicle 1300. The IMU sensor 1366 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 1366 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 1366 may include an accelerometer, a gyroscope, and a magnetometer.

[0167] In some embodiments, the IMU sensor 1366 may be implemented as a miniature, high-performance GPS-Aided Inertial Navigation System (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 1366 may enable the moving vehicle 1300 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS to the IMU sensor 1366. In some embodiments, the IMU sensor 1366 and the GNSS sensor 1358 may be combined in a single integrated unit.

[0168] The mobile vehicle may include a microphone 1396 placed inside and / or around the mobile vehicle 1300. The microphone 1396 may, among other things, be used for emergency vehicle detection and identification.

[0169] The mobile vehicle may further include any number of camera types, including a stereo camera 1368, a wide-view camera 1370, an infrared camera 1372, a surround camera 1374, a long-range and / or medium-range camera 1398, and / or other camera types. The cameras may be used to capture image data around the entire exterior surface of the mobile vehicle 1300. The type of camera used will depend on the embodiment and requirements of the mobile vehicle 1300, and any combination of camera types may be used to achieve the required coverage around the mobile vehicle 1300. In addition, the number of cameras may vary depending on the embodiment. For example, the mobile vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 13A and 13B.

[0170] The mobile vehicle 1300 may further include a vibration sensor 1342. The vibration sensor 1342 can measure vibrations of components of the mobile vehicle, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 1342 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0171] The mobile vehicle 1300 may include an ADAS system 1338. In some examples, the ADAS system 1338 may include a System of Control (SoC). The ADAS system 1338 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0172] The ACC system may utilize a radar sensor 1360, a lithium-ion sensor 1364, and / or a camera. The ACC system may include longitudinal ACC and / or transverse ACC. Longitudinal ACC monitors and controls the distance of vehicle 1300 to the vehicle immediately in front of it and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Transverse ACC performs distance maintenance and advises vehicle 1300 to change lanes when necessary. Transverse ACC is related to other ADAS applications such as LCA and CWS.

[0173] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through the network interface 1324 and / or wireless antenna 1326, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., the vehicle immediately in front of vehicle 1300, in the same lane as vehicle 1300), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about the vehicle ahead of vehicle 1300, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0174] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 1360, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual warnings, vibration, and / or quick brake pulses.

[0175] An AEB system can detect an impending forward collision with another moving vehicle or other object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or impending collision braking.

[0176] The LDW system warns the driver when the moving vehicle 1300 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0177] The LKA system is a modified form of the LDW system. The LKA system provides steering input or braking to correct the moving vehicle 1300 when it begins to drift out of its lane.

[0178] The BSW system detects and warns the driver of a moving vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or radar sensor 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0179] The RCTW system can provide visual, audible, and / or haptic notifications when an object is detected outside the range of the rear camera while the vehicle 1300 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 1360 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0180] Conventional ADAS systems warn the driver and allow the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have sometimes tended to produce misjudgment results that, while not usually catastrophic, can be troubling and distracting to the driver. In the autonomous vehicle 1300, however, if the results are contradictory, the vehicle 1300 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1336 or the second controller 1336). For example, in some embodiments, the ADAS system 1338 may also be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. The backup computer rationality monitor can run a variety of redundant software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 1338 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0181] In some implementations, a primary computer may be configured to provide a supervising MCU with a reliability score indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0182] The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer provides a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based FCW system, the neural network in the supervisory MCU can learn when the FCW identifies metal objects that are not actually dangerous, such as sewer grates or manhole covers that trigger an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments involving a neural network running on the supervisory MCU, the supervisory MCU may include at least one DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC1304.

[0183] In other examples, ADAS system 1338 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error.

[0184] In some examples, the output of the ADAS system 1338 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 1338 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0185] The mobile vehicle 1300 may further include an infotainment SoC 1330 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more separate components. The infotainment SoC 1330 may include a combination of hardware and software that can be used to provide the mobile vehicle 1300 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other mobile vehicle-related information). For example, the infotainment SoC 1330 may also include wireless, disc player, navigation system, video player, USB and Bluetooth® connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 1334, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1330 may be further used to provide information (for example, visual and / or audible) to the user of the vehicle, such as information from the ADAS system 1338, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0186] The infotainment SoC 1330 may include GPU functionality. The infotainment SoC 1330 can communicate with other devices, systems, and / or components of the vehicle 1300 via bus 1302 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 1330 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 1336 (e.g., the primary and / or backup computer of the vehicle 1300) failure. In such examples, the infotainment SoC 1330 can put the vehicle 1300 into a chauffeur-safe stop mode as described herein.

[0187] The mobile vehicle 1300 may further include an instrument cluster 1332 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1332 may include a controller and / or a supercomputer (e.g., a separate controller or supercomputer). The instrument cluster 1332 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 1330 and the instrument cluster 1332. In other words, the instrument cluster 1332 may be included as part of the infotainment SoC 1330, and vice versa.

[0188] Figure 13D is a system diagram of communication between the cloud-based server of Figure 13A and an exemplary autonomous vehicle 1300 according to some embodiments of the present disclosure. System 1376 may include a server 1378, a network 1390, and a mobile vehicle including the mobile vehicle 1300. Server 1378 may include a plurality of GPUs 1384(A) to 1384(H) (collectively referred to herein as GPU 1384), PCIe switches 1382(A) to 1382(H) (collectively referred to herein as PCIe switch 1382), and / or CPUs 1380(A) to 1380(B) (collectively referred to herein as CPU 1380). The GPUs 1384, CPUs 1380, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 1388 and / or PCIe connections 1386 developed by NVIDIA. In some examples, the GPU1384 is connected via NVLink and / or NVSwitch SoCs, and the GPU1384 and PCIe switch 1382 are connected via PCIe interconnects. Eight GPU1384s, two CPU1380s, and two PCIe switches are illustrated, but this is not intended to be an limitation. Depending on the embodiment, each server 1378 may contain any number of GPU1384s, CPU1380s, and / or PCIe switches. For example, server 1378 may contain eight, sixteen, thirty-two, and / or more GPU1384s, respectively.

[0189] Server 1378 can receive image data from mobile vehicles via network 1390, representing images showing unexpected or altered road conditions, such as recently commenced road construction. Server 1378 can transmit map information 1394, including information about traffic and road conditions, to mobile vehicles via network 1390, including information about neural networks 1392, updated neural networks 1392, and / or map information 1394. Updates to map information 1394 may include updates to HD maps 1322, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural networks 1392, updated neural networks 1392, and / or map information 1394 may have arisen from new training and / or experience represented in data received from any number of mobile vehicles in the environment, and / or based on training performed in a data center (for example, using server 1378 and / or other servers).

[0190] Server 1378 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a mobile device and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, it may be used by the mobile vehicle (for example, transmitted to the mobile vehicle via network 1390), and / or the machine learning model may be used by server 1378 to remotely monitor the mobile vehicle.

[0191] In some examples, Server 1378 can receive data from a mobile vehicle and apply it to a state-of-the-art real-time neural network for real-time intelligent inference. Server 1378 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 1384, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 1378 may include deep learning infrastructure that uses only CPU-powered data centers.

[0192] The deep learning infrastructure of server 1378 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware in mobile vehicle 1300. For example, the deep learning infrastructure can receive periodic updates from mobile vehicle 1300, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by mobile vehicle 1300, and if the results do not match and the infrastructure concludes that the AI ​​in mobile vehicle 1300 is not functioning properly, server 1378 can send a signal to mobile vehicle 1300 instructing the mobile vehicle's fail-safe computer to infer control, notify passengers, and complete a safe parking operation.

[0193] For inference, server 1378 may include GPU 1384 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when performance is not a major requirement, a server powered by a CPU, FPGA, and other processors may be used for inference.

[0194] Exemplary computing devices Figure 14 is a block diagram of an example of a computing device 1400 suitable for use in implementing some embodiments of the present disclosure. The computing device 1400 may include an interconnection system 1402 that indirectly or directly connects the following devices: memory 1404, one or more central processing units (CPUs) 1406, one or more graphics processing units (GPUs) 1408, a communication interface 1410, input / output (I / O) ports 1412, input / output components 1414, a power supply unit 1416, one or more presentation components 1418 (e.g., a display), and one or more logical units 1420. In at least one embodiment, the computing device 1400 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 1408 may include one or more vGPUs, one or more of the CPUs 1406 may include one or more vCPUs, and / or one or more of the logical units 1420 may include one or more virtual logical units. As such, the computing device 1400 may include individual components (e.g., an entire GPU dedicated to the computing device 1400), virtual components (e.g., a portion of a GPU dedicated to the computing device 1400), or a combination thereof.

[0195] The various blocks in Figure 14 are shown connected by lines via the interconnection system 1402, but this is not intended to be restrictive and is simply for clarity. For example, in some embodiments, a presentation component 1418, such as a display device, could be considered an I / O component 1414 (for example, if the display is a touchscreen). As another example, the CPU 1406 and / or GPU 1408 could include memory (for example, memory 1404 could represent a storage device in addition to the memory of the GPU 1408, CPU 1406, and / or other components). In other words, the computing devices in Figure 14 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 14 and are therefore not distinguished.

[0196] The interconnection system 1402 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1402 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 1406 may be directly connected to the memory 1404. Furthermore, the CPU 1406 may be directly connected to the GPU 1408. Where direct or point-to-point connections exist between components, the interconnection system 1402 may include PCIe links for implementing the connections. In these examples, the PCI bus does not need to be included in the computing device 1400.

[0197] Memory 1404 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 1400. The computer-readable media may include both volatile and non-volatile media, and removable and non-removable media. For example, but not limited to, the computer-readable media may include computer storage media and communication media.

[0198] Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1404 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 1400. In this specification, computer storage media does not include signals themselves.

[0199] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. For example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0200] The CPU 1406 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1400 to execute one or more of the methods and / or processes described herein. The CPU 1406 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads simultaneously. The CPU 1406 may include any type of processor, and depending on the type of computing device 1400 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 1400, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1400 may include one or more CPUs 1406 in one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0201] In addition to or instead of the CPU 1406, the GPU 1408 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1400 to execute one or more of the methods and / or processes described herein. One or more of the GPUs 1408 may be an integrated GPU (for example, with one or more of the CPUs 1406, and / or one or more of the GPUs 1408 may be discrete GPUs). In embodiments, one or more of the GPUs 1408 may be coprocessors of one or more of the CPUs 1406. The GPUs 1408 may be used by the computing device 1400 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPUs 1408 may be used for GPU-based general-purpose computing (GPGPU). It may be used for a GPU. The GPU 1408 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 1408 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU 1406 received via the host interface). The GPU 1408 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of memory 1404. The GPU1408 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU1408 can generate pixel data or GPGPU data for different parts of an output or different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs.

[0202] In addition to or instead of the CPU 1406 and / or GPU 1408, the logical unit 1420 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 1400 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 1406, GPU 1408, and / or the logical unit 1420 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logical units 1420 may be part of and / or integrated with one or more of the CPU 1406 and / or GPU 1408, and / or one or more of the logical units 1420 may be discrete components of the CPU 1406 and / or GPU 1408 or otherwise external to them. In embodiments, one or more of the logical units 1420 may be coprocessors of one or more of the CPU 1406 and / or one or more of the GPU 1408.

[0203] Examples of logical unit 1420 include one or more processing cores and / or components thereof, such as a Data Processing Unit (DPU), Tensor Core (TC), Tensor Processing Unit (TPU), Pixel Visual Core (PVC), Vision Processing Unit (VPU), Graphics Processing Cluster (GPC), Texture Processing Cluster (TPC), Streaming Multiprocessor (SM), Tree Traversal Unit (TTU), Artificial Intelligence Accelerator (AIA), and Deep Learning Accelerator (DLA). This includes an Accelerator, a Logical Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating-Point Unit (FPU), input / output (I / O) elements, a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) element, and / or similar.

[0204] The communication interface 1410 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1400 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1410 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 1420 and / or the communication interface 1410 may include one or more data processing units (DPUs) to transmit data received via the network and / or through the interconnection system 1402 directly to one or more GPUs 1408 (e.g., their memory).

[0205] I / O port 1412 can enable the computing device 1400 to be logically connected to other devices, including I / O components 1414, presentation components 1418, and / or other components, some of which can be built into (e.g., integrated into) the computing device 1400. Exemplary I / O components 1414 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 1414 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 1400 (as described in more detail below). The computing device 1400 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 1400 may include accelerometers or gyroscopes that enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 1400 to render immersive augmented reality or virtual reality.

[0206] The power supply device 1416 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 1416 can provide power to the computing device 1400 to enable the components of the computing device 1400 to operate.

[0207] The presentation component 1418 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 1418 can receive data from other components (e.g., GPU 1408, CPU 1406, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0208] Exemplary data center Figure 15 shows an exemplary data center 1500 that may be used in at least one embodiment of the present disclosure. The data center 1500 may include a data center infrastructure layer 1510, a framework layer 1520, a software layer 1530, and / or an application layer 1540.

[0209] As shown in Figure 15, the data center infrastructure layer 1510 may include a resource orchestrator 1512, grouped computing resources 1514, and node computing resources ("node CRs") 1516(1) to 1516(N), where "N" represents any integer or natural number. In at least one embodiment, the node CRs 1516(1) to 1516(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules. In some embodiments, one or more nodes CR1516(1) to 1516(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1516(1) to 15161(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more nodes CR1516(1) to 1516(N) may correspond to a virtual machine (VM).

[0210] In at least one embodiment, the grouped computing resources 1514 may include a separate group of nodes CR1516 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CR1516 within the grouped computing resources 1514 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR1516, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0211] The resource orchestrator 1512 can configure or otherwise control one or more nodes CR1516(1) to 1516(N) and / or grouped computing resources 1514. In at least one embodiment, the resource orchestrator 1512 may include a software design infrastructure ("SDI") management entity of the data center 1500. The resource orchestrator 1512 may include hardware, software, or any combination thereof.

[0212] In at least one embodiment, as shown in Figure 15, the framework layer 1520 may include a job scheduler 1533, a configuration manager 1534, a resource manager 1536, and / or a distributed file system 1538. The framework layer 1520 may include a framework to support the software 1532 of the software layer 1530 and / or one or more applications 1542 of the application layer 1540. The software 1532 or applications 1542 may each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1520 may also be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 1538 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1533 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 1500. The configuration manager 1534 may have the ability to configure different layers, for example, a software layer 1530 and a framework layer 1520 including Spark and a distributed file system 1538 to support large-scale data processing. The resource manager 1536 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 1538 and the job scheduler 1533. In at least one embodiment, the clustered or grouped computing resources may include computing resources 1514 grouped in the data center infrastructure layer 1510. The resource manager 1536 can coordinate with the resource orchestrator 1512 to manage these mapped or allocated computing resources.

[0213] In at least one embodiment, the software 1532 included in the software layer 1530 may include software used by at least a portion of nodes CR1516(1) to 1516(N), grouped computing resources 1514, and / or the distributed file system 1538 of the framework layer 1520. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0214] In at least one embodiment, the application 1542 included in the application layer 1540 may include one or more types of applications used by at least a portion of the nodes CR1516(1) to 1516(N), the grouped computing resources 1514, and / or the distributed file system 1538 of the framework layer 1520. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0215] In at least one embodiment, any of the configuration manager 1534, resource manager 1536, and resource orchestrator 1512 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 1500 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center.

[0216] The data center 1500 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to the data center 1500. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to the data center 1500 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0217] In at least one embodiment, the data center 1500 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) for performing training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0218] Exemplary network environment A network environment suitable for use in implementing the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 1400 in Figure 14, for example, each device may include similar components, features, and / or functionalities of the computing device 1400. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 1500, examples of which are further detailed herein with respect to Figure 15.

[0219] Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may include multiple networks, or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity.

[0220] Compatible network environments may include one or more peer-to-peer network environments (in which case servers may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices.

[0221] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In the embodiment, one or more client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”).

[0222] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may delegate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0223] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 1400 described herein with respect to Figure 14. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0224] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0225] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0226] The subject matter of this disclosure is described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the claimed subject matter may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, is explicitly stated.

Claims

1. A step of determining a shifted attitude of an attitude configuration space representing the attitude of a machine in an operating environment, at least on the basis of translating the coordinates of a set of attitudes corresponding to a trajectory in the attitude configuration space along at least one axis, wherein the translation includes accessing the coordinates using an access pattern that performs shifted memory access in order to adjust the coordinates to correspond to the shifted version of the trajectory. A step of determining the path from the first attitude to the second attitude, at least based on evaluating the reachability of the shifted attitude from the first attitude using parallel processing of the shifted version of the trajectory, The steps include: performing one or more calculations on the machine based on the determined path; Methods that include...

2. The method according to claim 1, wherein the shifted versions of the trajectory are parallel to each other along the at least one axis.

3. The method according to claim 1, wherein the at least one axis includes an axis that represents the orientation of the machine in the operating environment.

4. The method according to claim 1, wherein each of the trajectories includes a turn having a constant turn radius over the entire trajectory.

5. The method according to claim 1, wherein the parallel processing includes the step of evaluating a first subset of the shifted attitudes of the shifted trajectories from the shifted versions of the trajectory in parallel with a second subset of the shifted attitudes of the shifted trajectory.

6. The method according to claim 1, wherein the parallel processing includes the step of evaluating a first subset of the shifted attitudes of a first shifted trajectory among the shifted versions of the trajectory in parallel with a second subset of the shifted attitudes of a second shifted trajectory among the shifted versions of the trajectory.

7. The method according to claim 1, wherein the reachability is processed by a processor thread as a bit vector, and each bit of the bit vector corresponds to the attitude of the shifted attitude and the trajectory of the shifted version of the trajectory.

8. The method according to claim 1, wherein the translation includes a translational shift of a vector operation for memory access.

9. The method according to claim 1, wherein the translation of the coordinates of the set of attitudes comprises the steps of generating a deformed attitude configuration space in memory from the attitude configuration space, which includes the shifted attitude of the shifted version of the trajectory; and performing the evaluation of the reachability of the shifted attitude using the deformed attitude configuration space in memory.

10. One or more processors, When executed by the aforementioned one or more processors, one or more memory devices store instructions that cause the aforementioned one or more processors to execute the method. A system comprising the method, A step of translating the coordinates of a machine's trajectory in attitude configuration space to a shifted version of the trajectory in memory, wherein the shifted version of the trajectory includes at least sections of the machine's trajectory that are parallel to each other and to at least one axis of the attitude configuration space, and the translation includes accessing the coordinates using an access pattern that performs shifted memory access to adjust the coordinates to correspond to the shifted version of the trajectory. A step of processing at least the section of the shifted version of the trajectory in parallel along at least one axis to calculate an indicator of reachability associated with the shifted version of the trajectory, The steps include determining the path through the attitude configuration space based at least on the indicator of reachability, The steps include: performing one or more calculations corresponding to the behavior of the machine based on the determined path; A system that includes this.

11. The system according to claim 10, wherein the attitude configuration space is parameterized by at least (x, y, θ), where (x, y) represents the location of the machine in a two-dimensional (2D) plane, and θ represents the azimuth angle of the machine at the location.

12. The system according to claim 10, wherein the reachability indicator includes costs stored in one or more cost spaces parameterized by at least the attitude of the machine in the attitude configuration space, and the step of determining the path includes a step of backtracing using the one or more cost spaces from a first attitude of the path to a second attitude of the path.

13. A step of determining the free space of the posture of the machine body having a posture, at least based on collision texting the body of the machine having a posture with one or more obstacles, A step of storing indicators of the free space in at least one or more occupancy spaces parameterized by the attitude of the attitude configuration space, wherein the reachability indicators are at least based on evaluating the indicators of the free space in the one or more occupancy spaces against the shifted version of the trajectory. The system according to claim 10, further comprising:

14. The step of determining the aforementioned course is, In a first iteration corresponding to moving the shifted version of the trajectory for the first time, the steps include calculating a first set of indicators of reachability, A step of calculating a second set of reachability indicators from a first set of reachability indicators in a second iteration corresponding to moving the shifted version of the trajectory a second time; The system according to claim 10, including the following:

15. The aforementioned system Control systems for autonomous or semi-autonomous machines, Cognitive systems for autonomous or semi-autonomous machines A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The system according to claim 10, which is included in at least one of the following.

16. The reachability of a shifted version of a trajectory by an object with a shifted attitude is evaluated using parallel processing of the section of the shifted version of the trajectory, at least on the basis of accessing the coordinates using an access pattern that performs shifted memory access to adjust the attitude coordinates of the attitude configuration space corresponding to the trajectory to correspond to the shifted version of the trajectory, at least on the basis of translating the coordinates along at least one axis, Based at least on the reachability of the shifted posture, the path of the object from the first posture in the posture configuration space to the second posture in the posture configuration space is determined. A processor having one or more circuits for that purpose.

17. The parallel processing in the aforementioned section With respect to the first section of the section, the steps include calculating the reachability indicator that starts within the first section and leaves the first section, A step of calculating the first indicator of reachability of entering the second section of the section using the indicator of reachability of leaving the first section, A step of calculating a second indicator of reachability for one or more of the shifted postures within the first section from the first posture, using the first indicator of reachability for entering the second section. The processor according to claim 16, including the processor described in claim 16.

18. The processor according to claim 17, wherein the second indicator of reachability for one or more of the shifted postures includes one or more costs associated with the object reaching one or more of the shifted postures.

19. The processor according to claim 16, wherein the reachability of the shifted posture is evaluated with respect to the first posture.

20. The processor according to claim 16, wherein the trajectory includes a circle in the base plane of an environment formed by turns having a constant turn radius over the entire trajectory.

Images