Adjusted Component Interface Control Framework

Abstract

To provide a coordinated component interface control framework.SOLUTION: A framework may periodically reproduce behavior of a data processing pipeline. The framework may include a controller that controls input to, output from, and / or execution of a component of the pipeline (e.g., via an interface). The framework may also tune performance of the pipeline and / or enable parallel processing of the pipeline, even across different machines, while preserving the ability to periodically reproduce behavior of the pipeline. The interface may facilitate replacing a component with a different and / or updated component and / or changing a type of the controller that controls the component via the interface.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an adjusted component interface control framework.

Background Art

[0002] (Related Application) This PCT international application was filed on November 21, 2018, and claims the benefit of U.S. Patent Application No. 16 / 198,621, entitled "Coordinated Component Interface Control Framework," and U.S. Patent Application No. 16 / 198,653, entitled "Executable Component Interface and Controller," both filed on November 21, 2018, which are hereby incorporated by reference herein.

[0003] Various systems may rely on complex data processing pipelines to control the system. For example, data processing pipelines can be used to control manufacturing processes, track multiple conversations in a room, convert speech to text while identifying individual speakers, control the movement of autonomous robots through the environment, and regulate the movement of fine nanotubes through chemical stimuli and / or biofeedback. However, since these data processing pipelines are very complex, changes that improve one element of the processing pipeline can sometimes make the overall output of the pipeline actually deteriorate, making it difficult to adjust the performance of the data processing pipeline. Additionally, if the elements of the data processing pipeline are not executed optimally, the complexity of the processing pipeline can obscure one or more of the elements that are the cause.

Brief Description of the Drawings

[0004] The detailed description will be described with reference to the accompanying drawings. In the figures, the leftmost digit of the reference number identifies the figure in which that reference number first appears. The same reference number in different figures indicates a similar or equivalent item.

[0005]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0006] As described above, the system may include a complex data processing pipeline that masks problems with components of the processing pipeline and may prevent or interfere with efforts to tune the performance of the data processing pipeline. Further, the components of the pipeline can be of various types. A first component can be implemented in software executed by a processor, a second component can be implemented as hardware such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), or in an additional or alternative example, the first component can be implemented in the kernel layer of an operating system, the second component can be implemented in the application layer of the operating system, and a third component can be implemented in the application layer of a different operating system. The components can have different functions and thus operate on inputs to and outputs from the components.

[0007] Further, attempting to replay the operation of the data pipeline offline (i.e., attempting to reproduce the activity) can result in non-deterministic output, i.e., the data pipeline may not always produce the same output given a particular input. This is particularly exacerbated when components of the data pipeline operate in parallel on different computing nodes (e.g., different graphics processing unit (GPU) pipelines and / or different GPUs, different central processing unit (CPU) cores and / or different CPUs, various machines connected via a network, various virtual machines), which may be required in some applications to achieve the operating speeds useful for controlling a self-driving vehicle in real time.

[0008] Regardless of the use case, this application describes techniques related to components of a system (e.g., data processing pipelines, computer vision systems, perception components). As used herein, a system may include one or more components. A component, in turn, may include a system of multiple elements (e.g., various combinations of hardware and / or software).

[0009] For illustration purposes, a system for an autonomous vehicle may include an artificial intelligence (AI) stack that includes multiple systems such as, for example, a perception system, a localization system, a planner system, a drive module control system, an internal operations system, etc. Each of these systems may, in turn, include dozens, hundreds, or even thousands of components. The perception system may include, for example, one or more sensor data processing components, a component for segmenting images, a component for segmenting points of lidar data, a component for classifying objects identified in sensor data, a component for tracking objects, etc. Further, one of these components may include multiple sub-components such as a neural network and / or other machine learning models, instructions executable by a processor, and / or hardware.

[0010] This application describes techniques that include a framework for controlling inputs to and / or outputs from a component and / or for controlling when and / or for how long a component operates.

[0011] The framework described in this specification can configure a system such that actions performed by the components of the system can be deterministically reproduced, i.e., for the same input provided to the system implementing the framework, the same output is generated by the system. In some cases, the framework also facilitates the ability to replace components of a system with other components in a "plug and play" fashion, where previously developers had to specify the pipeline flow or rewrite parts of other components. In some examples, the techniques discussed in this specification include replacing components of a system with alternative components and comparing the behavior of the system before and after the replacement. For example, the framework can use log data to reproduce (and / or use simulation data to deterministically simulate) the behavior of the AI stack described above, and can obtain a first set of outputs from the AI stack and / or its components. The perception system of the AI stack may be replaced with an updated and / or different perception system, and the framework can execute the AI stack using the same log data and / or simulation data, enabling comparison of the behavior / performance of the AI stack before and after the replacement. In at least some examples, the framework can allow for maintaining a system variable (e.g., internal state, number of executions, etc.) despite having different components. In such examples, the system can classify how components affect the accuracy and precision of the overall system regardless of the required system resources.

[0012] In some cases, the framework may include a controller that provides signals to the input / output interface between the component and the controller. In some examples, the controller may be the destination and source of all input / output functions of the component, and / or the controller may cause the component to execute. In other examples, at least one input and / or output of the component may be input or output external to the controller. In addition to, or instead of, controlling the input to and / or output from the component, the controller may control the execution state of the component (e.g., cause the component to execute, prevent execution, control when and / or for how long the component executes, cause the internal state of the component to be set to a specified internal state). To control the input provided to the component, the output from the component, and / or the execution of the component, the controller may provide signals and / or other data to the input / output interface and / or control the status of the input / output interface (e.g., the controller may set a value / flag stored in a register associated with the component. The controller may call a process associated with the component). In an example where the component includes processor-executable instructions, the controller can operate the component by causing the processor-executable instructions to be executed by one or more processors (e.g., pushing the address associated with the process of the component onto the stack, setting a flag on an input / output interface configured such that the component pushes the address onto the stack in response to determining that the flag state indicates that the component should execute).

[0013] In some cases, the input / output interface includes a first input interface (the "initialization" input interface) configured to receive one or more variables from the controller, a second input interface (the "execution" input interface) configured to receive an instruction to execute from the controller (for example, the instruction may further include a period for execution and / or the instruction may include a flag to cause the component to execute while high), a third input interface configured to receive a keyframe from the controller and set the internal state of the component at least partially based on the keyframe (for example, this may include deserializing the keyframe to determine the internal state), a first output interface that outputs a keyframe at least partially based on the internal state of the component (for example, this may include serializing the internal state of the component to determine the keyframe, the component may output a keyframe periodically (for example, every 500 executions, every 1000 executions), and / or when the internal state changes by a degree of change that meets or exceeds a threshold degree of change), and / or a second output interface that outputs the result of the execution of the component.

[0014] Note that as used herein, the terms "result" and "output" (when used as nouns) may be used interchangeably to refer to the output of the execution of a component and / or the output of a system. For example, a component may output an output / result in response to execution. This output / result may be output as an output / result by the system and / or this output / product may be received by another component of the system that may also generate a second output / result. The system may additionally or alternatively output a second output / result. Using the term "result" avoids the confusing situation where a component may output (verb) an output (noun) via an output interface.

[0015] The techniques discussed in this specification can include various types of controllers. For example, without limitation, the types of controllers can include online controllers, playback controllers, and / or adjustment controllers. All different types of controllers can be configured to control inputs to components, outputs from components, and / or the execution state of components, and can be configured to do so unconsciously (e.g., without the need for knowledge of the physical components being controlled by the controller). As a result, a second component can replace a first component, while the same controller that was controlling the first component can continue to be used to control the second component after replacement without changing the operation of the controller, or in some cases without notifying the controller of the change. In some cases, components configured to include an input / output interface can be swapped with each other without reconfiguring the input / output interface, the component, and / or the controller. In other words, the controller can be agnostic to the component on the opposite side of the input / output interface. To facilitate controller agnosticism, the controller can receive a log file that can include a sequence of events captured from the online operation of a self-driving vehicle, user-defined timings, default timings, sensor data, and / or other non-deterministic values, as described below.

[0016] All different types of controllers may have the same core functionality, for example, controlling the input / output / execution of components and doing so without depending on the functionality of the component of which it is a controller. However, different types of controllers may include different additional functionality. For example, an online controller may, in addition to controlling the input / output / execution state of components, route messages between controllers (associated with other components). For example, an online controller may include subscriber components and publisher components. An online controller may receive messages from another controller (i.e., a publishing / upstream controller) via a subscriber component from a network channel and / or topic to which the online controller subscribes (e.g., in a publish-subscribe (pub-sub) architecture). Online control may additionally or alternatively issue messages to one or more subscriber and / or downstream controllers via network channels and / or topics. In some examples, the message may include an identifier of the topic and / or channel, the result of the execution of the component, the time at which a message containing the result was received by the controller from the component and / or a message containing the result was issued by the controller (the "generation time"), and / or a horizon time (e.g., the time until the component is re-executed earliest).

[0017] For example, a first component may output a first result, and a first online controller associated with the first component may receive the first result via a first input / output interface between the first online controller and the first component. The first online controller may issue a message including the first result to a second online controller associated with a second component. The second online controller may provide the first result as an input to the second component via a second input / output interface between the second online controller and the second component, and the second online controller may cause the second component to execute using the first result. The second online controller may receive a second result in response to the execution of the second component. The second online controller may issue a second message including the second result to one or more subscribing controllers in response to receiving the second result.

[0018] In some cases, an online controller may associate a generation time with a message issued by the online controller. In some examples, the online controller may provide, as an input to a component via an input / output interface, a clock signal, sensor data received from an autonomous vehicle, the result of one or more other components received by the online controller, and / or a non-deterministic value available to the component during execution. For example, the non-deterministic value may include a random number distribution and / or other values that may differ from execution to execution. In some cases, the non-deterministic value may include sensor data and / or a result generated by another component. The online controller may generate a non-deterministic value based at least in part on receiving a call for the non-deterministic value from a component, and upon generating the non-deterministic value, the online controller may provide the non-deterministic value to the component.

[0019] In some examples, the online controller may additionally or alternatively save events occurring in the component to a log file. For example, the online controller may save to the log file an event sequence, sensor data provided as input to the component, other inputs provided to the component, non-deterministic values provided as input to the component by the controller, output received from the component, diagnostic information regarding the execution of the component, and / or key frames received from the component. In some cases, the input / output interface may be configured to serialize the internal state of the component as key frames. The internal state may include hidden encapsulated data related to the execution of objects such as internal variables that may not be passed as output and / or may not be recoverable from re-executing the component based on the last one to five messages. For example, the internal state may include a counter value, an internal variable value based at least in part on a previous execution of a first component, an output context (e.g., a planner may generate multiple trajectories and select at least one to control an autonomous vehicle, and the planner may maintain a history of each position of the vehicle and / or positions achieved by each trajectory for each execution of the planner, this data may be taken into account when selecting a trajectory, this data may require multiple iterations to recreate the internal state at a particular execution of the planner for post hoc activities and / or recording data), and the like. In at least some examples, components may be designated as stateful (e.g., they depend on one or more variables previously calculated) and / or stateless (e.g., the calculations performed by the component are independent of previous calculations). As described above, stateful components may be designated as those that maintain a history of such internal states.In some examples, the input / output interface may generate keyframes when it detects them periodically (e.g., every 10 executions, every 500 executions, every 1000 executions) and / or when a significant state occurs (e.g., the internal state changes by a degree of change that meets or exceeds a threshold degree of change compared to the previous internal state. Whenever the component generates an output of a specified type).

[0020] In some examples, the sequence of events may define the order in which events occurred in a component and / or the order in which events occurred in a component relative to events in another component.

[0021] As described above, for the subsequent playback of the system execution to be deterministic and reproducible, the online controller may capture non-deterministic values generated by and / or provided to components associated with the controller (e.g., random number distributions, sensor data, results determined by another component).

[0022] The playback controller may include the functionality of the online controller and, additionally or alternatively, may replay previous events in the component. To reproduce the operation of the component in the same way as it previously operated (within the possible operating tolerance), the playback controller may receive a log file that includes the sequence of events and / or input data (e.g., generated by the online controller and / or an associated logger). In some cases, the log file may include one or more reproducibility messages previously generated by the online controller in relation to the operation of the component.

[0023] In an example where a playback controller is used to reproduce the operation of components that occurred in an autonomous vehicle (or other online scenario), the playback controller may receive a log file containing component (and / or its associated controller) and / or a sequence of events that occurred during the scenario in one or more other controllers, and / or sensor data from the autonomous vehicle, the output of an upstream component (although in some cases the output of the upstream component may not be used and / or may not be stored in the log file), key frames, and / or input data that may include non-deterministic values provided to the component. In some cases, the sequence of events may include references to data stored in a data store that can be read by the playback controller and supplied as input to the component in the order specified by the sequence of events.

[0024] In some examples, the playback controller can communicate with other controllers, make the framework deterministic, enable the scenario to be reproduced, and / or make it portable instead of allowing the component to output directly to the input of a downstream component. The playback controller sends messages to each other to simulate the flow of the pipeline.

[0025] For example, according to a native configuration (the original configuration of a data processing pipeline that does not implement a framework), a first component may receive sensor data from a sensor as input, may execute at least partially based on the sensor data, and may output the result of the execution to a second component (a downstream component). The second component may receive the result as input, execute at least partially based on the result, and output the second result to another downstream component and / or output the second result. According to the framework discussed herein, a first controller may be associated with the first component, and a second controller may be associated with the second component. Instead of enabling a component to receive input from other components, execute, and / or directly output results to other components, the first controller may receive the sensor data and determine when to provide the sensor data as input and when and for how long to execute the first component. The first controller may receive the result from the first component and / or determine when to issue a message including the result to the second controller associated with the second component. In at least some examples, the framework may determine the order of operation of various components at least partially based on the overall architecture of the system, optimize the order of computation to enable maximum parallel computation among the components while maintaining the order of operation of the overall system. Such an order may be implemented and / or determined based on data within one or more messages.

[0026] The techniques discussed in this specification can enable a user to force components to operate according to different sequences of events (e.g., a sequence specified by the user), either by enabling the user to edit a recorded sequence of events or by enabling the user to specify a sequence. A playback controller can receive a sequence of events that is at least partially based on user input (in addition to, or instead of, a sequence of events generated by an online controller). In some examples, the input data can additionally or alternatively include simulated sensor data generated by a simulation component and issued from a controller associated with the simulation component to the playback controller.

[0027] To ensure that the framework operates deterministically, a controller can associate a generation time and / or a horizon time with messages issued by the controller and / or control the order of messages provided as input to a component and / or the order of execution. The generation time can include an identifier of the time at which a result was received at the controller from a component, the time at which the controller generated a message including the result, and / or the time at which the controller issued a message. The horizon time minimally identifies the period until the controller executes a component. Basically, the horizon time functions like an agreement where the controller notifies another controller that a component will not be re-executed for another x period of time (e.g., 500 milliseconds, 100 microseconds, 1.2 seconds).

[0028] A controller that receives a message including a generation time and / or a horizon time may use the generation time and / or the horizon time to determine when to provide an input and / or to cause a component associated therewith to execute. For example, the controller may compare the generation time to a sequence of events and, in some instances where the controller receives multiple messages, rearrange the messages to correspond to the sequence of events and provide the messages as inputs to the component in the same order as reflected in the sequence of events, thereby ensuring that the sequence of events is replicated by the component (e.g., as compared to a sequence of events recorded by an online controller). In at least some instances, the execution times and / or periods of the components may vary as long as the order of execution and / or the total number is the same or similar. In some cases, the controller may additionally or alternatively determine the time and / or period at which the component can be executed, based at least in part on the horizon time. For example, the controller may determine to execute the component one or more times until the horizon time expires. In an example where the controller receives multiple messages and thus multiple horizon times, the controller may determine to execute the component one or more times until a minimum horizon time among the multiple horizon times and / or a horizon time shorter than at least a maximum horizon time among the multiple horizon times. By operating in such a manner, with respect to the generation time and / or the horizon time (regarding the sequence), the framework discussed herein may be able to reproduce previous operations of components and / or systems, including maintaining the order of operations and / or the state of the components and / or the system in operation.

[0029] Furthermore, the framework can make the system portable by operating in a computing environment that includes hardware different from an online (native) computing environment, and the framework reproduces the operations that occur in the native environment for the system. Further, the framework enables the system to be made portable by being processed on a distributed computing system and / or by a computationally limited computing device. For example, for the frameworks and the associated operations discussed herein, the controller controls the input / output / execution of components according to the sequence of events, generation time, and / or horizon time, so it does not matter whether the computing device that executes the controller and / or the components is a native computing device and / or is slower or faster than each other. Then, no component is "ahead" of another component, and no component is left without an input that should be received before execution.

[0030] Rather than being associated with a specific component, an adjustment controller can adjust the interaction between multiple controllers. For example, a system such as a perception system of an autonomous vehicle may include multiple components that form a data processing pipeline according to a native configuration. In at least one example, a controller may be associated with one of the multiple components, although it is contemplated that one controller may alternatively be associated with multiple components. In some examples, some of the multiple components may not include an input / output interface and thus may not be configured to interact with a controller. A component that does not include an input / output interface is referred to herein as a "non-adjusted controller." However, in some examples, each of the multiple components may be associated with a controller, such that the controller-to-component ratio is 1:1 (although other techniques are contemplated). In a native configuration, a specific component may receive input from one or more upstream components and output to one or more downstream components. However, according to the techniques described herein, a component that includes an input / output interface cannot communicate directly with other components as in a native configuration and instead can communicate exclusively with a controller associated with the component via the input / output interface. Next, the controllers can communicate directly with each other according to a processing pipeline pattern and / or via an adjustment controller.

[0031] In an implementation that uses an adjustment controller, the adjustment controller can receive messages issued by a controller, relay the messages to a subscriber controller, and / or relay the messages according to a sequence of events. For example, by receiving a message from a first controller, determining when to send the message to a second controller based at least in part on the sequence, and sending the message to the second controller at that time, the sequence can be ensured to be implemented. Additionally or alternatively, the adjustment controller can process messages addressed to unadjusted components (e.g., components that do not have an associated controller). For example, the adjustment controller can receive a message from a controller, determine that the message is addressed to at least one unadjusted component, convert the message to an unadjusted message type, and send the converted message to the unadjusted component. This can enable the controller to be agnostic as to whether messages issued by the controller are sent to adjusted or unadjusted components.

[0032] An arrangement that associates a controller with a component that includes an input / output interface is referred to herein as a "framework". As described above, the framework can assign a controller to each component (of the system) that includes an input / output interface. If the system includes unadjusted components, the framework can further include an adjustment controller for handling the interaction between the controller and the unadjusted components. By avoiding the native configuration of the pipeline and configuring the data pipeline according to the framework described herein, the operation of the pipeline (within the framework) can be deterministic, portable, maximize computing resources, and / or enable the reproduction of complex scenarios on computationally limited hardware.

[0033] The framework described in this specification enables a pipeline to deterministically process the same input, i.e., every time the same input data (e.g., sensor data received from a vehicle) is provided to the framework, the framework generates the same output each time. In some examples, the techniques discussed in this specification facilitate the replacement of a first component with a second component within the framework. For example, a controller may be associated with the first component, and by user selection, the first component is no longer associated with the controller, and the second component may be replaced with the second component such that it is immediately associated with the controller (and thus controlled by / communicates with the controller). For example, a first neural network may be replaced with a second neural network. This replacement may include loading the second component into memory, pushing the process associated with the second component onto the stack, replacing the address associated with the first component with the address associated with the second component in the register that the controller references when causing the component to execute, and / or when providing an input / receiving an output, etc. The "first system" may include the first component and / or other components, and the "second system" may include the second component that replaces the first component and / or other components.

[0034] The techniques discussed in this specification may additionally or alternatively include determining metrics associated with results output by a system and / or results output by a component, such as accuracy, precision, a precision-recall (PR) curve, a receiver operating characteristic (ROC) curve, latency, response time (e.g., number of executions before an object and / or track is identified), and / or acceleration (e.g., whether lateral acceleration meets or exceeds a threshold of lateral acceleration, whether forward acceleration meets or exceeds a threshold of forward acceleration associated with a perceived “jerk” of a vehicle). In some cases, the techniques may include determining accuracy, precision, a PR curve, and / or an ROC curve when a component is executed and / or when an output is received from a system that includes the component.

[0035] In some cases, the techniques may include determining a first metric associated with a first system and / or a first component, replacing the first component with a second component to form a second system, and determining a second metric associated with the second system and / or the second component. In some cases, the techniques may additionally or alternatively determine differences between the first system and the second system and / or the first component and / or the second component. In some cases, this may include comparing metrics associated therewith. In some cases, the techniques may include determining that the second metric is an improvement over the first metric, and transmitting instructions to replace at least partially the first component and / or the second component with the second component in an autonomous vehicle based at least in part on the determination that the second metric is an improvement over the first metric.

[0036] In some examples, the second metric may be an improvement over the first metric, the second metric indicates an accuracy and / or precision that meets or exceeds the accuracy and / or precision indicated by the first metric, the second metric indicates a PR curve, an ROC curve, and / or an acceleration that more closely matches the curve than the PR curve, ROC curve, and / or acceleration indicated by the first metric, and / or the second metric indicates a latency and / or response time that is shorter than the latency and / or response time indicated by the first metric.

[0037] In some cases, the technique may include receiving a log file, processing the log file by a first system, determining a metric associated with the first system, determining that one or more first components stored in memory correspond to one or more second components of the first system, and replacing at least one of the one or more first components with a corresponding one of the one or more second components, at least in part based on the determination of the correspondence. For example, components may be changed during the last update to a self-driving vehicle system. This second version of the component may be stored in a repository associated with version information. A computing device running the framework may determine that this changed component is available and corresponds to the previous component. The computing device may provide log data for reproducing the operation of the first system as input to the first system, replace the first component with the second version of the component to form a second system, provide the log data as input to the second system, and determine whether the operation of the second system is different from the first system. The framework may force the execution of the first system and / or the second system according to a sequence of events captured during the online operation of the system and / or according to a sequence of simulated events.

[0038] The technology described in this specification can improve the safety and efficiency of an autonomous vehicle by improving the performance of the autonomous vehicle's system. This technology can also remove components that are hindering the system's performance and / or can facilitate advanced troubleshooting. The framework discussed in this specification can deterministically operate a data processing pipeline and / or reproduce previous operations of the pipeline when the pipeline is operating on a non-native system and / or when using computing devices with different functions. Further, the framework can reduce the number of wasted computational cycles when executing the pipeline and can enable distributed execution of the pipeline (which is also deterministic).

[0039] (Exemplary Scenario) FIG. 1 shows an exemplary scenario 100 that includes an autonomous vehicle 102. According to the technology discussed in this specification, the autonomous vehicle 102 can receive sensor data from sensors 104 of the autonomous vehicle 102. For example, the sensor data can include a position signal (e.g., a GPS signal), an inertial signal (e.g., an accelerometer signal, a gyroscope signal, etc.), a magnetometer signal, a wheel encoder signal, a speedometer signal, a point cloud of accumulated lidar and / or radar points, an image (or images), an audio signal, a bariatric or other environmental signal, etc.

[0040] The self-driving vehicle 102 can provide sensor data received from such a sensor 104 to one or more systems to achieve various functions of the self-driving vehicle. For example, one or more systems can control the passenger's riding experience (e.g., temperature / humidity control, internal display control, door control function, seat function), the positioning and / or mapping system can generate a map of the surroundings of the self-driving vehicle and / or the position and / or orientation of the self-driving vehicle 102 within the map, the perception system can detect and track objects in the environment of the self-driving vehicle, the prediction system can predict the future position, speed, and / or acceleration of objects in the environment, and the acceleration and planning system can determine a trajectory for controlling the movement of the self-driving vehicle. These systems can each include one or more components. These components can include software and / or hardware components. For example, the components can include machine learning models such as neural networks and / or other instructions executable by a processor.

[0041] For example, FIG. 1 shows an exemplary system 106's native configuration (e.g., a data processing pipeline not operating using the framework described herein) that can represent a simplified perception system. The exemplary system 106 can receive sensor data from the sensor 104 as input, and based at least in part on the processing of the sensor data by the exemplary system 106, the exemplary system 106 can output a bounding box 108 and / or a semantic label 110. The bounding box 108 and / or the semantic label 110 can be the "result" 112 of the exemplary system 106 (i.e., the output generated in response to the operation of the exemplary system 106).

[0042] According to the native configuration of the exemplary system 106, the exemplary system 106 may include a sensor data processor 114 that can receive, normalize, smooth, transform, and / or otherwise preprocess sensor data. In some examples, the sensor data processor 114 may include subcomponents such as, for example, a Gaussian filter, a digital-to-analog converter (DAC), a sensor driver, and the like. The sensor data processor 114 may send the output generated by the sensor data processor 114 to one or more downstream components, which may include, in the illustrated example, a segmentation component 116 and a bounding box component 118. The segmentation component 116 may receive the processed sensor data from the sensor data processor 114 (the upstream component related to the segmentation component 116) and may identify a portion of the sensor data as being due to an object (e.g., labeling points of a LIDAR point cloud as being associated with an object, identifying pixels of an image, projecting LIDAR points onto an image, and identifying projected points and / or pixels as being associated with an object). The segmentation component 116 may output the segmented sensor data to downstream components, which may include, in the illustrated example, a bounding box component 118 and a semantic classification component 120.

[0043] The bounding box component 118 may receive inputs from the upstream component, the sensor data processor 114, and the segmentation component 116. The bounding box component 118 may generate a bounding box 108 of one or more objects that is reflected in the processed sensor data and corresponds to the segmented sensor data received from the segmentation component. The bounding box may send the generated bounding box 108 to a downstream component that may include an object identification component 122.

[0044] The semantic classification component 120 may generate semantic labels for a portion of the segmented sensor data based at least in part on the segmented sensor data received from the segmentation component 116. For example, the semantic labels may include exemplary semantic label 110, "four-wheeled vehicle", or other semantic labels (e.g., "pedestrian", "two-wheeled vehicle", "bicycle", "parked vehicle") configured to be output by one or more machine learning models of the semantic classification component 120 during evaluation.

[0045] The object identification component 122 may fuse various inputs received from upstream components, including the bounding box 108 output by the bounding box component 118 and / or the semantic labels of the semantic classification component 120, and output the result 112 to one or more downstream components. For example, the result 112 may be usable by a planning system to generate a trajectory for controlling the movement of the autonomous vehicle 102 and / or may be usable by a GUI component to generate a representation of the sensor data, the bounding box, and / or the semantic label 110 as shown at 124, and may include an object identifier formatted as a programming object.

[0046] In some examples, it is contemplated that the perception system may include more components and / or perform more functions. For example, the perception system may include multiple components for detecting objects from sensor data, multiple components for determining the classification of objects, multiple components for determining the current and / or future direction of travel, position, speed, and / or acceleration of objects, multiple components for tracking objects, multiple components for generating a map of the environment, multiple components for localizing objects within the map, multiple components for determining the bounding box of an object, and so on. For example, the segmentation component 116 may itself include multiple sub-components such as, for example, multiple machine learning models, processor-executable instructions, and / or hardware (e.g., FPGA and / or ASIC).

[0047] Exemplary system 106 is illustrated and described above to provide to those skilled in the art what is meant by simply "system", "component", and / or sub-component.

[0048] (Exemplary Framework) FIG. 2A shows a block diagram of an exemplary system 200 that includes one or more components configured according to a native configuration. Exemplary system 200 is depicted primarily to contrast the native configuration with the framework described herein. FIGS. 2A and 2B are described from the perspective of component 202 highlighted in bold lines.

[0049] An exemplary system 200 configured according to a native configuration may receive a system input 204. The system input 204 may include sensor data, results of another system, and the like. This input may be processed by upstream components 206(1)-(x). In some cases, component 202 may subscribe to the outputs of one or more upstream components 206(1)-(x) and / or otherwise receive the outputs of one or more upstream components 206(1)-(x). FIG. 2B shows both components 206(1) and 206(x) providing outputs to component 202, but not all components of the first layer necessarily provide outputs to component 202. A first component may be “upstream” from a second component by providing the output of the first component as an input to the second component to the second component. Thus, a component that receives the system input 204 and does not provide the output of that component to component 202 may not be considered an “upstream” component from the perspective of component 202. For example, from the perspective of independent component 208, component 206(1) is not an upstream component for independent component 208 because independent component 208 does not receive the output generated by component 206(1) as an input to independent component 208.

[0050] Independent component 208 is independent from component 202 in that the output of independent component 208 does not affect the output of component 202, but the operation of component 202 is at least partially based on the results of at least upstream components 206(1) and 206(x).

[0051] Based at least in part on receiving the results from the upstream component 206(1) and / or 206(x) as input, the component 202 may be executed, and in response to the execution, may generate an output. The component 202 may provide this output to the downstream components 210(1)-(y), for example, by issuing the output via a topic and / or channel to which the downstream components 210(1)-(y) subscribe. Also in this case, these downstream components 210(1)-(y) may receive the output of the component 202 as input, may be executed at least in part based on the output of the component 202, may generate respective outputs, and the downstream components 210(1)-(y) may send them to subsequent downstream components 212 and the like. In some cases, the downstream component 212 may be present in the last layer of the system 200, and as a result, the output from the downstream component 212 is the result 214 of the system 200. For example, if the system 200 is a planning system for an autonomous vehicle, the result 214 may include a trajectory (e.g., speed, steering angle, and / or yaw rate), and if the system 200 is a perception system, the result 214 may include an object list including the position, speed, acceleration, and / or classification of the detected objects.

[0052] Figure 2B shows a block diagram of an exemplary framework 216 for controlling inputs to, outputs from, and / or the execution state of one or more components of system 200. In some cases, to implement the framework, an input / output interface 218 (referred to as "input / output 218" in Figure 2B for brevity) may be included in a component for which deterministic reproduction and testing are desired and may exhibit non-deterministic behavior. In some cases, implementing the exemplary framework 216 may include associating controllers (220-230) with each component controlled via the exemplary framework 216. In some cases, a controller may be assigned to each component of system 200 that includes an input / output interface 218 in a 1:1 ratio, although other configurations are contemplated (e.g., one controller associated with three components, one controller associated with the components of one layer of a pipeline, one controller associated with components of a specified type within system 200). The exemplary framework 216 shown in Figure 2B shows a 1:1 ratio.

[0053] Rather than enabling a component to always receive input, execute, and / or provide output to other components, the controller 220-230 may control these and other functions. The controller 220-230 may include different types of controllers, and the controller 220-230 may control the input to the components associated with each controller, the execution state of that component, and / or the output to the (downstream) components subscribed to from that component. In some examples, a component controlled by a controller (e.g., component 202 is controlled by controller 224) may be separated from other components, and the controller associated therewith may provide input to the component and output from the component as defined by the processing pipeline.

[0054] However, a component that does not include an input / output interface 218, such as the downstream component 210(y), may receive input, execute, and / or transmit output according to its native configuration. This type of component is referred to herein as an "unregulated component". In some cases, a special type of controller, a regulating controller, may ensure that the output from the regulated controller is properly received by the unregulated controller (e.g., this may include determining the time to provide input to the unregulated controller and converting the message type to a message type usable by the unregulated controller). Subsequent figures describe the interaction of the controller with the input / output interface 218 and / or the components associated with the controller in more detail.

[0055] In some cases, the controllers 220-230 may receive and / or transmit data from / to other ones of the controllers 220-230 via one or more middleware and / or network topologies. For example, component 202 may be associated with a first process executed on a first hardware. Controller 224 may · via a first type of middleware (e.g., Data Distribution Service (DDS)) with a second controller associated with a component executed on a second hardware and / or using a different software language · via a second type of middleware (e.g., Robot Operating System (ROS), operating system's inter-process communication (IPC) tool) associated with a third controller associated with a component executed on the first hardware and / or associated with a second process · via a third type of middleware (e.g., InProcess) associated with a third controller associated with a component also associated with the first process receive and / or transmit data.

[0056] In other words, the controllers 220-230 may be configured to receive and / or transmit data to other controllers via different computing contexts associated with and / or different implementations of different middleware / inter-process communication (IPC). For example, during online operation, controller 302 may have received a message via the operating system's IPC, but during playback of an online scenario, as described below, controller 302 may read a message from a computer-readable medium (e.g., this may include decoding a video file and / or playing back one of the video frames at a time).

[0057] (Operation of an exemplary framework) FIG. 3-6 shows a block diagram of component 300, input / output interface 302, and different types of controllers, as well as components and functions of each of the different types of controllers. In some examples, component 300 may include instruction 306. Instruction 306 is shown as including neural network 308, but instruction 306 may additionally or alternatively include processor-executable instructions. Although not shown for simplicity, component 300 may additionally or alternatively include hardware such as, for example, DAC, ASIC, FPGA, etc.

[0058] In some examples, the input / output interface 302 may include an initialization input interface 310, an execution input interface 312, an output interface 314, a save keyframe output interface 316, and / or a load keyframe input interface 318. Generally, the initialization input interface 310 receives one or more variables from a controller and provides them as inputs to the instruction 306, such that the execution of the instruction 306 may be at least partially based on the one or more variables. In some cases, the initialization input interface 310 may be configured to set the initial state of the component 300 and / or the instruction 306 and / or to provide initial variables to the instruction 306, and / or in additional or alternative examples, the initialization input interface 310 may receive variables 334 from the controller and provide them as inputs to the instruction 306 at any point during the execution of the instruction 306 (e.g., before execution, during execution, at least partially based on receiving a request for variables from the instruction 306). In at least some examples, such variables may be passed via another interface (e.g., in addition to the initialization input interface 310). The execution input interface 312 receives instructions for execution from the controller and may cause the component 300 to execute, or otherwise, without receiving instructions, may pause or prevent the execution of the component 300. The output interface 314 may receive the results of the execution of the instruction 306 and provide the results to the controller. The save keyframe output interface 316 may cause the component 300 to serialize its internal state as a keyframe and output the keyframe to the controller. For example, the internal state may be a counter value, an output context (e.g., a planner may generate multiple trajectories and select at least one to control an autonomous vehicle, and for each execution of the planner, the vehicle may maintain a history of each position it was at and / or positions achieved by the trajectories).This data can be taken into account when selecting a trajectory, and this data may include, for example, multiple iterations for backward activities and / or recording data, reconstructing the internal state at a specific execution of the planner, etc. The load keyframe input interface 318 can receive keyframes from the controller, can deserialize the keyframes to the component 300, and can set the internal state of the component 300 to the internal state specified by the deserialized keyframes. Loading a keyframe can allow the developer to skip to an individual ("nth") execution of the component, and without re-executing all components from the first or last few executions, accurately reproduce the behavior of the component during that individual execution while maintaining the determinism of the execution and accurately setting the internal state so that determinism is maintained. For example, the internal state of a component may depend on the last m executions of the component itself and / or other components. Simply jumping to the "nth" execution of the component and providing modified input to the component without setting the internal state of the component does not accurately reproduce the behavior of the component, and the internal state may depend on non-deterministic factors, so determinism is not maintained. In some examples, a keyframe can specify an individual execution with which the keyframe is associated (e.g., by associating the keyframe with a generation time). In another example, a user who wants to view the 98th time step of an execution can effectively "fast forward" to the nearest keyframe (e.g., the one stored at the 95th time step) and simply advance the time by 3 units.

[0059] Figure 3 shows a block diagram of component 300, input / output interface 302, and online controller 320. The online controller can be a controller implemented during the real-time execution of the system. For example, online controller 320 can be used in an autonomous vehicle to provide messages from an upstream controller to component 300 and from component 300 to a downstream controller, thereby abstracting the input / output functionality of component 300 and / or controlling when component 300 is executed. Online controller 320 can also be considered a baseline controller since other controller types can include one or more components and / or functions of online controller 320.

[0060] In some examples, online controller 320 can receive message 322 from upstream controller 324 at subscriber 326 of online controller 320. According to the examples discussed herein, the framework can include a publish-subscribe architecture between controllers such that the pipeline flow can be easily changed by instructing the subscriber 326 of a controller to remove and / or add subscriptions to channels and / or topics without the controller having to specifically address another controller. For clarity, this is the only architecture discussed herein for relaying the output from a component to another controller, but it is understood that any other suitable transmission protocol can be implemented.

[0061] In any case, subscriber 326 may receive message 322 and may determine that message 322 includes data 328 and / or generation time 330. The generation time may include the time at which a result (e.g., associated with data 328) was received by upstream controller 324 from an upstream component 332 associated therewith, the time at which upstream controller 324 generated message 322, a monotonically increasing value that uniquely identifies a particular execution that resulted in upstream component 332 and message 322, and / or an identifier of the time at which upstream controller 324 issued message 322. In some cases, data 328 may include the result of an execution of upstream component 332 associated with upstream controller 324.

[0062] Subscriber 326 may provide data 328 or at least a portion thereof to initialization input interface 310 as one of one or more variables 334 provided by online controller 320 to initialization input interface 310. Variable 334 may include values that can be used by instruction 306 for execution. For example, instruction 306 may include a process configured to receive a Gaussian random number distribution, a LIDAR point cloud, and a covariance matrix, and a neural network configured to receive a LIDAR point cloud and semantic labels. Online controller 320 may generate non-deterministic variable values with non-deterministic variable generator 336. For example, this may include generating a random number distribution or the like. In other words, instead of enabling the generation of values that component 300 could not deterministically reproduce, non-deterministic variable generator 336 may generate values. Additionally or alternatively, clock 338 may provide a clock signal to initialization input interface 310 as one of variables 334.

[0063] When component 300 is controlled by online controller 320, in some cases, online controller 320 may control when and / or for how long component 300 operates. For example, online controller 320 may perform a callback via execution interface 312 by providing component 300 with executable code that can cause component 300 to execute, at least in part, based on the executable code provided to component 300, and / or in some examples, may include variable 334. In some examples, the callback may be a synchronous callback, but it is contemplated that the callback may be asynchronous as long as online controller 320 controls when component 300 is executed. In additional or alternative examples, online controller 320 may set execution input interface 312 to a "high" value, whereby component 300 can execute when it receives variable 334 via initialization input interface 310. In some examples, execution input interface 312 may include a flag in a register that online controller 320 can set to a "high" flag value indicating that component 300 can execute and / or execute based at least in part on variable 334 received via initialization input interface 310. Conversely, a "low" flag value may pause and / or prevent the execution of component 300 until the flag transitions to a "high" value. In additional or alternative examples, online controller 320 may push the address associated with instruction 306 onto the stack to cause component 300 to execute, and / or online controller 320 may call or otherwise notify execution input interface 312, in response to which it may push the address onto the stack. When on the stack, one or more processors may execute the instruction associated with the address when they reach the address in the stack.In the latter example, component 300 may send an address (and / or any other relevant address if component 300 includes a plurality of sub-components) to execution input interface 312 in preparation for the execution input interface 312 to push the address onto the stack for execution by one or more processors.

[0064] In an additional or alternative example where an online controller 320 is associated with component 300, the online controller 320 may cause component 300 to execute periodically (e.g., at 30 MHz, 500 MHz, 1 GHz) and / or when a subscriber 326 determines that a message has been received from all upstream controllers that the online controller 320 has subscribed to since the last execution of component 300.

[0065] The output interface 314 can receive the result 340 of the execution of the component 300 and can provide the result 340 to the publisher 342 of the online controller 320. Based at least in part on receiving the result 340, the publisher 342 can generate a message 344 that includes data 346 (which can include, for example, the result 340 and / or its representation), and / or a generation time 348. The publisher 342 can also include appropriate routing information associated with the message 344, such as, for example, a channel identifier and / or a topic identifier for a pub-sub architecture. The publisher 342 can issue the message 344, and the message 344 can be received by subscribing a (downstream) controller 350. This application mainly describes the use of a pub-sub architecture for sending the output of one component to the input of another component, but it is understood that other network configurations are considered and / or otherwise a pipeline configuration is established. Other network configurations can include, for example, assigning static IP addresses to controllers and / or unregulated components, addressing controllers and / or unregulated components using virtual routing and forwarding (VRF) schemes, etc.

[0066] In some cases, in addition to message 344 (or, if component 300 is a sink, i.e., if component 300 digests the input but does not send the output to another controller / component, instead of message 344), the publisher 342 may issue a reproducibility message 352. In some cases, the reproducibility message 352 may be received by a logger 354 that receives reproducibility messages from one or more components via a logging channel different from the channel for data messages ("data channel"). Messages 322 and 344 may be examples of data messages. In some cases, the data store 356 may "listen" to the data channel and thereby receive one or more data messages such as message 322 and / or message 344. The data store 356 may, in some examples, store messages received via the data channel. The playback controller may deterministically reproduce the operation of component 300 using the reproducibility messages stored by the logger 354 and / or the data messages stored by the data store 356. In some cases, the online controller 320 may operate as a pure transport layer and may not generate reproducibility messages.

[0067] In some cases, the reproducibility message 352 may include reproducibility data 358. In some cases, the reproducibility data 358 may include information sufficient to reproduce a particular operation of component 300 (e.g., information sufficient to re-execute a particular execution of instruction 306, such as outputting result 340 and / or the last execution that issued message 344), and may provide an indication of the order of the operations (e.g., executions) of the component.

[0068] FIG. 4 shows a block diagram of data store 356 and logger 354, as well as the respective data received, generated, and / or stored thereby. In some examples, data store 356 may receive data messages such as message 344 issued by a controller via a data channel, and logger 354 may receive reproducibility messages such as reproducibility message 352 via a logging channel. In some cases, data store 356 may store the received data message (e.g., message 344-400 that may be received from one or more controllers associated with one or more components) in a data repository such as a non-transitory computer-readable medium, and / or data store 356k may additionally or alternatively store the reproducibility messages received via the logging channel and / or received from logger 354.

[0069] In some cases, a data message such as message 344 may include data 346 including a generation time 348 and the result 340 of the execution of component 300. In some cases, generation time 348 may include a monotonically increasing value that uniquely identifies a particular execution of the component that generated the message and / or the component that resulted in message 322. For example, if component 300 performs its 3,462nd execution, generation time 348 may include an identifier of the 3,462nd execution by the component and, in some examples, an individual identifier of the component itself.

[0070] In some cases, a reproducibility message such as the reproducibility message 352 may include the same identifier as the corresponding message 344 and / or may otherwise be identified as related to the message 344. In some cases, the reproducibility data 358 may include sufficient information to deterministically reproduce the operation of the component 300. For example, the reproducibility data 358 may include a sequence of events 402, non-deterministic values generated by the non-deterministic variable generator 336 and / or provided as inputs to the component by the controller, diagnostic information regarding the execution of the component (e.g., execution time, number of cores and / or threads dedicated to the execution of the component, state of the execution of the component (e.g., success, failure, pause, etc.)), and / or key frames received from the component. As a non-limiting example, the reproducibility data 358 may include information about variables / components necessary to execute the computation so that the order of the computation can be preserved during playback.

[0071] In some examples, the event sequence 402 may include the order in which events occurred at the component 300 (and / or input / output interface 302 and / or online controller 320), and / or the order in which events occurred at the component 300 in relation to events at another component. For example, the event sequence 402 may define the order in which input phases, execution phases, and / or output phases occurred at the component (optionally in relation to input phases, execution phases, and / or output phases at another component). For example, the input phase may include providing an input to the component (e.g., via the initialization input interface 310, the execution input interface 312, and / or the load keyframe input interface 318), which may include the order of messages and / or variables provided to the component, the order in which messages were received at the controller, etc. The execution phase may include the delay between receiving a message and / or providing an input to the component and executing the component, the relative time at which the controller caused the component to start execution (e.g., in relation to other events at another controller), the period during which the component was executed and / or the number of times the component was executed, the relative time at which execution ended and / or an output was received from the component, etc. The output phase may include receiving an output from the component (e.g., whether as a result of execution and / or a keyframe), generating a message including the output, and / or issuing the message. The event sequence 402 may additionally or alternatively identify the order in which the online controller 320 provided messages as inputs to the component, and / or the identifiers of messages provided as inputs such as the generation times 348 and / or 404. The message identifier may correspond to one or more messages stored in the data store 356.In additional or alternative examples, the event sequence 402 may identify message 344 as an output associated with the reproducibility message 352 (e.g., by including the generation time 348 of message 344 in an output field of the event sequence 402).

[0072] For example, FIG. 4 shows an event sequence 402 associated with message 344 and including the order of operations at component 300 and the identification of related messages. For example, the event sequence 402 may identify that the online controller 320 input message 322 to the component at 406 (which potentially as an initialization parameter since it occurred before the execution command was given at 410) before the online controller 320 starts the execution of component 300 at 408 via the execution interface 312. Following the execution of component 300 at 408, the online controller 320 provides message 400 as an input to component 300 at 410, then the result 340 is received via the output interface 314 at 412, and then the online controller 320 issues message 344 at 414.

[0073] In some examples, in addition to the order identified by the event sequence 402, the event sequence may identify messages associated therewith stored in a data store (e.g., identified by generation time and / or unique identifier, and the identified messages may have been provided as inputs to component 300 and the identified messages may have been output by the component in response to execution), identify and / or store non-deterministic data such as non-deterministic variable values provided to component 300, clock data provided to the component, etc. For example, the event sequence 402 may not only identify the relative order of providing message 322 as an input at 406, but may also include the generation time 330 and / or other identifiers suitable for reading message 322 from the data store 356.

[0074] The event sequence 402 shows message input, start of execution, result reception, and message issuance, but it should be noted that any of the operations of the input / output interface 302 and / or the online controller 320 discussed herein can be recorded by the event sequence 402. - For example, any part of the input phase, execution phase, and / or output phase discussed herein, their order, references to related data (e.g., when the data is stored in the data store 356), and / or when the data store 356 does not store data, and / or when the data is present in the reproducibility message and / or the log file 416, the data itself. For example, the variable 334, the result 340, any non-deterministic variable values provided to the component, clock data, etc. can be stored in the data store 356 and referenced by the event sequence 402. To give another example, the event sequence 402 can show the order in which messages are provided to the component 300 for waypoints during the execution of the component 300 (e.g., requests received from the component 300 in response to execution, messages received from the component 300 marking the waypoint, output of results by the component 300).

[0075] In some cases, the logger 354 may store the reproducibility messages received by the logger 354. Each reproducibility message stored by the logger 354 may include the respective sequence of events, non-deterministic variable values, clock data generated by the controller's clock (e.g., the period until various events occur, the delays during that period, and / or the time of occurrence), and / or key frames associated with the execution of the components related thereto. In additional or alternative examples, the logger 354 may generate a log file 406 based at least in part on the reproducibility messages received by the logger 354. The log file 406 may include the entire sequence of events generated from the respective sequences of events indicated by the reproducibility messages received by the logger 354 for a particular component. In some cases, the logger 354 may generate a log file for each component of the system, such that each component has a sequence of associated events that identifies the order and / or related messages (and / or other non-deterministic data such as non-deterministic variable values, clock data, key frames, etc.). In additional or alternative examples, the logger 354 may generate the entire sequence of events reflecting the order of events in two or more components based at least in part on the sequences received from the controllers associated with the two or more components. In other words, the log file 416 may include sequence events reflecting the order of events from two or more components of the system to all components of the system, such that the events are ordered with respect to all components. In additional or alternative examples, the log file 406 may additionally include the data 346 and / or the generation time 348. In some examples, the log file 406 may include the reproducibility messages received from the controller associated with the component. In some cases, the logger 354 may be part of the online controller 320.

[0076] Figure 4 shows a block diagram of component 300, input / output interface 302, and playback controller 400. Figure 4 repeats the input phase shown in Figure 3 and details additional or alternative functions and / or components that may be included in playback controller 400. Figure 5 continues the description of playback controller 400 with respect to the execution and / or output phase of Figure 3. Playback controller 400 may operate for the purpose of capturing the operation of component 300 such that the operation of component 300 can be reproduced as closely as possible to the operation that occurred when it was captured and the operation can be reproduced deterministically. Figure 5 shows a block diagram of component 300, input / output interface 302, and playback controller 500. In some cases, playback controller 500 may include one or more components and / or functions of online controller 320 and / or, additionally or alternatively, may reproduce previous events deterministically and portably with components. Playback controller 500 may receive log file 502, which may represent log file 416 and / or another log file. For example, online controller 320 (and / or associated logger 354) may generate log file 502, store it in memory accessible by playback controller 500, and / or transmit it. In some cases, log file 502 may include sequence of events 402. In some cases, upon instantiation of playback controller 500, playback controller 500 may provide a unique identifier associated with component 300 for reading log file 502.

[0077] In some examples, playback controller 500 may include a timing enforcer 504 that synchronizes operations / events at component 300, such that the operation of the system including component 300 may be potentially deterministically reproduced in a non-real-time manner (e.g., faster or slower than when online hardware operates, and operates on different types of hardware with different computing resource capabilities and / or functions). Since different components of the system may require different lengths of time to execute, a global clock that synchronizes the execution of the components of the system is insufficient to ensure that each component has an input used for execution, or that each component does not receive an input to be received for execution after the execution has occurred.

[0078] Furthermore, the system can be configured to operate on different types of software and / or the framework can distribute the system's processing across nodes of a distributed computing system that may include different capabilities and thus different operating speeds and / or hardware for functions. For example, one component may include a sub-component process running on a central processing unit (CPU), while a second component may include thousands of sub-component threads executable by a graphics processing unit (GPU) core / pipeline. The sub-components not only differ in complexity and runtime, but the CPU and GPU may include different computing resource management functions that exhibit different behaviors when computing resources are unavailable for sub-component processing. For example, the CPU may include process management such as sophisticated schedulers and / or hyperthreading that block processes from execution until computing resources become available, while the GPU may include a more basic system that may fail threads when computing resources are unavailable. Thus, the system may not be accurately reproduced when native process management functions of the CPU, GPU, and / or other processors control component scheduling.

[0079] The frameworks discussed herein that include a timing enforcer 504 can allow components to be out of sync with respect to time recorded by another controller, but can control the timing of inputs to components, outputs from components, and / or the execution of components, such that the overall operation of the system remains deterministic and components receive the inputs they use before execution. The framework can reproduce events at components in the same order in which they occurred when the events were recorded, even when components do not maintain a common time. In some cases, the framework's controller may have individual clocks for each component, and these clocks are permitted to become synchronized or unsynchronized.

[0080] To overcome the above problems that can arise from non-simultaneous / asynchronous execution of components, messages transmitted between playback controllers can further include sufficient data, such as generation time and / or horizon time, to retroactively synchronize the controllers. For example, the upstream controller 508 can include a playback controller, and its publisher generates and issues a message 344 to include data 328, generation time 348, and / or horizon time 506. In some examples, the horizon time 506 can be provided in the message 344 to indicate that the next execution of component 300 cannot be affected by the execution of an upstream component 332 that has not yet been executed.

[0081] In some examples, the generation time and / or the horizon time may increase monotonically. In another example, the horizon time may include a period and / or a number of executions and may not increase monotonically. In some examples, the horizon time 506 may be adjusted in view of the delay in the delivery of the message 344 to the component 300 and / or the playback controller 500. In some examples, the horizon time may define the earliest time at which a component is executed and / or the next message is issued by the controller in response to the execution of the component. The horizon time may be a lower bound, i.e., the actual issue time may be later than the horizon time. In some examples, the playback controller 500 may issue to the publisher 342 a first message including the result of the execution of the component 300 and a first generation time identifying the earliest estimated time at which the publisher 342 issues the next message in response to the execution of the component 300. In some examples, the publisher 342 may issue a second message before the component 300 is re-executed to update the horizon time if the timing enhancer 504 determines that the earliest time is later than the previously estimated time. Thereby, the use of computing resources may be maximized throughout the framework. Since the component 300 has not been re-executed yet, the second message may forego the inclusion of the result.

[0082] In some cases, the timing enhancer 504 may cause the component 300 to execute (e.g., via the execution input interface 312) at least in part based on receiving a message (i.e., message-based execution) and / or determining that a period has elapsed since the previous time-based execution (i.e., time-based execution). The execution of the component may be defined by the time at which the execution occurs (message-based execution or time-based execution, or both), the internal state of the component immediately prior to execution, and a set of messages provided to the component 300 during / at the time of execution via the initialization input interface 310. The timing enhancer 504 may be configured with rules to retroactively ensure that the component 300 is synchronized. The rules may include causing the execution of the component 300 in response to the determination that (1) a future event could not change the time at which the execution occurs (e.g., the execution is time-safe), (2) a future event could not change the set of messages presented to the component during / at the time of execution (e.g., the execution is message-safe), and (3) a future event could not change the internal state held by the component immediately prior to execution (e.g., the execution is state-safe). The timing enhancer 504 may pause the component 300 (e.g., maintain the internal state, prevent the internal state of the component 300 from changing during execution, and / or prevent instructions associated with the component 300 from being executed) until the timing enhancer 504 determines to cause the component 300 to execute.

[0083] In some cases, message-based execution may be at least partially based on receiving one or more messages from a subscriber that triggers the next execution of component 300. For example, the timing enforcer 504 may determine that component 300 may be configured to receive a first set of messages from the first and second issuing controllers and a second set of messages from the third issuing controller, and component 300 may execute using either or both sets. Component 300 may output an indication of the set of messages via the initialization input interface 310. In additional or alternative examples, the sequence of events in the log file 502 may indicate the messages on which the execution was based. Thereby, the timing enforcer 504 may "expect" the receipt of messages to satisfy the set of messages available for use by component 300 for execution. Thus, the timing enforcer 504 may cause component 300 to execute when it determines that the subscriber 326 has received all messages to satisfy the first set and / or the second set. The timing enforcer 504 may also provide the messages that satisfy the first set and / or the second set to component 300 via the initialization input interface 310. In some cases, the set may include a single message from a single publisher, but it is contemplated that the set may include multiple messages from a single publisher and / or multiple publishers.

[0084] In some examples, the timing enforcer 504 may additionally or alternatively determine the next time to cause the component to execute based at least in part on determining the minimum of the shorter of the time of the next time-based execution or the horizon time and the next message issuance time. This can be expressed as follows: (next execution time) = min((next timing-based execution) ∪ (min((horizon time, next message issuance time): for each publisher that triggers execution))).

[0085] In some cases, the timing enforcer 504 may additionally or alternatively determine a period (510) during which the timing enforcer 504 may cause and / or permit the execution of the component 300. For example, the timing enforcer 504 may cause the component 300 to be executed one or more times until the expiration of the period. In some cases, the timing enforcer 504 may determine that the period corresponds to the horizon time 506. However, a plurality of messages that may include different horizon times may be received at the subscriber 326. For example, FIG. 5 shows receiving a first message at 512 that includes a horizon time 514 of 500 milliseconds and receiving a second message at 516 that includes a horizon time 518 of 300 milliseconds. In this example, the rules that configure the timing enforcer 504 may determine that the period during which the component 300 is executed may be less than or equal to the 300-millisecond horizon time 518 because it is the minimum time. This is because executing the component 300 for longer than 300 milliseconds may pose a risk of compromising execution time safety, message safety, and / or execution safety. In some examples, the timing enforcer 504 may prevent the component 300 from being executed even if the period has not yet expired if the next execution uses a message that has not yet been received.

[0086] In some cases, the timing enhancer 504 may additionally or alternatively order the messages and / or set of messages provided as input to the component 300 based at least in part on the generation time and / or horizon time associated with the sequence of messages and / or events read from the log file 502. For example, the upstream component 332 may execute more quickly and / or periodically than the component 300, and thus the subscriber 326 may have multiple messages from the upstream controller 508 prior to the next execution of the component 300. The timing enhancer 504 may ensure that messages are provided to the component in ascending order of generation time. In an additional or alternative example, the timing enhancer 504 may provide messages to the component 300 in an order corresponding to the sequence of events from the log file. In some examples, the timing enhancer 504 may additionally or alternatively cause the execution of the component 300 and / or cause the issuance of messages according to the sequence of events 620.

[0087] In some examples, the timing enhancer 504 may determine one or more messages 510 referenced by the sequence of events of the log file 502, read the messages 510 from the data store, and provide the messages 510 to the component 300 according to the sequence of events. For example, the timing enhancer 504 may determine that the identifier of the reproducible message stored in the log file 502 corresponds to the next execution of the component 300 (e.g., the next execution of the component 300 is the 200 番目 th execution, and the timing enhancer 504 determines that the log file 502 has the 200 番目(determine to include a reproducibility message related to the execution of), and it may be determined that the reproducibility message identifies message 510 and its order. Depending on the usage example, the playback controller 500 may wait for the reception of message 510 from the upstream controller (since the framework is deterministic, the upstream controller issues a message with the same identifier as message 510), and / or the playback controller 500 may read message 510 from memory (e.g., data store 356). The timing enforcer 504 may determine the order in which to provide message 510 as input. Further, if message 510 includes a message issued by the online controller 320, when the component 300 has been changed, as described below, the message previously issued (and included in message 510) by the online controller 320 may be read for the same execution of the component 300 (e.g., the 200th execution in the above example) and compared with the result of the component 300.

[0088] In some examples, the timing enforcer 504 may additionally or alternatively cause other data from the log file 502 (and / or data store 356) to be provided to the component 300 in the order specified by the sequence of events. For example, this may include providing sensor data, non-deterministic values, clock signals, component results, etc. as input to the component 300. This may enable the framework including the playback controller 500 to deterministically reproduce the scenarios that occurred during the operation of the system including the component 300.

[0089] In some cases, the event sequence can be defined by the user instead of or in addition to the recorded sequence. Further, the data included in the log file 502 can be provided by an online controller, the user who defines the data, and / or a simulation component. For example, the simulation component can be configured to simulate sensor data and / or the response of the autonomous vehicle to the output of the system including component 300. For example, the simulation component can form part of the feedback loop of the system including component 300. The simulation component itself can be associated with the controller.

[0090] FIG. 6 shows a block diagram of a framework implementation including component 300, input / output interface 302, first controller 600, adjustment controller 602, other controllers 604(1)-(x) and components 606(1)-(x) associated therewith, and non-adjustment component 608.

[0091] In some examples, rather than being associated with a particular component, the adjustment controller 602 may adjust actions among multiple controllers. For example, the adjustment controller 602 may receive a message 610 from other controllers 604(1)-(x), and ensure that the message 610 is routed to an appropriate subscriber. In some examples, upon instantiation of the framework, the adjustment controller 602 may request a parameter set from a controller (e.g., either the first controller 600 and / or any of the other controllers 604(1)-(x)). The first controller 600 may generate a parameter set based at least in part on querying the component 300 to verify the input, execution, and / or output characteristics of the component 300. The first controller 600 may retain at least a portion of the input, execution, and / or output characteristics, and send to the adjustment controller 602 a parameter set specifying inputs used by the component 300 to execute (e.g., other components to which the component subscribes, messages consumed by the component 300, non-deterministic variables consumed by the component 300), outputs generated by the component (e.g., message types, subscribers to which the component 300 outputs messages), and the like. For example, the inputs used by the component 300 to execute may include a set of messages consumed by the component 300 to generate an output and / or identifiers of non-deterministic variables. The parameter set may further include delay times associated with the component 300, such as, for example, initialization delay, average execution time (at a given processing frequency).

[0092] The parameter set received from the controller may be sufficient to configure the conditioning controller 602 to route messages between controllers according to the pipeline configuration. However, to further decouple the components from the processing pipeline flow, the conditioning controller 602 may receive a configuration that specifies the data processing pipeline configuration. For example, the configuration may identify a pub-sub pattern that defines the relative positions of components / controllers within the flow (e.g., the component that component 300 subscribes to, the component that component 300 publishes to).

[0093] Based at least in part on the parameter set and / or configuration, the adjustment controller 602 may route the message 610 received from other controllers 604(1)-(x) to an appropriate subscribing controller. In some cases, the parameter set associated with the component 300 may be identified as corresponding to a set of messages consumable by the component 300 as input for execution, and the adjustment controller 602 may be configured to collect messages that identify such. The adjustment controller 602 may collect messages until the adjustment controller 602 determines that the collected messages satisfy a set of messages, at which point the adjustment controller 602 may send the set of collected messages 612 to a first controller 600 associated with the component 300. In some examples, the adjustment controller 602 may additionally or alternatively delay the issuance of messages to the first controller 600 and / or otherwise time it based at least in part on the generation time associated with the message, the horizon time associated with the message, and / or the sequence of events stored in and / or provided by the user to the log file. For example, the adjustment controller 602 may achieve substantially the same operation as above, except that the adjustment controller 602 does not interact directly with the input / output interface 302, but rather may send instructions and / or messages to the first controller 600, which may then provide messages and / or instructions to the component 300 via the input / output interface 302, and may include a timing enforcer 504.

[0094] In some examples, adjustment controller 602 may additionally or alternatively exchange message 614 with unadjusted component 608. Unadjusted component 608 may be a deterministic component, may not include input / output interface 302, and / or may not be associated with a controller other than adjustment controller 602. In some examples, messages transmitted between controllers via a pub-sub network may be differently formatted than messages transmitted between unadjusted components 608. Further, unadjusted component 608 may exchange messages via channels and / or topics of a pub-sub network different from those of adjusted components (e.g., components 300 and / or components 606(1)-(x) in this example).

[0095] In some examples, the conditioning controller 602 may receive a message from a controller, may determine that the message needs to be routed to a non-conditioned component (even if the message is formatted for a conditioned component), may convert the message to a non-conditioned message type, and / or may issue the converted message 614 to the non-conditioned component (e.g., via the non-conditioned component portion of a pub-sub architecture). In some examples, converting the message may include issuing a message converted via a second network channel and / or a second topic when the original message is received via a first network channel and / or a first topic, and / or removing information related to the conditioned component (e.g., removing the horizon time, removing instructions for interacting with the input / output interface, removing references to the input / output interface). The conditioning controller 602 may perform this process in reverse for a message received from the non-conditioned component 608 (e.g., the conditioning controller 602 may associate a reception time with the message received from the non-conditioned component, and the conditioning controller 602 may issue the message via the conditioned portion of the pub-sub architecture). In some examples, since the non-conditioned component 608 is not associated with a controller, the conditioning controller 602 may determine the time to provide the message to the non-conditioned component so that time safety and / or message safety are not violated at the non-conditioned component 608.

[0096] In some examples, the conditioning controller 602 may perform a discovery operation to discover a mapping of the pipeline configuration by issuing a parameter set received from a controller associated with the conditioning controller 602 and receiving parameter sets from other conditioning controllers until it has received parameter sets from all other conditioning controllers.

[0097] (Exemplary Processing) FIG. 7 shows a flowchart of an exemplary process 700 for adjusting the performance of a system using the framework discussed herein. The system can be a first component and / or a first system that includes other components such as, for example, processor-executable instructions, a machine learning model, and / or hardware. In some cases, a tuning model, a tuning controller, an input / output interface, and / or any type of controller discussed herein can execute all or part of the exemplary process 700.

[0098] In operation 702, the exemplary process 700 can include receiving a sequence of sensor data and / or events according to any of the techniques discussed herein. For example, the sequence of sensor data and / or events can be stored in a log file. In some examples, an online controller can store the sequence of sensor data and / or events at least in part based on monitoring the operation of a component during the online operation of the component. It is contemplated that the log file can be generated by one or more online controllers and / or components associated therewith. In additional or alternative examples, the sensor data can include simulated sensor data generated by a component operated by the framework. In other words, one of the components of the framework can be a simulator that generates data used as input to other components of the framework. In some examples, at least one component can be a source (e.g., the component generates an output and does not consume an input), and / or at least one component can be a sink (e.g., the component receives an input but does not generate an output).

[0099] In operation 704, exemplary process 700 may include processing sensor data by a first system, at least in part based on operating components of the first system according to a sequence of events in accordance with any of the techniques discussed herein. For example, components of the first system may be associated with a playback controller (or any other type of controller) that enforces the rules described above (e.g., via a timing enhancer, message data such as generation time, horizon time, etc.), achieving a deterministic replay of the operation of the first system when the sensor data and / or sequence of events were stored and the first system processed the sensor data. In some examples, to achieve operation 704, the framework discussed herein may first be instantiated by a controller associated with components that include an adjustment controller along with an input / output interface and / or unregulated components of the first system. In some examples, the instantiation may occur prior to operation 704 and may further include receiving a parameter set and / or configuration as discussed above.

[0100] Operating the first component and other components according to a sequence of events may include controlling inputs, outputs, and / or execution states to the components of the first system such that execution of the components produces a deterministic output. This means that the framework discussed herein causes the first system to produce the same output each time the same sensor data and / or the same sequence of events are provided to the first system that is coordinated by the framework.

[0101] In operation 706, exemplary process 700 may include receiving a first output, at least in part based on processing sensor data by a first system according to any of the techniques discussed herein. For example, the first output may be an output of the overall system, which may be the output of a component in the last layer of the first system (i.e., the layer of the component that is the most downstream and last to generate an output in the flow of the data processing pipeline). In another example, the first output may be an output of a particular component, depending on the target of the adjustment procedure. In some examples, the first output may be the output of the system, as improving the performance of a particular component may actually degrade the performance of the overall system. For example, even if a particular component operates more appropriately, it may operate longer or create an output that degrades the accuracy and / or precision of the system, due to increased complexity of the pipeline being difficult to understand (e.g., neural networks and / or other machine learning models may include activation functions that can be stimulated by hidden layers and / or modified inputs, which may cause variations in the output). As a non-limiting example, a new detection component may have a higher pedestrian detection rate than a previous version, but a lower overall object detection rate.

[0102] In operation 708, exemplary process 700 may include determining a first metric associated with a first output according to any of the techniques discussed herein. For example, the metric may include accuracy, precision, PR curve, ROC curve, latency, response time (e.g., number of executions before an object and / or track is identified), and / or acceleration (e.g., whether lateral acceleration meets or exceeds a lateral acceleration threshold, whether forward acceleration meets or exceeds a forward acceleration threshold associated with a perceived “jerk” of the vehicle) associated with the first output. These metrics may be suitable for measuring the performance of various components where ground truth may be known, although additional or alternative metrics may be suitable for measuring the performance of the system based on the output and / or any input, ground truth, etc.

[0103] In operation 710, exemplary process 700 may include determining whether an alternative component is available according to any of the techniques discussed herein. For example, when a component is changed, e.g., by an update by a source code developer, a hardware change, and / or by additional machine learning training techniques, the changed component may be stored in a repository. The changed component may be stored in association with version control and / or other identifier data, such that in operation 710, exemplary process 700 may determine whether a changed version of the component is stored in the repository.

[0104] In some examples, when alternative components are not available, the exemplary process may transition to operation 712. In operation 712, the exemplary process 700 may include determining whether a first metric meets or exceeds a threshold metric. Meeting or exceeding the threshold metric may indicate proper performance of the first system. If operation 712 determines that the first metric meets or exceeds the threshold metric, the exemplary process 700 may return to operation 702, which may include receiving new sensor data and / or a new sequence of events.

[0105] However, if the threshold does not meet the threshold metric and alternative components are not available, the exemplary process may transition to operation 714. In operation 714, the exemplary process 700 may include causing a notification to be displayed via a user interface to notify the user that the first system does not meet the threshold metric. Additionally or alternatively, operation 714 may include determining a metric associated with a particular component of the first system when the first metric is associated with the output of the system, and / or determining whether the metric meets or exceeds the threshold metric.

[0106] Upon returning to operation 710, operation 710 may determine that an alternative component is available. For example, this determination may include a determination that a modified version of the component is available in the repository and / or a determination that an instruction indicating a change to the pipeline configuration has been received from the user interface. In additional or alternative examples, the user may supply the alternative component directly (e.g., by storing the component in a memory accessible to the computing device executing the exemplary process 700 and providing a pointer to the address associated with the component, by physically changing the hardware, by flashing the memory of an ASIC and / or FPGA). Based at least in part on determining that an alternative component is available, the exemplary process 700 may transition to operation 712.

[0107] In operation 716, the exemplary process 700 may include replacing a first component of the first system with a second component in accordance with any of the techniques discussed herein to form a second system. Thus, the second system may include the second component and other components of the first system. In some examples, operation 716 may replace one component at a time to maintain experimental control, although other replacement strategies are contemplated. For example, if the second component requires input from an additional component that is not present in the first system, replacing the first component with the second component may include swapping the first component with the second component and adding the additional component to the system and / or replacing another component of the first system with the additional component. It is contemplated that multiple components may be changed at a time.

[0108] In some examples, operation 716 may include, additionally or alternatively, requesting user permission for the replacement before replacing the first component with the second component. In another example, operation 716 may include automatically causing the replacement and subsequent operations without user input, at least partially based on determining that the alternative component is available in operation 710. For example, the exemplary process 700 may be fully automated. For example, an autonomous vehicle may periodically record sensor data and / or a sequence of events, check a remote repository, and / or send the sensor data and / or the sequence of events to a remote device including the repository. The remote device may then automatically execute the exemplary process 700.

[0109] In operation 718, the exemplary process 700 may include processing sensor data by the second system, at least partially based on operating the components of the second system according to a sequence of events according to any of the techniques discussed herein. The framework controller, input / output interface, and / or conditioning controller may operate as in operation 704, except that one of the controllers controls the second component instead of the first component. In some examples, the controller may not be constrained by this change and may not require additional data.

[0110] In operation 720, the exemplary process 700 may include receiving a second output, at least partially based on processing sensor data by the second system according to any of the techniques discussed herein. Since the framework enables both the first system and the second system to operate deterministically by exchanging the first component with the second component, the difference between the first output and the second output may be due to the exchange. This enables adjusting the performance of the system in ways that would not be possible without the framework.

[0111] In some examples, since the framework operates the system deterministically, the first output and the second output may be associated with the same identifier (e.g., the same generation time associated with a message including each output, as defined above). In some cases, the tuning model receives a first message from the playback controller (e.g., during playback of the operation of a system and / or component captured during operation of an online controller), and may identify a second message recorded during operation of the online controller having the same generation time (or other identifier) as the first message from a data store. The tuning model may then compare the first output of the first message with the second output of the second message and / or determine one or more metrics associated with the first message and the second message, as further described below.

[0112] In operation 722, exemplary process 700 may include determining a second metric associated with the second output according to any of the techniques discussed herein.

[0113] In operation 724, exemplary process 700 may determine, according to any of the techniques discussed herein, whether a second metric is an improvement over a first metric. If the second metric is not an improvement over the first metric, exemplary process 700 may return to operation 702. Additionally or alternatively, exemplary process 700 may store an indication that, with respect to the second component, the second metric was not an improvement over the first metric and / or a threshold metric. If the second metric is an improvement over the first metric (and / or a threshold metric), exemplary process 700 may transition to operation 726. The second metric may be an improvement over the first metric, the second metric may indicate an accuracy and / or precision that meets or exceeds the accuracy and / or precision indicated by the first metric, the second metric may indicate a PR curve, ROC curve, and / or acceleration that more closely matches a curve than the PR curve, ROC curve, and / or acceleration indicated by the first metric, and / or the second metric may indicate a latency and / or response time that is shorter than the latency and / or response time indicated by the first metric. A metric may include other indicators of system and / or component performance, and thus it is understood that an improvement of one metric value over another may vary depending on the metric type (e.g., the improvement may sometimes be indicated by a larger value, sometimes by a smaller value, sometimes by a higher degree of conformance to a standard and / or threshold). In some examples where a metric indicates multiple types of metrics, the various metrics may be weighted and a weighted sum of the differences may be calculated to determine whether the second metric is an improvement over the first metric.

[0114] In operation 726, exemplary process 700 may include transmitting, according to any of the techniques discussed herein, instructions to an autonomous vehicle to replace a first component with a second component.

[0115] (Exemplary Architecture) FIG. 8 shows a block diagram of an exemplary architecture that implements the frameworks discussed herein. In some cases, system 800 may include a vehicle 802 that may correspond to the autonomous vehicle 102 of FIG. 1. In some examples, vehicle 802 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the U.S. National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions throughout a trip and where a driver (or passenger) is not expected to control the vehicle at any time. However, in other examples, vehicle 802 may be a fully or partially autonomous vehicle having other levels or classifications. Additionally, in some cases, the techniques described herein may be usable by a non-autonomous vehicle.

[0116] Vehicle 802 may include a vehicle computing device 804, one or more sensors 806, one or more emitters 808, one or more network interfaces 810, at least one and / or one or more drive modules 812.

[0117] In some cases, sensor 806 may include a light detection and ranging (LIDAR) sensor, a RADAR sensor, an ultrasonic transducer, a sonar sensor, a position sensor (e.g., a global positioning system (GPS), a compass, etc.), an inertial sensor (e.g., an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, etc.), a camera (e.g., red-green-blue (RGB), infrared (IR), intensity, depth, time of flight, etc.), a microphone, a wheel encoder, an environmental sensor (e.g., a temperature sensor, a humidity sensor, a light sensor, a pressure sensor, etc.), and the like. Sensor 806 may include multiple instances of each of these or other types of sensors. For example, the lidar sensors may include individual lidar sensors located at the corners, front, rear, sides, and / or top of vehicle 802. As another example, the cameras may include multiple cameras arranged at various locations with respect to the exterior and / or interior of vehicle 802. Sensor 806 may provide inputs to vehicle computing device 804.

[0118] Vehicle 802 may also include an emitter 808 for emitting light and / or sound as described above. The emitter 808 in this example may include internal audio and visual emitters for communicating with the passengers of vehicle 802. By way of non-limiting example, the internal emitters may include speakers, lights, signs, display screens, touch screens, tactile emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.), and the like. The emitter 808 in this example may also include external emitters. By way of non-limiting example, the external emitters in this example may include lights or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.) for signaling the direction of travel, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for communicating audibly with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0119] Vehicle 802 may also include a network interface 810 that enables communication between vehicle 802 and one or more other local or remote computing devices. For example, network interface 810 may facilitate communication between vehicle 802 and other local computing devices on vehicle 802 and / or drive module 812. Additionally or alternatively, network interface 810 may enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Additionally or alternatively, network interface 810 may enable vehicle 802 to communicate with computing device 814.

[0120] Network interface 810 may include a physical and / or logical interface for connecting vehicle computing device 804 to a network such as another computing device or network 816. For example, network interface 810 may enable Wi-Fi-based communication via frequencies defined by the IEEE800.11 standard, short-range radio frequencies such as Bluetooth®, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that enables each computing device to interface with other computing devices. In some cases, vehicle computing device 804 and / or sensor 806 may transmit sensor data to computing device 814 at a particular frequency via network 816 after a predetermined period of time, in near real-time, etc.

[0121] In some cases, vehicle 802 may include one or more drive modules 812. In some cases, vehicle 802 may have a single drive module 812. In some cases, drive module 812 may include one or more sensors for detecting the state around drive module 812 and / or vehicle 802. By way of non-limiting example, sensors of drive module 812 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive module, camera or other image sensors, ultrasonic sensors for acoustically detecting objects around the drive module, LIDAR sensors, RADAR sensors, and the like. Some sensors, such as wheel encoders, may be specific to drive module 812. In some cases, sensors on drive module 812 may overlap with or supplement corresponding systems of vehicle 802 (e.g., sensor 806).

[0122] The drive module 812 may include a high-voltage battery, a motor for propelling the vehicle, an inverter that converts direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including a hydraulic or electric actuator, a suspension system including hydraulic and / or pneumatic components, a stability control system that reduces traction loss and distributes braking force to maintain control, an HVAC system, lighting (e.g., head / tail lights that illuminate the exterior perimeter of the vehicle), and many vehicle systems including one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, a DC / DC converter, a high-voltage junction, high-voltage cables, a charging system, a charging port, and other electrical components). Further, the drive module 812 may include a drive module controller that can receive data from sensors and perform preprocessing, and can control the operation of various vehicle systems. In some cases, the drive module controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more modules and can execute various functions of the drive module 812. Additionally, the drive module 812 may also include one or more communication connections that enable communication with one or more other local or remote computing devices by each drive module.

[0123] The vehicle computing device 804 may include one or more processors 818 and a memory 820 communicatively coupled to the one or more processors 818. The computing device 814 may also include a processor 822 and / or a memory 824. The processors 818 and / or 822 can be any suitable processors capable of processing data and executing instructions for performing operations as described herein. By way of non-limiting example, the processors 818 and / or 822 can include one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs), etc.), gate arrays (e.g., field programmable gate arrays (FPGAs), etc.), and / or other devices or portions of devices that process electronic data and convert that electronic data into other electronic data that can be stored in registers and / or memory.

[0124] The memory 820 and / or 824 can be examples of non-transitory computer-readable media. The memory 820 and / or 824 can store an operating system and one or more software applications, instructions, programs, and / or data, and can implement the functions resulting from the methods and various systems described herein. In various implementations, the memory can be implemented using suitable memory technologies such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying drawings are merely illustrative examples relevant to the description herein.

[0125] In some cases, memory 820 and / or memory 824 may store components 826 and / or 834, such as components of a primary perception system, a secondary perception system, a prediction system, a localization system, a mapping system, a planning system, a ride management system, etc. These (and other) systems may include various components that make up a data processing pipeline. Although components 826 and / or 834 are shown as being stored in memories 820 and / or 824, components 826 and / or 834 may include processor-executable instructions, machine learning models (e.g., neural networks), and / or hardware. In some examples, components 826 and / or 834 to be adjusted according to the frameworks discussed herein may be configured to include an input / output interface 302.

[0126] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer of a neural network can also include another neural network or can include any number of layers (regardless of whether convolutional). As can be understood in the context of the present disclosure, neural networks can utilize machine learning, which can refer to a broad class of such algorithms where outputs are generated based on learned parameters.

[0127] Although described in the context of neural networks, any type of machine learning can be used in accordance with this disclosure. For example, machine learning algorithms can include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS), example-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression tree (CART), iterative dichotomiser 3 (ID3), chi-squared automatic interaction detection (CHAID), decision stump, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, Average one-depence estimators (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-median, expectation maximization (EM), hierarchical clustering), correlation rule learning algorithms (e.g., perceptron, backpropagation, Hopfield network, radial basis function network (RBFN)), deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep belief network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixture discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machine (GBM), gradient boosting regression tree (GBRT), random forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc.Examples of additional architectures include neural networks such as ResNet70, ResNet101, VGG, DenseNet, PointNet, and the like.

[0128] Memory 820 and / or 824 may additionally or alternatively store controllers 828 and / or 836, which may include any of the controllers discussed herein. For example, controllers 828 and / or 836 may include an online controller, a playback controller, and / or an adjustment controller. In some examples, memory 820 and / or 824 may additionally or alternatively store tuning models 830 and / or 840 configured to execute at least a portion of exemplary process 700. For example, tuning model 830 may periodically query computing device 814 to determine whether alternative component 838 is available. Alternative component 838 may include a modified version of component 826, a new component, a change in pipeline configuration, and the like. If an alternative component is available, tuning model 830 may cause the online controller to generate a log file of the current configuration of the system on the vehicle that includes components corresponding to the alternative component. Once the log file is generated, tuning model 830 may send the log file of tuning model 840 stored in memory 824 to computing device 814 to execute exemplary process 700 and / or tuning model 830 may execute exemplary process 700 on the vehicle. In another example, tuning model 830 may periodically save a log file and send it to computing device 814 regardless of the availability of alternative components. Next, the computing device may use the periodically received log files to test the difference between the performance of the current configuration of the system on the vehicle and the performance of a second system incorporating the alternative component.

[0129] Memory 820 may additionally or alternatively store one or more system controllers 832, which may be configured to control steering, propulsion, braking, safety, emitters, communications, and other systems of vehicle 802. These system controllers 832 may communicate with and / or control corresponding systems of drive module 812 and / or other components of vehicle 802.

[0130] In some cases, computing device 814 may include data store 842 and / or logger 844. In some cases, data store 842 may be configured to store messages issued over a data channel (e.g., messages issued between controllers for consumption by components), and logger 844 may be configured to store messages issued over a reproducibility channel. In some cases, logger 844 may additionally or alternatively generate a log file as described herein, at least in part based on the reproducibility messages received thereby.

[0131] In some cases, component 826 may process sensor data, and an associated controller 828 may transmit respective outputs to one or more computing devices via one or more networks 816 (e.g., at a particular frequency, after a predetermined period of time, substantially in real time, etc.).

[0132] In some examples, computing device 814 may include one or more nodes of a distributed computing system (e.g., a cloud computing architecture). A framework including controller 836 associated with component 834 may enable component 834 to be executed on heterogeneous computing devices while maintaining the determinism and reproducibility of component 834.

[0133] Although FIG. 8 is shown as a distributed system, it should be noted that in an alternative example, components of vehicle 802 may be associated with computing device 814 and / or components of computing device 814 may be associated with vehicle 802. That is, vehicle 802 may perform one or more functions related to computing device 814 and vice versa.

[0134] (Exemplary section) A. Receiving sensor data from sensors of an autonomous vehicle, processing the sensor data by a first system, the first system including a first component and one or more other components, the processing being at least partially based on controlling the operation of the first component and the one or more other components, recording a sequence of events at least partially based on receiving one or more messages from a first controller associated with the first component during processing of the sensor data by the first system, receiving a first output at least partially based on processing the sensor data by the first system, obtaining a second system including a second component and one or more other components by exchanging the first component with the second component, processing the sensor data by the second system at least partially based on operating the second component according to the sequence, receiving a second output at least partially based on processing the sensor data by the second system, and sending an instruction to the autonomous vehicle to exchange the first component with the second component at least partially based on determining a difference between the first output and the second output.

[0135] B. The method of paragraph A, wherein the event includes at least one of providing an input to the first component, causing the first component to execute, or receiving an output from the first component, and causing the second component to operate according to the sequence includes at least one of providing an input, transmitting an output, or executing as specified by the sequence.

[0136] C. The method of paragraph A or B, further including determining at least one of a first metric related to the first output and a second metric related to the second output, and determining the difference includes determining that at least one of the second metrics is an improvement over the first metric.

[0137] D. The method of any one of paragraphs A - C, wherein processing sensor data by the first system is performed by a first controller associated with the first component, receiving an upstream message from one or more upstream controllers associated with one or more upstream, the message including a horizon time that identifies the period until the component is re - executed, determining, by the first controller, the order of the upstream messages, at least partially based on the sequence, determining, by the first controller, a period, at least partially based on the horizon time, providing, by the first controller, the upstream message as an input to the component in order during the period, causing, by the first controller, the first component to be executed one or more times until the period expires, receiving, by the first controller, an output from the first component in response to the execution of the first component, and issuing, by the first controller, an output message including the output, the generation time identifying the individual execution of the first component that produced the output, and the horizon time, to one or more subscribing controllers.

[0138] E. A method according to any one of paragraphs A - D, wherein one of the one or more subscribing controllers includes a simulation controller associated with a simulation component that simulates the response of the autonomous vehicle to the output message.

[0139] F. A method according to paragraph A, further comprising receiving from the first controller a key frame that identifies the internal state of the first component based at least in part on the individual execution of the first component, and wherein processing the sensor data by the second system comprises skipping to an execution of the second component that matches the execution identified by the key frame and setting the internal state of the second component to match the internal state identified by the key frame.

[0140] G. One or more processors, a first component, and a first controller associated with the first component, which, when executed by the one or more processors, cause the system to receive a sequence of events at the first component related to the operation of one or more other components, receive a first message from an upstream controller to which the first controller subscribes, the message including a first output of the execution of a second component, determine the order of one or more messages, at least in part based on the sequence, the one or more messages including the first message, provide the one or more messages to the first component as input in the order, determine when to execute the first component, at least in part based on the sequence, at that time, execute the first component, at least in part based on receiving the one or more messages at the first component, receive an output of the first component in response to the execution of the first component, and issue a message constituting the output to one or more subscribing controllers. A system including a memory storing a first controller that performs the operations including these.

[0141] H. The system of paragraph G, wherein the first message includes a first horizon time, at least one of the one or more messages includes a second horizon time, and the operations further include determining, at least in part based on the first horizon time and the second horizon time, a period during which the first component can be executed, the period being the shorter of the first horizon time or the second horizon, and the first component is executed during that period.

[0142] I. A system according to paragraph G or H, wherein the first system includes a first component and one or more other components, and the memory, when executed by one or more processors, causes the system to replace the first component with a third component to obtain a second system including the third component and one or more other components, and further includes instructions to repeat an operation on the second system using the third component, the operation including receiving, in response to execution of the third component, a second output of the third component during a period and issuing a second message including the second output to one or more subscribing controllers.

[0143] J. A system according to any one of paragraphs G-I, wherein the first system determines a first result at least in part based on a first message, the second system determines a second result at least in part based on a second message, and the memory, when executed by one or more processors, causes the processor to determine a first metric at least in part based on the first result and a second metric at least in part based on the second result, and further includes instructions to send an instruction to the autonomous vehicle to replace the first component with a third component based at least in part on determining that the second metric is an improvement over the first metric.

[0144] K. A system according to any one of paragraphs G-J, wherein one of the one or more subscribing controllers is associated with a simulation component that simulates the response of the autonomous vehicle to one or more messages received from one or more controllers associated with one or more components in the simulation component.

[0145] A system according to any one of paragraphs G-K, wherein the operation further includes preventing a first component from being executed upon expiration of at least one period, or determining that a sequence indicates that at least one other component must be executed before the first component is executed.

[0146] A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to receive log data including sensor data and a sequence of events, execute at least two components in relation to each other according to the sequence, the two components being associated with two controllers that execute the at least two components, receive a first output of a first system including the at least two components, at least in part based on executing the at least two components, replace one of the at least two components with a replacement component to form a second system, execute the components of the second system according to the sequence, receive a second output of the second system, at least in part based on executing the components of the second system, determine a difference between the first output and the second output, and transmit instructions for using the replacement component in an autonomous vehicle, at least in part based on determining the difference between the first output and the second output.

[0147] A non-transitory computer-readable medium of paragraph N, causing at least two components to be executed according to a sequence includes at least one of determining an order of providing an input to a component, providing an output from a component to another component of at least two components, or providing an input to another component, providing an output from another component to one of at least one component or an additional component, or causing a component to be executed in relation to at least one of causing another component to be executed.

[0148] A non-transitory computer-readable medium of paragraph O, determining a difference includes determining a first metric of a first output and a second metric of a second output, and determining that the second metric is an improvement over the first metric.

[0149] A non-transitory computer-readable medium of any one of paragraphs M - O, when the instructions are executed by one or more processors, further causing the one or more processors to receive a keyframe that serializes an internal state of a first component of at least two components during execution, at least in part based on causing the first component to be executed, after receiving the keyframe, causing the internal state of the first component to be set to match the internal state of the keyframe, and causing the execution of the first component to be reproduced at least in part based on the internal state.

[0150] Q. A non-transitory computer-readable medium of any one of paragraphs M-P, wherein when the instructions are executed by one or more processors, the processors are further caused to store at least one of a non-deterministic variable or a clock provided as an input to at least one of the at least two components during the execution of the at least two components, and causing a component of a second system to execute includes reading at least one of the non-deterministic variable or the clock data and providing at least one of the non-deterministic variable or the clock data to at least one of the components of the second system during the execution of at least one of the components.

[0151] R. A non-transitory computer-readable medium of any one of paragraphs M-Q, wherein the at least two components include a first component and a second component, a first controller is associated with the first component, a second controller is associated with the second component, and causing the first component to execute according to a sequence includes, at the first controller, receiving a first message from the second controller, the first message including an output of the second component and a first horizon time, determining that the first horizon time is shorter than another horizon time of another message, and determining to cause the first component to execute before the expiration of the first horizon time.

[0152] S. A non-transitory computer-readable medium of any one of paragraphs M-R, wherein when the instructions are executed by one or more processors, the one or more processors are further caused to receive an output from a second component in response to the second component being executed, generate a second message including the time when the output was received from the second component and a second horizon time indicating a period, and issue the second message to one or more subscribing controllers.

[0153] A non-transitory computer-readable medium of any one of paragraphs T-M-S, wherein at least two components include a second component and a third component, a first controller is associated with the second component, a second controller is associated with the third component, the second component generates an output usable by a computing device to control the operation of the autonomous vehicle, the first controller transmits a message including the output to the second controller, the second controller receives the message, provides the output as an input to the third component, and the third component includes a simulation component that emulates a response of the autonomous vehicle and outputs simulated data.

[0154] U. The non-transitory computer-readable medium stores a first software component including an input interface configured to receive, from a controller, instructions for executing a first software component and one or more variables, and a first instruction that, when executed by one or more processors, causes the one or more processors to execute the first software component in response to receiving the instructions, at least partially based on the one or more variables, and to receive a result of the execution of the first software component, and an output interface configured to output the result to the controller. The controller includes a second instruction that, when executed by one or more processors, causes the one or more processors to control values of the one or more variables provided to the input interface, the result of the execution is at least partially based on the values, and the first software component executes at least partially based on at least partially transmitting instructions to the input interface, and to perform an operation including receiving the result from the output interface.

[0155] V. A non-transitory computer-readable medium of paragraph U, which, when executed by one or more processors, further includes a third instruction that causes the one or more processors to exchange a first software component with a second software component, and the second software component includes an input interface and an output interface.

[0156] W. A non-transitory computer-readable medium of paragraph U or V, wherein the second instruction, when executed by one or more processors, further causes the one or more processors to provide, as an input via the input interface, at least one of a clock signal, sensor data received from sensors of the autonomous vehicle, or a non-deterministic value that can be used for execution by the first software component.

[0157] X. A non-transitory computer-readable medium of any one of paragraphs U-W, wherein the second instruction, when executed by one or more processors or one or more second processors, further causes the one or more processors to receive, from an upstream controller associated with a third software component, a second message including a second result of the third software component, and provide the second result as one of one or more variables as an input via the input interface.

[0158] Y. A non-transitory computer-readable medium of any one of paragraphs U-X, wherein the first instruction, when executed by one or more processors, further causes the one or more processors to serialize an internal state of the first software component as a key frame, and the output interface is configured to output the key frame to a controller.

[0159] A non-transitory computer-readable medium of any one of paragraphs U-Y, wherein when the first instruction is executed by one or more processors, the one or more processors are further caused to serialize the internal state at least periodically or at least partially based on determining that the degree of change in the internal state meets or exceeds a threshold degree of change.

[0160] A non-transitory computer-readable medium of any one of paragraphs U-Z, wherein when the first instruction is executed by one or more processors, the one or more processors are further caused to skip to the individual execution of the first software component specified by the key frame following the serialization of the key frame, deserialize the key frame, set the internal state of the first software component to match the internal state serialized by the key frame, and start from the individual execution and execute the first software component.

[0161] AB. A system comprising one or more processors and a memory, the memory being a first component including instructions executable by a computing device, the instructions being a first component that generates a result that is at least partially used by the computing device to control at least one operation of a machine control device or a data processing pipeline, a first input interface configured to receive one or more variables, a second input interface configured to receive an instruction for executing the instructions, and a first output interface configured to output a result based at least in part on receiving the one or more variables and the instruction. The system further includes a first controller associated with the first component and including a first instruction, which, when executed by the one or more processors, causes the one or more processors to control, via the first input interface, the order in which the one or more variables are provided to the first input interface, cause the first component to be executed via the second input interface when the instruction is sent to the second input interface, cause the result to be received via the first output interface, and issue, to one or more downstream controllers, a message including the result and the time at which the result was received by the first controller.

[0162] AC. The system of paragraph AB, wherein the first controller is an online controller, and the first instruction, when executed by the one or more processors, further causes the one or more processors to store in a log file the order of events at the interface for one or more events in another component, the events including at least one of providing the one or more variables to the first input interface, causing the first component to be executed via the second input interface, or receiving an output from the first component via the first output interface.

[0163] AD. A system of paragraph AB or AC, wherein when the first instruction is executed by one or more processors, the one or more processors are further caused to store in a log file at least one of sensor data received from sensors of an autonomous vehicle, non-deterministic data consumed by a first component, or diagnostic information regarding the execution of the first component.

[0164] AE. A system of any one of paragraphs AB - AD, wherein the first controller is a playback controller, and when the first instruction is executed by one or more processors, the one or more processors are further caused to receive a log file including a sequence of events in one or more components, the sequence of events defines the order in which events occurred due to one or more components having been previously executed, and based at least in part on determining that receiving or executing one or more variables would disrupt the sequence, cause the first component to be prevented from receiving or executing at least one of receiving or executing one or more variables, providing one or more variables, executing the first component, or issuing a message at least in part based on the sequence.

[0165] AF. A system of one of paragraphs AB - AE, wherein the interface further includes a second output interface that serializes a first internal state of the first component as a first keyframe and outputs the first keyframe, the first internal state including at least one of non-deterministic variable values, counter values, output contexts, or internal variable values based at least in part on a previous execution of the first component, and a third input interface that receives a second keyframe, deserializes the second keyframe into a second internal state, and sets the internal state of the first component to the second internal state.

[0166] A system of one of paragraphs AG - AF, wherein when the first instruction is executed by one or more processors, the one or more processors are further caused to provide, as an input via a first input interface, at least one of a clock signal, sensor data received from sensors of an autonomous vehicle, or a non - deterministic value usable by a first software component.

[0167] AH. Controlling, by a first controller, values of one or more variables provided to an input interface associated with a first software component; determining, by the first controller, a time to execute the first software component based at least in part on at least one of receipt of a message, a specified period, a sequence of events, or a horizon time; at that time, causing, by the first controller, the first software component to be executed, wherein execution of the first software component is based at least in part on one or more variables; receiving, by the first controller, a first output of the execution from the first software component; and issuing, by the first controller, to one or more subscribers, a first message including at least one of the first output, a period before the first controller causes the first software component to be re - executed, or a generation time indicating when the first output was received by the first controller or generated by the first software component.

[0168] A method according to paragraph AH, comprising sensor data provided as input to an input interface, a sequence of events, where saving the sequence of events includes monitoring the operation of a first software component for one or more events in another software component, the events including at least one of receiving a message, issuing a first message, providing an input to the input interface, providing a start of execution, providing a completion of execution, or receiving an output, a sequence of events, data provided as input to the input interface, data received as output from an output interface, or at least one of key frames generated by the first software component and output via the output interface, and further comprising saving at least one of them to a log file.

[0169] A method according to paragraph AH or AI, wherein the first software component includes first processor-executable instructions, the method further comprising replacing the first software component with a second software component such that the input interface provides one or more variables to the second software component, the second software component being executed at that time by a controller, a second output of the execution of the second software component being received via the output interface, and the second software component including second processor-executable instructions.

[0170] One of the methods of paragraphs AH - AJ, comprising receiving a first message with a second controller associated with a second software component; determining, by the second controller, a first time to provide the first message to the second software component based at least in part on a sequence of events; determining, by the second controller, a second time to execute the second software component based at least in part on the sequence; providing, by the second controller, the first message to the second software component at the first time; executing, by the second controller, the second software component one or more times at the second time; receiving, by the first controller, a second output of the execution of the second software component from the second software component; and issuing, by the second controller, a second message including the second output and at least one of a second horizon time, or when the second output was received by the second controller, or a second generation time indicating an individual execution of the second software component that generated the second output, to one or more second subscribers.

[0171] One of the methods of paragraphs AH - AK, wherein determining the time is determining that the first software component should be executed before or after another event at another controller, and the other event at the other controller is specified by a second message received from an upstream controller at the first controller or at least in part based on the sequence.

[0172] One of the methods of paragraphs AH - AL, further comprising determining a period during which the first software component can be executed, and preventing the first software component from being executed before the time and when the period expires.

[0173] One of the methods of paragraphs AH - AM, comprising: providing sensor data as one of one or more variables as an input via an input interface; and determining a time based at least in part on determining a sequence of events regarding one or more events in a first software component or at least one of one or more other components based at least in part on a log file, and further comprising processing the sensor data via the first software component, wherein the one or more events are indicated by one or more messages received by a first controller from one or more other controllers associated with one or more other components.

[0174] Receiving sensor data from sensors of an autonomous vehicle and processing the sensor data by a first system, the first system including a first component and one or more other components, the processing being at least partially based on controlling the operations of the first component and the one or more other components, recording a sequence of events at least partially based on receiving one or more messages from a first controller associated with the first component during processing of the sensor data by the first system, receiving a first output at least partially based on processing the sensor data by the first system, replacing the first component with a second component to obtain a second system including the second component and one or more other components, processing the sensor data by the second system at least partially based on operating the second component according to the sequence, receiving a second output at least partially based on processing the sensor data by the second system, and transmitting an instruction to replace the first component with the second component to the autonomous vehicle at least partially based on determining a difference between the first output and the second output.

[0175] The method of paragraph AO, wherein the event includes at least one of providing an input to the first component, causing the first component to execute, or receiving an output from the first component, and operating the second component according to the sequence includes at least one of providing an input, transmitting an output, or executing as specified by the sequence.

[0176] The method of paragraph AO or AP, further comprising determining at least one of a first metric associated with a first output and a second metric associated with a second output, and determining the difference includes determining that at least one of the second metrics is an improvement over the first metric.

[0177] The method of any one of paragraphs AO - AQ, wherein processing sensor data by a first system is by a first controller associated with a first component, receiving an upstream message from one or more upstream controllers associated with one or more upstream, the message including a horizon time, the horizon time identifying the period until the component is re - executed, determining, by the first controller, the order of the upstream messages, at least in part based on a sequence, determining, by the first controller, a period, at least in part based on the horizon time, providing, by the first controller, the upstream message as an input to the component in order during the period, causing, by the first controller, the first component to be executed one or more times until the period expires, receiving, by the first controller, an output from the first component in response to the execution of the first component, and issuing, by the first controller, an output message including the output, the generation time identifying the individual execution of the first component that produced the output, and the horizon time, to one or more subscribing controllers.

[0178] The method of any one of paragraphs AO - AR, wherein one of the one or more subscribing controllers includes a simulation controller associated with a simulation component that simulates the response of an autonomous vehicle to the output message.

[0179] A method according to paragraph AO, further comprising receiving from a first controller a keyframe identifying an internal state of a first component, at least partially based on an individual execution of the first component, wherein processing sensor data by a second system comprises skipping to an execution of a second component that matches the execution identified by the keyframe and setting an internal state of the second component to match the internal state identified by the keyframe.

[0180] A system comprising one or more processors, a first component, and a first controller associated with the first component, which, when executed by the one or more processors, causes the system to receive a sequence of events in the first component related to the operation of one or more other components, receive a first message from an upstream controller to which the first controller subscribes, the message including a first output of an execution of a second component, determine an order of one or more messages, at least partially based on the sequence, the one or more messages including the first message, provide the one or more messages to the first component as inputs, in order, determine a time to execute the first component, at least partially based on the sequence, at that time execute the first component, at least partially based on receiving one or more messages in the first component, receive an output of the first component in response to the execution of the first component, and issue a message including the output to one or more subscribing controllers.

[0181] The system of paragraph AU, wherein the first message includes a first horizon time, at least one of the one or more messages includes a second horizon time, and the operation further includes determining a period during which a first component can be executed, at least in part based on the first horizon time and the second horizon time, the period being the shorter of either the first horizon time or the second horizon, and the first component being executed during that period.

[0182] The system of paragraph AU or AV, wherein the first system includes a first component and one or more other components, and when executed by one or more processors, the memory causes the system to obtain a second system including a third component and one or more other components by replacing the first component with the third component, and further includes instructions to repeat the operation on the second system using the third component, and the operation includes receiving a second output of the third component during the period in response to execution of the third component, and issuing a second message including the second output to one or more subscribing controllers.

[0183] The system of any one of paragraphs AU - AW, wherein a first system determines a first result at least in part based on a first message, a second system determines a second result at least in part based on a second message, and when executed by one or more processors, the memory causes the processor to determine a first metric at least in part based on the first result, determine a second metric at least in part based on the second result, and send instructions to the autonomous vehicle to exchange the first component with a third component, at least in part based on determining that the second metric is an improvement over the first metric.

[0184] A system according to any one of paragraphs AU - AX, wherein one of the one or more subscribing controllers is associated with a simulation component that simulates the response of the autonomous vehicle to one or more messages received from one or more controllers associated with one or more components in the simulation component.

[0185] A system according to any one of paragraphs AU - AY, wherein the operation further includes preventing a first component from being executed upon expiration of at least one period, or determining that a sequence indicates that at least one other component must be executed before the first component is executed.

[0186] A non - transitory computer - readable medium storing processor - executable instructions that, when executed by one or more processors, cause the processor - executable instructions to cause the one or more processors to receive log data including sensor data and a sequence of events, execute at least two components in relation to each other according to the sequence, the two components being associated with two controllers that cause the at least two components to be executed, receive a first output of a first system including at least two components based at least in part on causing the at least two components to be executed, form a second system by exchanging one of the at least two components with a replacement component, execute the components of the second system according to the sequence, receive a second output of the second system based at least in part on executing the components of the second system, and transmit instructions for using the replacement component to the autonomous vehicle based at least in part on determining a difference between the first output and the second output.

[0187] BB. A non-transitory computer-readable medium of paragraph BA, causing at least two components to be executed according to a sequence includes at least one of: providing an input to a component; determining an order of providing an output from a component to another component among at least two components; causing a component to be executed with respect to at least one of providing an input to another component, providing an output from another component to at least one component or an additional component; or causing another component to be executed.

[0188] BC. A non-transitory computer-readable medium of paragraph BA or BB, determining a difference includes determining a first metric of a first output and a second metric of a second output, and determining that the second metric is an improvement over the first metric.

[0189] BD. A non-transitory computer-readable medium of any one of paragraphs BA-BC, when executed by one or more processors, the instructions further cause the one or more processors to receive a keyframe that serializes an internal state of a first component among at least two components during execution, at least in part based on causing the first component to be executed; after receiving the keyframe, set the internal state of the first component to match the internal state of the keyframe; and reproduce the execution of the first component at least in part based on the internal state.

[0190] A non-transitory computer-readable medium of any one of paragraphs BA-BD, which, when executed by one or more processors, further causes the processors to further store at least one of a non-deterministic variable or a clock provided as an input to at least one of the at least two components during the execution of the at least two components, and executing a component of a second system includes reading at least one of the non-deterministic variable or the clock data, and providing at least one of the non-deterministic variable or the clock data to at least one of the components of the second system during the execution of at least one of the components.

[0191] A non-transitory computer-readable medium of any one of paragraphs BA-BE, wherein the at least two components include a first component and a second component, the first controller is associated with the first component, the second controller is associated with the second component, and causing the first component to be executed according to a sequence includes, at the first controller, receiving a first message from the second controller, the first message including an output of the second component and a first horizon time, determining that the first horizon time is shorter than another horizon time of another message, and further determining to execute the first component before the expiration of the first horizon time.

[0192] A non-transitory computer-readable medium of any one of paragraphs BA-BF, which, when executed by one or more processors, further causes the one or more processors to receive an output from a second component in response to the second component being executed, generate a second message including the time when the output was received from the second component and a second horizon time indicating a period, and issue the second message to one or more subscribing controllers.

[0193] A non-transitory computer-readable medium of any one of paragraphs BH-BG, wherein at least two components include a second component and a third component, a first controller is associated with the second component, a second controller is associated with the third component, the second component generates an output usable by a computing device to control the operation of an autonomous vehicle, the first controller transmits a message including the output to the second controller, the second controller receives the message and provides the output as an input to the third component, and the third component includes a simulation component that emulates a response of the autonomous vehicle and outputs simulated data.

[0194] BI. The non-transitory computer-readable medium stores a first software component including an input interface configured to receive, from a controller, instructions for executing a first software component and one or more variables, and a first instruction that, when executed by one or more processors, causes the one or more processors to execute the first software component in response to receiving the instructions, at least partially based on the one or more variables, and to receive a result of the execution of the first software component, and an output interface configured to output the result to the controller. The controller includes a second instruction that, when executed by one or more processors, causes the one or more processors to control values of the one or more variables provided to the input interface, the result of the execution is at least partially based on the values, and the first software component is executed at least partially based on the first software component transmitting instructions to the input interface at least partially, and to perform operations including receiving the result from the output interface.

[0195] A non-transitory computer-readable medium of paragraph BI, which, when executed by one or more processors, further includes a third instruction for causing the one or more processors to exchange a first software component with a second software component, and the second software component includes an input interface and an output interface.

[0196] A non-transitory computer-readable medium of paragraph BI or BJ, wherein the second instruction, when executed by one or more processors, further causes the one or more processors to provide, as an input via the input interface, at least one of a clock signal, sensor data received from sensors of the autonomous vehicle, or a non-deterministic value that can be used for execution by the first software component.

[0197] A non-transitory computer-readable medium of any one of paragraphs BI - BK, wherein the second instruction, when executed by one or more processors or one or more second processors, further causes the one or more processors to receive, from an upstream controller associated with a third software component, a second message including a second result of the third software component, and provide the second result as one of one or more variables as an input via the input interface.

[0198] A non-transitory computer-readable medium of any one of paragraphs BI - BL, wherein the first instruction, when executed by one or more processors, further causes the one or more processors to serialize the internal state of the first software component as a key frame, and the output interface is configured to output the key frame to the controller.

[0199] A non-transitory computer-readable medium of any one of paragraphs BI-BM, wherein when the first instruction is executed by one or more processors, the one or more processors are further caused to serialize the internal state, at least one of which is periodically or at least partially based on determining that the degree of change in the internal state meets or exceeds a threshold degree of change.

[0200] A non-transitory computer-readable medium of any one of paragraphs BI-BN, wherein when the first instruction is executed by one or more processors, the one or more processors are further caused to skip to the individual execution of the first software component specified by the keyframe following the serialization of the keyframe, deserialize the keyframe, set the internal state of the first software component to match the internal state serialized by the keyframe, and start from the individual execution and execute the first software component.

[0201] BP. A system comprising one or more processors and a memory, wherein the memory is a first component including instructions executable by a computing device, the instructions being a first component that generates a result at least partially used by the computing device to control at least one operation of a machine control device or a data processing pipeline, a first input interface configured to receive one or more variables, a second input interface that receives an instruction for executing the instructions, and a first output interface that outputs a result based at least in part on receiving the one or more variables and the instruction; and a first controller associated with the first component and including a first instruction, which, when executed by the one or more processors, causes the one or more processors to control, via the first input interface, the order in which the one or more variables are provided to the first input interface, cause the first component to be executed via the second input interface when the instruction is sent to the second input interface, cause the result to be received via the first output interface, and issue, to one or more downstream controllers, a message including the result and the time at which the result was received by the first controller.

[0202] BQ. The system of paragraph BP, wherein the first controller is an online controller, and the first instruction, when executed by the one or more processors, further causes the one or more processors to store, in a log file, the order of events at the interface for one or more events in another component, the events including at least one of providing one or more variables to the first input interface, causing the first component to be executed via the second input interface, or receiving an output from the first component via the first output interface.

[0203] A system of paragraph BP or BQ, wherein when the first instruction is executed by one or more processors, the one or more processors are further caused to store in a log file at least one of sensor data received from sensors of an autonomous vehicle, non-deterministic data consumed by a first component, or diagnostic information regarding the execution of the first component.

[0204] A system of any one of paragraphs BP - BR, wherein the first controller is a playback controller, and when the first instruction is executed by one or more processors, the one or more processors are further caused to receive a log file including a sequence of events in one or more components, the sequence of events defining the order in which events occurred due to one or more components having been previously executed, and based at least in part on determining that receiving or executing one or more variables would disrupt the sequence, prevent the first component from receiving or executing at least one of the one or more variables, cause at least one of providing one or more variables, executing the first component, or issuing a message based at least in part on the sequence.

[0205] A system of one of paragraphs BP - BS, wherein the interface further includes a second output interface that serializes the first internal state of the first component as a first keyframe and outputs the first keyframe, the first internal state including at least one of a non-deterministic variable value, a counter value, an output context, or an internal variable value based at least in part on a previous execution of the first component, and a third input interface that receives a second keyframe, deserializes the second keyframe into a second internal state, and sets the internal state of the first component to the second internal state.

[0206] A system of one of paragraphs BP - BT of BU, where when the first instruction is executed by one or more processors, the one or more processors are further caused to provide, as input via a first input interface, at least one of a clock signal, sensor data received from sensors of an autonomous vehicle, or a non - deterministic value available to a first software component.

[0207] BV. Controlling, by a first controller, the values of one or more variables provided to an input interface associated with a first software component; determining, by the first controller, the time to execute the first software component based at least in part on at least one of receipt of a message, a specified period, a sequence of events, or a horizon time; at that time, causing, by the first controller, the first software component to be executed, where the execution of the first software component is based at least in part on one or more variables; receiving, by the first controller, a first output of the execution from the first software component; and issuing, by the first controller, to one or more subscribers, a first message including at least one of the first output and the period before the first controller causes the first software component to be re - executed, or the generation time indicating when the first output is received by the first controller or generated by the first software component.

[0208] The method of paragraph BV of BW, which includes sensor data provided as an input to an input interface, a sequence of events, and monitoring the operation of a first software component for one or more events in another software component. The events include at least one of receiving a message, issuing a first message, providing an input to the input interface, starting execution, providing completion of execution, or receiving an output. The method further includes storing in a log file at least one of the data provided as an input to the input interface, the data received as an output from the output interface, or a key frame generated by the first software component and output via the output interface.

[0209] The method of paragraph BV or BW, where the first software component includes first processor-executable instructions. The method further includes replacing the first software component with a second software component such that the input interface provides one or more variables to the second software. The second software component is executed at that time by a controller, and a second output of the execution of the second software component is received via the output interface. The second software component includes second processor-executable instructions.

[0210] One of the methods of paragraphs BV - BX, comprising receiving a first message with a second controller associated with a second software component; determining, by the second controller, a first time to provide the first message to the second software component based at least in part on a sequence of events; determining, by the second controller, a second time to execute the second software component based at least in part on the sequence; providing, by the second controller, the first message to the second software component at the first time; executing, by the second controller, the second software component one or more times at the second time; receiving, by the first controller, a second output of the execution of the second software component from the second software component; and issuing, by the second controller, a second message including the second output and at least one of a second horizon time, or when the second output was received by the second controller, or a second generation time indicating an individual execution of the second software component that generated the second output, to one or more second subscribers.

[0211] BZ. One of the methods of paragraphs BV - BY, wherein determining the time comprises determining that the first software component should be executed before or after another event at another controller, and the another event at the another controller is specified by a second message received from an upstream controller at the first controller or is based at least in part on the sequence.

[0212] CA. One of the methods of paragraphs BV - BZ, further comprising determining a period for which the first software component is to be executed and preventing the first software component from being executed before the time and when the period expires.

[0213] One of the methods of paragraph BV-CA of CB, which at least partially based on providing sensor data as one of one or more variables as an input via an input interface, processes the sensor data via a first software component, and determines time at least partially based on determining a sequence of events regarding one or more events in the first software component or one of one or more other components at least partially based on a log file, wherein one or more events are further indicated by one or more messages received by the first controller from one or more other controllers associated with one or more other components.

[0214] (Conclusion) Although the subject matter of the invention has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter of the invention as defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

[0215] The modules described herein may be stored on any type of computer-readable medium and represent instructions that may be implemented in software and / or hardware. All of the above methods and processes may be fully automated via software code modules and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof. Alternatively, some or all of the methods may be embodied in specialized computer hardware.

[0216] Unless otherwise expressly stated, conditional terms such as "is possible", "obtain", "can", or "may" are understood, inter alia, within the context of presenting an example that includes, among other things, a particular feature, element, and / or step that other examples do not include. Thus, such conditional terms generally do not imply that a particular feature, element, and / or step is required in any way by one or more examples, or that one or more of those examples necessarily include the logic for determining whether a particular feature, element, and / or step should be included in, or executed in, any particular example, whether with or without user input, or a prompt.

[0217] Connectives such as the phrase "at least one of X, Y, or Z" are to be understood, unless otherwise expressly stated, to mean that items, terms, etc. may be any one of X, Y, or Z, or any combination thereof, including sets of their respective elements. Unless expressly described as singular, "a" means both singular and plural.

[0218] Any description, element, or block of a routine in the flow diagrams described herein and / or shown in the accompanying figures is to be understood as potentially representing a module, segment, or portion of code that includes one or more computer-executable instructions for implementing a particular logical function or element in the routine. Alternative implementations are within the scope of the examples described herein, in which elements or functions may be executed out of order, deleted, or executed substantially simultaneously or in reverse order, as would be understood by one of ordinary skill in the art, depending on the relevant functions. Further, such elements and / or blocks in the flow diagrams, when drawn continuously, may alternatively be arranged in any order and / or executed in parallel.

[0219] It should be emphasized that many variations and modifications can be made to the above examples, and those elements should be understood as being among other acceptable examples. All such changes and variations are intended to be included herein within the scope of the present disclosure and protected by the following claims.

Claims

1. Controlling, by a first controller, values ​​of one or more variables provided to an input interface associated with the first software component; determining, by the first controller, a time to cause the first software component to execute based at least in part on at least one of receipt of a message, a specified time period, a sequence of events, or a horizon time indicating the earliest time before the first software component will execute again; causing, at said time, by said first controller, execution of said first software component, wherein execution of said first software component is based at least in part on said one or more variables; receiving, at the first controller, a first output of the execution from the first software component; publishing, by the first controller, to one or more subscribers, a first message including the first output and at least one of the horizon time, a period of time until the first controller causes the first software component to re-execute, or a generation time indicating when the first output was received at the first controller or generated by the first software component; The method includes:

2. sensor data provided as input to the input interface; a sequence of events, wherein storing the sequence of events includes monitoring operation at the first software component for one or more events at another software component, the events including at least one of receiving a message, issuing the first message, providing an input to the input interface, starting the execution, completing the execution, or receiving the output; data provided as input to said input interface; Data received as output from an output interface, or The keyframes generated by the first software component and output via the output interface. The method of claim 1 , further comprising saving at least one of the following in a log file:

3. the first software component includes first processor-executable instructions; The method comprises: replacing the first software component with a second software component such that the input interface provides one or more variables to the second software component such that the second software component is executed by the controller at the time, wherein a second output of execution of the second software component is received via an output interface; The method of claim 1 or 2, wherein the second software component comprises second processor-executable instructions.

4. receiving the first message at a second controller associated with a second software component; determining, by the second controller, a first time to provide the first message to the second software component based at least in part on a sequence of events; determining, by the second controller, a second time to cause execution of the second software component based at least in part on the sequence; providing, by the second controller at the first time, the first message to the second software component; causing the second software component to be executed one or more times by the second controller at the second time; receiving, at the first controller, from the second software component, a second output of the execution of the second software component; publishing, by the second controller, a second message to one or more second subscribers, the second message including the second output and at least one of a second horizon time, or when the second output was received at the second controller, or a second generation time indicative of a separate execution of the second software component that generated the second output; The method of claim 1 further comprising:

5. 5. The method of claim 1, wherein determining the time comprises determining that the first software component should execute before or after another event at another controller, the other event at the other controller being specified by a second message received at the first controller from an upstream controller or based at least in part on the sequence.

6. determining a time period during which the first software component is allowed to execute; preventing the first software component from executing before the time and upon expiration of the period; The method of claim 1 , further comprising:

7. providing sensor data as one of the one or more variables as input via an input interface; and determining the time based at least in part on determining a sequence of events for one or more events at at least one of the first software component or one or more other components based at least in part on a log file; processing the sensor data via the first software component based at least in part on 7. The method of claim 1, wherein the one or more events are indicated by one or more messages received at the first controller from one or more other controllers associated with the one or more other components.

8. one or more processors; A non-transient memory for storing processor-executable instructions; the processor-executable instructions, when executed by the one or more processors, cause the system to: Controlling, by a first controller, values ​​of one or more variables provided to an input interface associated with the first software component; determining, by the first controller, a time to cause the first software component to execute based at least in part on at least one of receipt of a message, a specified time period, a sequence of events, or a horizon time indicating the earliest time before the first software component will execute again; causing, at said time, by said first controller, execution of said first software component, wherein execution of said first software component is based at least in part on said one or more variables; receiving, at the first controller, a first output of the execution from the first software component; publishing, by the first controller, to one or more subscribers, a first message including the first output and at least one of the horizon time, a period of time until the first controller causes the first software component to re-execute, or a generation time indicating when the first output was received at the first controller or generated by the first software component; A system that causes a system to perform an operation including:

9. The operation includes: sensor data provided as input to the input interface; a sequence of events, wherein storing the sequence of events includes monitoring operation at the first software component for one or more events at another software component, the events including at least one of receiving a message, issuing the first message, providing an input to the input interface, starting the execution, completing the execution, or receiving the output; data provided as input to said input interface; Data received as output from an output interface, or The keyframes generated by the first software component and output via the output interface. The system of claim 8 , further comprising saving at least one of the following in a log file:

10. the first software component includes second processor-executable instructions; The operation includes: replacing the first software component with a second software component such that the input interface provides one or more variables to the second software component such that the second software component is executed by the controller at the time, wherein a second output of execution of the second software component is received via an output interface; The system of claim 8 or 9, wherein the second software component comprises a third processor-executable instruction.

11. The operation includes: receiving the first message at a second controller associated with a second software component; determining, by the second controller, a first time to provide the first message to the second software component based at least in part on a sequence of events; determining, by the second controller, a second time to cause execution of the second software component based at least in part on the sequence; providing, by the second controller at the first time, the first message to the second software component; causing the second software component to be executed one or more times by the second controller at the second time; receiving, at the first controller, from the second software component, a second output of the execution of the second software component; publishing, by the second controller, a second message to one or more second subscribers, the second message including the second output and at least one of a second horizon time, or when the second output was received at the second controller, or a second generation time indicative of a separate execution of the second software component that generated the second output; The system of claim 9 further comprising:

12. 12. The system of claim 8, wherein determining the time includes determining that the first software component should execute before or after another event at another controller, the other event at the other controller being specified by a second message received at the first controller from an upstream controller or based at least in part on the sequence.

13. The operation includes: determining a time period during which the first software component is allowed to execute; preventing the first software component from executing before the time and upon expiration of the period; The system of claim 8 , further comprising:

14. The operation includes: providing sensor data as one of the one or more variables as input via an input interface; and determining the time based at least in part on determining a sequence of events for one or more events at at least one of the first software component or one or more other components based at least in part on a log file; processing the sensor data via the first software component based at least in part on 14. The system of claim 8, wherein the one or more events are indicated by one or more messages received at the first controller from one or more other controllers associated with the one or more other components.

Images