Detecting objects and determining their behavior

Abstract

One embodiment provides a method. The method includes receiving first input data representing an environment at the input of a first machine learning model. The method further includes determining a set of objects within the environment based on the first input data using the first machine learning model. The method further includes determining a set of behaviors for a second set of objects using a second machine learning model. The input of the second machine learning model is coupled to a set of intermediate layers of the first machine learning model. The object set and the set of behaviors for the object set are determined at least partially simultaneously.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to machine learning models. More specifically, embodiments of this disclosure relate to detecting objects in an environment and determining the behavior of those objects. Background Technology

[0002] As devices become increasingly complex and more devices operate autonomously (e.g., autonomous vehicles (AVs)), machine learning (ML) models, artificial intelligence (AI) models, and the like are frequently used to control the operation of these complex and / or autonomous devices. Machine learning models can also be used to generate inferences, actions, decisions, etc., based on data received from various devices (e.g., based on sensor data such as digital images / videos). Attached Figure Description

[0003] Embodiments of this disclosure are illustrated by way of example rather than limitation in the accompanying drawings, wherein the same reference numerals denote the same elements.

[0004] Figure 1 This is a diagram illustrating an example of a neural network according to one or more embodiments of the present disclosure.

[0005] Figure 2 This is a diagram illustrating an example of a neural network according to one or more embodiments of the present disclosure.

[0006] Figure 3 This is a block diagram illustrating an example of a perception / prediction system according to one or more embodiments of the present disclosure.

[0007] Figure 4 This is a block diagram illustrating an example of a perception / prediction system according to one or more embodiments of the present disclosure.

[0008] Figure 5 These are example diagrams illustrating one or more embodiments of the present disclosure.

[0009] Figure 6 A flowchart illustrating an example of a process for detecting an object and / or determining the behavior of an object according to one or more embodiments of the present disclosure.

[0010] Figure 7 A block diagram illustrating an example of a computing device according to one or more embodiments of the present disclosure. Detailed Implementation

[0011] Various embodiments and aspects of this disclosure will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and drawings are illustrative of the disclosure and should not be construed as limiting the disclosure. Numerous specific details are described herein to provide a thorough understanding of various embodiments of this disclosure. However, in some cases, well-known or conventional details have not been described for the sake of a concise discussion of embodiments of this disclosure.

[0012] In this specification, the phrase "an embodiment" or "an embodiment" refers to a particular feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this disclosure. The phrase "in an embodiment" appearing in different places in the specification does not necessarily refer to the same embodiment.

[0013] As mentioned above, machine learning models are frequently used to enable devices / machines to operate autonomously. Furthermore, machine learning models can also be used to generate inferences, actions, decisions, etc., based on data received from various devices. Machine learning models can be used in vehicles, such as autonomous vehicles and semi-autonomous vehicles, to control vehicle operation. For example, machine learning models can be used to detect objects, such as pedestrians, and / or predict pedestrian behavior (e.g., predicting the direction of pedestrian movement). Machine learning models are often combined sequentially. For example, the output of a first learning model used to detect objects can be fed as input to a second machine learning model used to predict the behavior of objects. This can increase the time it takes for the machine learning model to detect objects and predict their behavior. The increased time for object detection and behavior prediction can reduce the reliability and / or safety of the vehicle. For example, an autonomous vehicle may need more time to detect and avoid pedestrians crossing the road.

[0014] Therefore, reducing the latency of machine learning models in detecting objects and / or predicting their behavior can be useful. The embodiments, implementations, and / or examples described herein allow machine learning models to detect objects and predict their behavior simultaneously (e.g., in parallel). By detecting objects and predicting their behavior, machine learning models can generate outputs more quickly to control vehicle operation. This can improve vehicle safety and reliability.

[0015] Figure 1 This is a diagram illustrating an example neural network 100 according to one or more embodiments of the present disclosure. The neural network 100 can be used to model relationships between (e.g., complex) inputs and outputs, or to search for patterns in data where the dependencies between inputs and outputs may not be easily determined. The neural network 100 can also be a computational model that can be used to determine features in input data through various computations. For example, the neural network 100 can determine features (e.g., numbers, shapes, patterns, etc.) in input data (e.g., audio data, image data, video data, etc.) based on a structure that defines the sequence of computations to be performed.

[0016] Neural network 100 may be a convolutional neural network (CNN). A CNN may be a feedforward neural network. A feedforward neural network can be a type of neural network in which connections between nodes do not form loops. For example, signals, messages, data, information, etc., flow forward from left to right from the input layer 110 (e.g., from the input node) through the intermediate layer 120 to the output layer 130 (e.g., to the output node) of neural network 100. Signals, messages, data, information, etc., must not flow backward through the neural network (e.g., must not flow from right to left). CNNs can be used for image analysis. Connections and / or their associated weights may take the form of convolutional filters (and / or convolutional kernels) that can be applied to the input (e.g., to different pixels of an image). While this disclosure may relate to image analysis using CNNs, in other embodiments, CNNs may be used for other types of data and inputs.

[0017] Neural network 100 includes an input layer 110, intermediate layers 120, and an output layer 130. Each of the input layer 110, intermediate layers 120, and output layer 130 includes one or more nodes 105. Each of the input layer 110, intermediate layers 120, and output layer 130 may have a different number of nodes 105. Neural network 100 may be a deep neural network (DNN) or a deep CNN. If there is more than one intermediate layer 120 (e.g., if there are four, ten, or other appropriate numbers of intermediate layers 120), the neural network may be deep (e.g., a deep neural network). Figure 2 As shown, the neural network 100 includes two intermediate layers 120 (e.g., two columns of nodes 105). In one embodiment, the intermediate layer 120 may include nodes 105 and connections / weights coupled to the nodes 105 in the intermediate layer 120. The nodes of the intermediate layer can receive inputs (e.g., outputs generated from previous layers, such as feature maps) for the intermediate layer 120. Weights (e.g., kernels / filters) may be applied to the inputs to generate the output (e.g., feature maps) of the current intermediate layer.

[0018] Each node 105 in a layer is connected to a node 105 in the next level (e.g., the next sublayer) or a node 105 in another layer, as indicated by the arrows / lines between nodes 105. For example, a node 105 in the input layer is coupled to at least one node 105 in the first intermediate layer 120. The neural network 100 may be a fully connected neural network. For example, each node 105 in each layer or level is connected to each node in a subsequent layer or level (e.g., a node 105 in the output layer 130 is not connected to any other node).

[0019] Each connection can be associated with a weight or weight value (e.g., it can have weights). Weights or weight values ​​can define coefficients applied to the computation. For example, a weight or weight value can be a scaling factor between two or more nodes 105. Each node 105 can represent the sum of its inputs, and the weights or weight values ​​associated with a connection can represent coefficients or scaling factors multiplied by the output of the node 105 in that connection. The weights between nodes 105 can be determined, computed, generated, assigned, learned, etc., during the training of the neural network. For example, backpropagation can be used to set the weights so that the neural network 100 produces the expected output value, the corresponding value in the labeled training data. Therefore, the weights of the intermediate layer 120 can be viewed as encoding meaningful patterns in the data. The weights of the connections between nodes 105 can be modified through additional training.

[0020] Although neural network 100 is described as having a specific number of nodes 105 layers and connections, various neural network architectures / configurations can be used in other embodiments. For example, different fully connected neural networks and partially connected neural networks (e.g., all nodes in adjacent layers are not connected) can be used.

[0021] While this disclosure may relate to convolutional neural networks, other types of neural networks and / or deep neural networks may be used in other embodiments. For example, different fully connected neural networks and partially connected neural networks (e.g., all nodes in adjacent layers are not connected) may be used.

[0022] Figure 2 This is a diagram illustrating an example neural network 200 according to one or more embodiments of the present disclosure. The neural network 200 can be used to model the relationship between inputs and outputs, or to search for patterns in data where the dependencies between inputs and outputs may not be easily determined. The neural network 200 can also be a computational model that can be used to determine features in the input data through various computations.

[0023] In another embodiment, neural network 200 may be a recurrent neural network (RNN). An RNN can be a neural network where the connections between nodes can form a directed graph. An RNN can be a type of neural network where the connections between nodes can form loops and / or can proceed in a backward direction. For example, signals, messages, data, information, etc., can flow both forward (e.g., from left to right) and backward (e.g., from right to left) between nodes 205 of the neural network. The RNN may also use memory units, such as Long Short-Term Memory (LSTM) units, gated recurrent units (GRUs), etc. GRU and / or LSTM units can allow the RNN to backpropagate errors through layers over time (e.g., backward through layers or sublayers). This can allow the RNN to exhibit temporally dynamic behavior. While this disclosure may relate to sequence analysis and / or prediction using RNNs, in other embodiments, RNNs may be used for other types of data and inputs.

[0024] In one embodiment, neural network 200 may be a convolutional LSTM neural network (e.g., ConvLSTM, ConvLSTMNN, etc.). The convolutional LSTM neural network may be a type of LSTM / RNN that performs convolution operations within LSTM units. This may allow the convolutional LSTM neural network to process additional types of input data, such as video (e.g., digital video), images, etc.

[0025] Neural network 200 includes an input layer 210, an intermediate layer 220, and an output layer 230. Each of the input layer 210, intermediate layer 220, and output layer 230 includes one or more nodes 205. Each of the input layer 210, intermediate layer 220, and output layer 230 may have a different number of nodes 205. Neural network 200 may be a deep neural network or a deep RNN. If the intermediate layer 220 includes multiple levels (e.g., multiple columns of node 205, multiple sublayers of node 205), the neural network may be deep. Figure 2 As shown, neural network 200 includes two levels (or sub-layers) of node 205 (e.g., two columns of node 205). RNNs can be used for sequence analysis and / or prediction. For example, RNNs can identify patterns in data sequences, such as handwriting, text, numerical time series data, etc.

[0026] Each node 205 in the layer is connected to at least one other node, as indicated by the arrows / lines between nodes 205. Each connection may be associated with a weight or weight value (e.g., it may have a weight). The weight or weight value can define a coefficient applied to the computation. Each node 205 may represent the sum of its inputs, and the weight or weight value associated with a connection may represent a coefficient or scaling factor multiplied by the output of the node 205 in that connection. The weights between nodes 205 can be determined, computed, generated, assigned, learned, etc., during the training of the neural network. Therefore, the weights of intermediate layers can be viewed as encoding meaningful patterns in the data. The weights of the connections between nodes 205 can be modified through additional training.

[0027] Although neural network 200 is described as having a specific number of nodes 205, layers, and connections, various neural network architectures / configurations can be used in other embodiments. For example, different fully connected neural networks and partially connected neural networks (e.g., where all nodes in adjacent layers are not connected) can be used.

[0028] Figure 3 This is a diagram illustrating an exemplary perception / prediction system 300 according to one or more embodiments of the present disclosure. The perception / prediction system 300 includes a machine learning model 305 and a machine learning model 370. In one embodiment, the machine learning model 305 may be a convolutional neural network (CNN), which may be a deep neural network type / category. The machine learning model 305 (e.g., a CNN) can use convolution and pooling operations to process inputs and generate and output, such as inferences, decisions, etc. CNNs are often used to implement image analysis and / or processing. The machine learning model 305 may also be referred to as a deep neural network or a deep network. A deep neural network may be a neural network including multiple intermediate layers (e.g., multiple layers of nodes and / or weights / connections between nodes).

[0029] like Figure 3As shown, an input 310 can be provided (e.g., input, feed, etc.) to the machine learning model 305. For example, the input 310 may include one or more images (e.g., digital images, photographs, etc.) that will be processed and / or analyzed by the machine learning model 305. The input 310 may be processed by a first filter 315. The first filter 315 may also be referred to as a convolution filter. The convolution filter may include one or more kernels (e.g., convolution kernels). For example, the input (e.g., an image) may have multiple channels (e.g., multiple input channels, such as red, blue, and green input channels for each pixel of the image). The first filter 315 may include filters for each channel. The first filter 315 can be used to perform a convolution operation on the input 310. A convolution operation can refer to an operation that fuses two sets of information into an output. For example, the first filter 315 may include weights (e.g., values) that can be applied to the input to generate the output. The first filter 315 may also be referred to as a layer (e.g., an intermediate layer) of the machine learning model 305.

[0030] In one embodiment, the output generated by the first filter 315 may be a feature map 320. The feature map 320 may be the result of applying the first filter 315 (e.g., a set of weights) to the input values. For example, the feature map 320 may be the result of element-wise matrix multiplication and the sum of those results.

[0031] In one embodiment, the machine learning model 305 may further include and / or implement pooling operations that can be performed on the feature map 320. Pooling operations may refer to downsampling the feature map to reduce its height and weight while preserving the same depth. For example, max pooling (e.g., a pooling type that uses the maximum value in a pooling window) may be applied to the feature map 320. The feature map 320 may be the output of the first filter 315 (e.g., the output of a first layer) or the input provided to the second filter 325 (e.g., the input to a subsequent second layer).

[0032] The second filter 325 can receive feature map 320 (e.g., input feature map) and can apply convolution operation to feature map 320 to generate feature map 330. As described above, one or more pooling operations can be performed on feature map 330. Feature map 330 can be the output of the second filter 325 (e.g., the output of a layer) or the input provided to the third filter 335 (e.g., the input of another subsequent layer). The third filter 335 can receive feature map 330 (e.g., input feature map) and can apply convolution operation to feature map 330 to generate feature map 340. As described above, one or more pooling operations can be performed on feature map 340. Feature map 340 can be the output of the third filter 335 (e.g., the output of a layer) or the input provided to the fourth filter 345 (e.g., the input of another subsequent layer).

[0033] The fourth filter 345 can receive feature map 340 (e.g., input feature map) and can apply convolution operations to feature map 340 to generate feature map 350. As described above, one or more pooling operations can be performed on feature map 350. Feature map 350 can be the output of the fourth filter 345 (e.g., the output of a layer) or the input provided to the fully connected layer 351.

[0034] like Figure 3 As shown, the machine learning model 305 also includes fully connected layers 351 and 352. In one embodiment, the fully connected layers 351 and 352 can use the outputs of the previous layers (e.g., feature maps 320, 330, 340 and / or 350, which may be referred to as intermediate outputs, intermediate layer outputs, etc.) and can generate the final output of the machine learning model 305 (e.g., final inference, decision, etc.).

[0035] In one embodiment, machine learning model 370 can determine (e.g., predict) one or more behaviors of one or more objects within an environment. Machine learning model 370 includes layers 371, 372, and 373. Machine learning model 370 can be an RNN (e.g., a neural network where connections between nodes can form a directed graph). Machine learning model 370 can also use memory units, such as LSTM units and GRUs. GRUs and / or LSTM units can allow machine learning model 370 to backpropagate errors over time through layers (e.g., backward through layers or sublayers). This can allow machine learning model 370 to exhibit temporally dynamic behavior. In one embodiment, machine learning model 370 can be a convolutional LSTM neural network (e.g., an LSTM / RNN type that performs convolution operations within LSTM units).

[0036] like Figure 3As shown, machine learning models 305 and 370 can operate sequentially. For example, machine learning model 305 can receive input and / or input data (e.g., digital images, digital videos, etc.) at time T0. The input can pass through filters 315, 325, 335, and 345 (to generate feature maps 320, 330, 340, and 350) and a fully connected layer 351. The output of machine learning model 305 can be generated at time T5. For example, machine learning model 370 can generate a list of the locations / coordinates (e.g., starting position and height / width within an image) of one or more objects detected in the input / input data. Machine learning model 370 may not begin determining (e.g., predicting) one or more behaviors of one or more objects until after time T5, because machine learning model 370 uses the output of machine learning model 305 to determine one or more behaviors of objects. Machine learning model 370 can receive a set of objects detected by machine learning model 305 as input and can generate an output (e.g., a set of behaviors) at time T9. This results in higher latency (e.g., more delay, more time, etc.) for the perception / prediction system 300 in identifying (e.g., perceiving) objects and predicting their behavior. The higher latency of the perception / prediction system 300 may increase the amount of time it takes for an autonomous vehicle to react to objects in its environment.

[0037] Furthermore, since machine learning model 305 and machine learning model 370 are run sequentially, they can also be trained separately. For example, a first training dataset can be used to train machine learning model 305, and a different second training dataset can be used to train machine learning model 370. Because two training datasets can be used, the amount of time required to train machine learning model 305 and machine learning model 370 may increase.

[0038] Figure 4 This is a diagram illustrating an example perception / prediction system 400 according to one or more embodiments of the present disclosure. The perception / prediction system 400 includes a machine learning model 305, a machine learning model 470, and an output module 480. In one embodiment, the machine learning model 305 may be a CNN (as described above), and the machine learning model 470 may be an RNN, such as an LSTM network, a convolutional LSTM network, etc. The perception / prediction system 400 may reside on one or more computing devices and / or virtual environments (e.g., virtual machines, containers, etc.). For example, the machine learning model 305, the machine learning model 470, and the output module 480 may reside on a single computing device and / or may be distributed across multiple computing devices (machine learning model 305 resides on a first computing device, machine learning model 470 resides on a second computing device, etc.).

[0039] In one embodiment, the machine learning model 305 can receive input 310 (e.g., input data, such as digital video, digital images, etc.). For example, input 310 can be received through one or more input nodes, input layers, etc. Input 310 (e.g., input data) can be generated, captured, acquired, etc., by various sensors of the vehicle (e.g., autonomous vehicle, semi-autonomous vehicle, etc.). For example, input 310 can be generated by the vehicle's cameras, LiDAR devices / sensors, etc. Input 310 can represent the environment in which the vehicle is located. For example, input 310 can represent or depict objects that may be in the environment (e.g., pedestrians, other vehicles, cyclists, etc.), traffic control devices (e.g., stop lights, signs, lane markings / line markings, crosswalks, etc.), sidewalks, lanes / streets, vegetation (e.g., trees, shrubs, thickets, etc.), buildings, parking lots, etc.

[0040] In one embodiment, machine learning model 305 can perceive the environment in which the vehicle is located based on input 310. For example, machine learning model 305 can perceive (e.g., sense, detect, identify, etc.) objects that may be in the environment (e.g., pedestrians, other vehicles, cyclists, etc.), traffic control devices (e.g., stop lights, signs, lane markings / line markings, crosswalks, etc.), sidewalks, lanes / streets, vegetation (e.g., trees, shrubs, thickets, etc.), buildings, parking lots, etc. Machine learning model 305 can determine (e.g., identify, detect, etc.) a first set of objects (e.g., one or more objects) located in the environment based on input 310 (e.g., based on digital images, digital video, LiDAR data, etc.). For example, machine learning model 305 can detect one or more pedestrians located in the environment along with the vehicle.

[0041] In one embodiment, machine learning model 470 can determine a set of behaviors (e.g., one or more behaviors) of a set of objects detected by machine learning model 305. For example, machine learning model 470 can determine whether an object will move. In another example, machine learning model 470 can determine the direction of movement of the object (e.g., left, right, up, down, diagonal, etc.).

[0042] In one embodiment, machine learning model 305 can determine (e.g., identify, detect, etc.) one or more objects, and machine learning model 470 can determine (e.g., identify, detect, predict, etc.) one or more behaviors, with the object determination and behavior determination occurring at least partially simultaneously. For example, machine learning model 305 may receive input 310 at time T0 and generate an output identifying one or more objects at time T5. Machine learning model 470 may receive outputs generated by one or more intermediate layers of the machine learning model and may begin determining one or more behaviors between time T0 and T5. In some embodiments, machine learning model 305 and machine learning model 470 may generate outputs simultaneously. For example, machine learning model 305 may generate an output indicating a set of objects, and machine learning model 470 may generate an output indicating a set of behaviors at substantially the same time (e.g., both machine learning model 305 and machine learning model 370 may generate their respective outputs at time T5).

[0043] like Figure 4 As shown, machine learning model 470 can be coupled to a set of intermediate layers (e.g., one or more intermediate layers) of machine learning model 305. In one embodiment, the input of machine learning model 470 (e.g., the left side of machine learning model 470) can be coupled to an intermediate layer of machine learning model 305. For example, filter 315 (e.g., a layer) can generate feature map 320 (e.g., can generate an output). Feature map 320 can be provided to the input of machine learning model 470, as shown by the dashed line.

[0044] In other embodiments, the input to machine learning model 470 (e.g., the left side of machine learning model 470) may be coupled to multiple intermediate layers of machine learning model 305. For example, filter 315 (e.g., a layer) may generate feature map 320 (e.g., may generate an output), and filter 325 may generate feature map 330. Feature map 320 may be provided to the input of machine learning model 470, as shown by dashed lines. Feature map 330 may be provided to the input of machine learning model 470, as shown by dotted dashed lines.

[0045] In one embodiment, one or more intermediate layers of machine learning model 470 may be coupled to one or more intermediate layers of machine learning model 305. For example, filter 345 may generate feature map 350. Feature map 350 may be provided as input to layer 472 (e.g., an intermediate layer) of machine learning model 470, along with the output of layer 471.

[0046] In one embodiment, machine learning model 305 can generate multiple intermediate outputs (e.g., feature maps) over a period of time. For example, input 310 can be a digital video, which may include multiple frames (e.g., digital images). Each frame can be provided to machine learning model 305, and each filter can generate different feature maps for multiple frames. For example, at time T1, filter 315 can generate a first feature map based on the first frame of the digital video, at time T2, filter 315 can generate a second feature map based on the second frame of the digital video, and at time T3, filter 315 can generate a third feature map based on the third frame of the digital video. The first, second, and third feature maps (generated at times T1, T2, and T3, respectively) can be provided as input to machine learning model 470. For example, the first, second, and third feature maps can be collected together and simultaneously provided as input to machine learning model 470.

[0047] In one embodiment, multiple intermediate output sets generated by different layers at different times can be provided to the machine learning model 470. For example, filter 315 can generate a first intermediate output set (e.g., feature maps generated at times T1, T2, and T3), and filter 325 can generate a second intermediate output set (e.g., feature maps generated at times T1, T2, and T3). The first and second intermediate output sets can be provided to the machine learning model 470.

[0048] In one embodiment, output module 480 may receive a first output generated by machine learning model 305 and a second output generated by machine learning model 470. For example, output module 480 may receive both the first and second outputs simultaneously. The first output may indicate a first set of objects identified, detected, etc., by machine learning model 305. The second output may indicate a set of behaviors (e.g., a predicted behavior set, a prediction set, etc.) of the object set detected by machine learning model 305. Output module 480 may identify matches, correlations, associations, etc., between the first set of objects and the set of behaviors. For example, machine learning model 470 may determine that motion (e.g., behavior) may occur at different locations within an image (e.g., input). Output module 480 may identify behaviors that match the locations of objects (within the image) detected by machine learning model 305. Output module 480 may select, identify, determine, etc., behaviors from the set of behaviors that correspond to the locations of the object sets detected by machine learning model 305. For example, if the first set of objects includes four objects in four locations, and the machine learning model determines that movement may occur in five locations, where four locations match the four locations of the four objects, then the output module 480 can select, identify, and determine (e.g., predict) the behavior that is determined for the object in the four matching locations.

[0049] In one embodiment, machine learning model 305 may not be modifiable, changeable, etc. For example, machine learning model 305 may be a gray-box machine learning model. A gray-box machine learning model may allow modules, components, etc., outside the gray-box machine learning model to access the intermediate outputs generated by the intermediate layers of the gray-box machine learning model. A gray-box machine learning model may not allow modification of its filters (e.g., weights).

[0050] In one embodiment, the machine learning model 305 can be modifiable, changeable, etc. For example, the machine learning model 305 can be a white-box machine learning model. A white-box machine learning model can allow modules, components, etc., outside of the gray-box machine learning model to access intermediate outputs generated by the intermediate layers of the gray-box machine learning model. The white-box machine learning model can further allow the filters (e.g., weights) of the white-box machine learning model to be modified. For example, the white-box machine learning model can allow the filters of the white-box machine learning model to be modified through training and / or retraining (e.g., by passing training data to the white-box machine learning model).

[0051] In one embodiment, machine learning model 305 and machine learning model 470 can be trained simultaneously using a single training dataset. Figure 4 As shown, machine learning model 470 can receive input 310 provided to machine learning model 305. Machine learning model 470 may not directly receive training data that can be used to train machine learning model 305. However, machine learning model 470 uses intermediate outputs generated by intermediate layers of machine learning model 305, as described above. Therefore, when machine learning model 305 is trained using training data, machine learning model 470 will be trained using the intermediate outputs generated by machine learning model 305 while machine learning model 305 processes the training data. This allows machine learning module 470 to be trained without directly receiving training data.

[0052] In one embodiment, machine learning model 470 can also determine one or more attributes (e.g., characteristics, traits, etc.) of one or more objects. For example, machine learning model 470 can determine the gender, height, weight, hair color, etc. of a pedestrian in the environment. In another example, machine learning model 470 can determine the direction a pedestrian is likely facing / looking at. In yet another example, machine learning model 470 can determine the color of one or more parking lights in the environment.

[0053] As described above, machine learning model 305 and machine learning model 470 can generate their respective outputs simultaneously. For example, both machine learning model 305 and machine learning model 470 can generate their respective outputs at time T5, instead of at time T9, as shown above. Figure 3As discussed earlier. This may allow the perception / prediction system 400 to detect objects and / or predict their behavior more quickly, since machine learning model 470 and machine learning model 305 are running simultaneously. Reducing the latency of the perception / prediction system 400 in detecting objects and / or predicting their behavior may allow the vehicle to react to objects more quickly and operate more rapidly, efficiently, and / or safely.

[0054] Furthermore, compared to the perception / prediction system 300, the perception / prediction system 400 may be able to provide more accurate predictions of object behavior. The machine learning model 370 of the perception / prediction system 300 (such as...) Figure 3 (As shown) The machine learning model 305 uses objects (e.g., pedestrians) detected by the machine learning model 305 as input. For example, the machine learning model 370 may receive a portion of the image describing the object, rather than the complete image (e.g., a cropped portion of the image describing the object, instead of the entire image). Therefore, the machine learning model 370 may not be able to use features that might exist in other parts of the complete image. For example, the machine learning model 370 may not be able to identify the correlation between the color of the traffic light and the predicted behavior of the pedestrian, because the machine learning model 370 may only be provided with the portion of the image that includes the pedestrian and may not be provided with other portion of the image that includes the traffic light. Because the machine learning model 470 of the perception / prediction system 400 is using feature maps generated based on the complete image, the machine learning model 470 may be able to identify correlations between multiple objects in the environment and may be able to predict the behavior of a particular object more accurately. For example, if a pedestrian and a traffic light are detected simultaneously, and the machine learning model 470 is able to determine that the traffic light is red, then the pedestrian is more likely not to move because of the red traffic light (e.g., will not cross the road).

[0055] Furthermore, since machine learning models 305 and 470 operate concurrently, they can be trained simultaneously. For example, a training dataset can be provided to machine learning model 305. While machine learning model 305 processes the training data, the feature set generated during its training can be simultaneously provided to machine learning model 470 to train it. This reduces the amount of time required to train both machine learning models 305 and 470.

[0056] Although machine learning models 305 and 470 are discussed in terms of detecting and predicting the behavior of objects, they may implement other functions and / or make other predictions in other embodiments.

[0057] Figure 5This is a diagram illustrating exemplary image 500 according to one or more embodiments of the present disclosure. Image 500 is divided into a grid having columns 1 to 8 and rows A to F. Each grid can be referred to by a row-column pair. For example, A1 can refer to the topmost and leftmost grid. Image 500 depicts, represents, etc., the environment in which vehicles may be moving and / or are located. Image 500 depicts traffic light 501, pedestrian 502, and road 503.

[0058] In one embodiment, refer back to Figure 4 Image 500 can be the output generated by machine learning model 305 of perception / prediction system 400. As described above, machine learning model 305 can detect one or more objects in image 500. In particular, machine learning model 305 can detect pedestrian 502. Image 500 (e.g., output) can have a shape / size H×W×Anch×5. H represents the height of image 500 in terms of the number of grids (e.g., H = 6). W represents the width of image 500 in terms of the number of grids (e.g., W = 8). Anch can represent the number of anchor boxes for each grid that may have to detect objects (e.g., pedestrians). The fourth dimension can represent five values, the first of which can be the probability that an object (e.g., a pedestrian) will move in a certain direction, and the other four values ​​can be the corners of bounding box 510.

[0059] In one embodiment, image 500 may be the output generated by machine learning model 470 of perception / prediction system 400. As described above, machine learning model 470 can determine one or more behaviors of one or more objects in image 500. Image 500 (e.g., output) may have a shape / size H×W×Anch×5. H represents the height of image 500 in terms of the number of grids (e.g., H = 6). W represents the width of image 500 in terms of the number of grids (e.g., W = 8). Anch may represent the number of anchor boxes for each grid that may have to detect objects (e.g., pedestrians). The fourth dimension may indicate five values, the first value of which may indicate the probability that the first object (e.g., pedestrian) will move in a certain direction. The first value may also indicate one or more probabilities of whether the first object is a certain object type. For example, the first value may also indicate the probability that the object is a pedestrian, the probability that the object is a cyclist, the probability that the object is another vehicle, and so on. The values ​​two through five may be the corners of bounding box 510.

[0060] As described above, the output module 480 can receive the outputs generated by the machine learning model 305 and the machine learning model 470, respectively.

[0061] Output module 480 can identify matches between the location of an object detected by machine learning model 305 and the location of a movement detected by machine learning model 470. For example, output module 480 can identify pedestrian 502 as a matching object. Output module 480 can use the predicted behavior for pedestrian 502 and can generate output indicating that pedestrian 502, located within bounding box 510, may move in a certain direction (e.g., may move to the right).

[0062] Figure 6 A flowchart illustrating an example process 600 for detecting an object and / or determining the behavior of an object according to one or more embodiments of the present disclosure is provided. Process 600 may be implemented by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, processor, processing device, central processing unit (CPU), system-on-a-chip (SoC), etc.), software (e.g., instructions that run / execute on the processing device), firmware (e.g., microcode), or a combination thereof. In some embodiments, process 600 may be implemented by a computing device, a sensing / predicting system (e.g., a computing device, a sensing / predicting system, etc.), or a combination thereof. Figure 4 The perception / prediction system 400 shown), machine learning model (e.g.) Figure 4 The machine learning models 305 and 470 shown) and / or output modules (e.g.) Figure 4 One or more implementations of the output module 480 shown in s).

[0063] Reference Figure 6 Procedure 600 illustrates exemplary functionality used in various embodiments. Although specific functional blocks (“blocks”) are disclosed in Procedure 600, these blocks are exemplary. That is, embodiments are well-suited to implementing various other blocks or variations of the blocks described in Procedure 600. It is understood that blocks in Procedure 600 may be implemented in a different order than presented, and not all blocks in Procedure 600 may be implemented. Furthermore, it is possible to… Figure 6 Insert extra (in) between the blocks shown Figure 6 Other blocks (not shown in the text).

[0064] Process 600 begins at block 605, where process 600 can selectively train first and second machine learning models. The input and / or other layers of the second machine learning model can be coupled to intermediate layers of the first machine learning model, as described above. This allows the first and second machine learning models to be trained simultaneously. At block 610, process 600 can receive input data, such as images, videos, etc., at the input nodes / layers of the first machine learning model.

[0065] At block 615, process 600 can determine (e.g., detect, identify, etc.) a set of objects and can determine (e.g., predict, identify, etc.) a set of behaviors for that set of objects, wherein the determination of the set of objects and the determination of its set of behaviors are performed at least partially simultaneously as described above. Block 615 includes block 616. At block 616, the second machine learning model can receive intermediate output generated by the intermediate layer of the first machine learning model, as described above. At block 620, process 600 can determine the final set of behaviors for the set of objects. For example, the first machine learning model can detect a first set of objects. The second machine learning model can determine that motion may occur at different locations. Process 600 can determine the location of motion (e.g., behavior) that matches the location of the objects detected by the first machine learning model. The location of the motion that matches the location of the detected objects can be the final set of behaviors.

[0066] At block 625, process 600 can control the operation of one or more vehicles (e.g., autonomous vehicles, semi-autonomous vehicles, etc.). For example, process 600 can control the operation of one or more vehicles and / or control the vehicle based on object sets and behavior sets determined by the first and second machine learning models, respectively. Process 600 can control various components, systems, modules, circuits, etc., of the vehicle. For example, process 600 can control the vehicle's engine, transmission, brakes / braking system, headlights, horn, steering system, etc., based on object sets and behavior sets. Controlling the various components, systems, modules, circuits, etc., of the vehicle allows process 600 to control the movement and / or operation of the vehicle. For example, process 600 can control one or more of the brakes, steering system, and engine to prevent the vehicle from colliding with pedestrians (e.g., objects) detected by the first machine learning model.

[0067] Figure 7 This is a block diagram of an example computing device 700 according to some embodiments. The computing device 700 can be connected to other computing devices in a local area network, intranet, extranet, and / or the Internet. The computing device can operate as a server machine in a client-server network environment or as a client in a peer-to-peer network environment. The computing device can be provided as a personal computer (PC), set-top box (STB), server, network router, switch, or bridge, or any machine capable of executing a set (sequential or otherwise) of instructions specifying actions to be taken by that machine. Furthermore, although only a single computing device is described, the term "computing device" should also be considered as a collection of any computing devices that individually or jointly execute a set (or more) of instructions to implement the methods discussed herein.

[0068] Example computing device 700 may include processing device (e.g., general-purpose processor, programmable logic device (PLD) etc.) 702, main memory 704 (e.g., synchronous dynamic random access memory (DRAM), read-only memory (ROM)), static memory 706 (e.g., flash memory), and data storage device 718), which can communicate with each other via bus 730.

[0069] Processing device 702 may be provided by one or more general-purpose processing devices, such as microprocessors, central processing units, or similar devices. In illustrative examples, processing device 702 may include a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor implementing other instruction sets or combinations of instruction sets. Processing device 702 may also include one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, or similar devices. Processing device 702 may be configured to perform the operations described herein according to one or more aspects of this disclosure in order to implement the operations and steps discussed herein.

[0070] The computing device 700 may further include a network interface device 708 that can communicate with the network 720. The computing device 700 may also include a video display unit 710 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 712 (e.g., a keyboard), a cursor control device 714 (e.g., a mouse), and a sound signal generation device 716 (e.g., a speaker). In one embodiment, the video display device 710, the alphanumeric input device 712, and the cursor control device 714 may be combined into a single component or device (e.g., an LCD touchscreen).

[0071] Data storage device 718 may include computer-readable storage medium 728, on which one or more sets of sensing / prediction system instructions 725, such as instructions for performing the operations described herein, may be stored, according to one or more aspects of this disclosure. The sensing / prediction system instructions 725 may also reside wholly or at least partially in main memory 704 and / or processing device 702 during execution by computing device 700, which also constitute computer-readable media. The sensing / prediction system instructions 725 may further be transmitted or received on network 720 via network interface device 708.

[0072] Although computer-readable storage medium 728 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing the set or more sets of instructions. The term "computer-readable storage medium" should also be considered to include any medium capable of storing, encoding, or carrying a set of instructions for machine execution and causing the machine to perform the methods described herein. Therefore, the term "computer-readable storage medium" should include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0073] Additional examples, implementations, and embodiments are described in Appendix A, which is incorporated herein by reference in its entirety.

[0074] Unless otherwise specified, terms such as “receive,” “determine,” and “train” refer to actions and processes performed or implemented by a computing device that manipulate data represented as physical (electronic) quantities in the registers and memory of the computing device and convert them into other data also represented as physical quantities in the memory or registers of the computing device or other such information storage, transmission, or display devices. Furthermore, the terms “first,” “second,” “third,” “fourth,” etc., used herein are labels to distinguish different elements and do not necessarily have an ordering significance according to their numerical names.

[0075] The examples described herein also relate to devices for implementing the operations described herein. Such devices may be specifically constructed for the desired purpose, or they may comprise general-purpose computing devices selectively programmed by computer programs stored in the computing device. Such computer programs may be stored in a computer-readable, non-transitory storage medium.

[0076] The methods and illustrative examples described herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized devices to implement the required methodological steps. The necessary structures for these various systems will appear in the foregoing description.

[0077] The foregoing description is illustrative and not restrictive. Although this disclosure has been described with reference to specific illustrative examples, it will be appreciated that this disclosure is not limited to the described examples. The scope of this disclosure should be determined by referring to the following claims and all their equivalents.

[0078] As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that, when used herein, the term “comprising” specifies the presence of said features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. Therefore, the terminology used herein is for describing particular embodiments only and is not intended to be limiting.

[0079] It should also be noted that in some alternative implementations, the indicated functions / behaviors may not occur in the order shown in the diagram. For example, two diagrams shown consecutively may actually be executed substantially simultaneously, or sometimes in reverse order, depending on the functions / actions involved.

[0080] Although the method operations are described in a specific order, it should be understood that other operations may be implemented between the described operations, the described operations may be adjusted so that they occur at slightly different times, or the described operations may be distributed across a system that allows processing operations to occur within various intervals associated with the processing.

[0081] Various units, circuits, or other components may be described or claimed to be "configured to" or "configurable to" perform one or more tasks. In this context, the phrase "configured to" or "configurable to" is used to imply a structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit) that performs one or more tasks during operation. Thus, it can be said that a unit / circuit / component is configured to perform a task, or is configurable to perform a task, even if the specified unit / circuit / component is not currently running (e.g., not turned on). Units / circuit / components used with the language "configured to" or "configurable to" include hardware such as circuits, memory storing program instructions executable to perform operation, etc. The statement that a unit / circuit / component is "configured to" or "configurable to" perform one or more tasks expressly indicates that 35 U.S.SC112 paragraph 6 is not invoked for that unit / circuit / component. Furthermore, "configured to" or "configurable to" can include general structures (e.g., general-purpose circuits) manipulated by software and / or firmware (e.g., FPGAs or general-purpose processors executing software) to operate in a manner capable of performing the relevant tasks. "Configurable to" may also include adjusting the manufacturing process (e.g., a semiconductor manufacturing facility) to manufacture a device (e.g., an integrated circuit) suitable for performing or implementing one or more tasks. "Configurable to" expressly means that it does not apply to blank media, unprogrammed processors or unprogrammed general-purpose computers, or unprogrammed programmable logic devices, programmable gate arrays or other unprogrammed devices, unless accompanied by programming media that endow the unprogrammed device with the ability to configure it to perform the disclosed functions.

[0082] For purposes of explanation, the foregoing description has been given with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive, nor is it intended to limit the invention to the precise forms disclosed. In view of the foregoing teachings, many modifications and variations are possible. The embodiments were chosen and described in order to best explain the principles of the embodiments and their practical application, thereby enabling others skilled in the art to best utilize the embodiments and various modifications as they may be suited to the particular uses contemplated. Therefore, the embodiments of the invention should be considered illustrative rather than restrictive, and the invention is not limited to the details given herein, but can be modified within the scope of the appended claims and equivalent alternatives.

Claims

1. A method for controlling one or more operations of a vehicle, comprising: At the input end of the first machine learning model, the first input data representing the environment is received; The first machine learning model determines the set of objects within the environment based on the first input data; and The set of behaviors for this object set is determined using a second machine learning model, where: Couple the input of the second machine learning model with the intermediate layer set of the first machine learning model; At least partially, simultaneously determine the set of objects and the set of behaviors for the set of objects; A final set of actions is determined for the object set based on a first set of objects determined by a first machine learning model and a set of actions determined by a second machine learning model. This involves identifying actions at positions that match the positions of objects determined by the first machine learning model, the actions being determined by the second machine learning model, and selecting these identified actions as the final set of actions for the object set. Based on the set of objects and the final set of behaviors, control one or more operations of the vehicle.

2. The method according to claim 1, further comprising: At the input of the second machine learning model, an intermediate output set is received from the intermediate layer set of the first machine learning model.

3. The method according to claim 2, wherein, Each intermediate output from the intermediate output set is received from different intermediate layers of the first machine learning model.

4. The method according to claim 2, wherein, Each intermediate output in the intermediate output set is generated at different times by the same intermediate layer of the first machine learning model.

5. The method according to claim 2, wherein, Different subsets of the intermediate output set are generated by different intermediate layers at different times.

6. The method according to claim 1, wherein, The second set of intermediate layers of the second machine learning model is coupled to one or more intermediate layers of the first machine learning model.

7. The method according to claim 2, wherein: The intermediate output set includes one or more feature maps generated by one or more filters of the intermediate layer set.

8. The method according to claim 1, further comprising: The first machine learning model and the second machine learning model are trained based on the training dataset, wherein the second machine learning model is trained without receiving the training dataset at the input of the second machine learning model.

9. The method according to claim 1, wherein: The first set of objects within the environment includes one or more pedestrians within the environment; and The set of behaviors includes one or more directions of movement for one or more pedestrians.

10. The method according to claim 1, wherein, The second machine learning model also needs to determine the attribute set of the first object set.

11. A device for controlling one or more operations of a vehicle, comprising: Memory configured to store data; and A processing device coupled to a memory, the processing device being configured to perform the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to perform one or more methods for controlling the operation of a vehicle according to any one of claims 1 to 10.

Images