VEHICLE CONTROL SYSTEMS BASED ON A VEHICLE CAMERA AND PEDESTRIAN IMAGE PROCESSING

Abstract

A method for controlling the automatic acceleration and braking of a vehicle comprises obtaining an image using at least one vehicle camera of a host vehicle, extracting machine learning model feature inputs based on the obtained image, detecting one or more objects in the obtained image, wherein the one or more objects include at least one pedestrian, assigning attention weights to areas of the obtained image according to the positions of the one or more objects in the obtained image, combining the attention weights with corresponding machine learning model feature inputs according to the areas of the obtained image, executing a machine learning model to generate an output predicting a crossing intention associated with the at least one pedestrian, and responding to the output predicting a crossing intention,which exceeds a threshold for the intention to cross, triggers automatic braking of the host vehicle according to the position of at least one pedestrian.

Description

Summary

[0001] The information provided in this section serves the purpose of presenting the general context of the disclosure. The work of the inventors mentioned herein, insofar as it is described in this section, as well as aspects of the description that do not otherwise describe the prior art at the time of filing, are neither expressly nor implicitly recognized as prior art with respect to the present disclosure.

[0002] The present disclosure relates generally to vehicle control systems for controlling automatic vehicle braking, which incorporate machine learning models to predict the intention of pedestrians to cross based on image processing of a vehicle camera image.

[0003] Some vehicles are equipped with cameras configured to capture images of the area in front of the vehicle along its path. Automated driving systems can be configured to control acceleration, braking, etc., based on objects in front of the vehicle. Often, pedestrians are standing and waiting to cross a road the vehicle is traveling on, and they can pose a collision risk if a pedestrian crosses while the vehicle is moving. Summary

[0004] A method for controlling the automatic acceleration and braking of a vehicle comprises obtaining an image using at least one vehicle camera of a host vehicle, extracting machine learning model feature inputs based on the obtained image, detecting one or more objects in the obtained image, wherein the one or more objects include at least one pedestrian, assigning attention weights to areas of the obtained image according to the positions of the one or more objects in the obtained image, combining the attention weights with corresponding machine learning model feature inputs according to the areas of the obtained image, executing a machine learning model to generate an output predicting a crossing intention associated with the at least one pedestrian, and responding to the output predicting a crossing intention,which exceeds a threshold for the intention to cross, triggers automatic braking of the host vehicle according to the position of at least one pedestrian.

[0005] In other features, the assignment of attention weights includes assigning a first intensity value to a first area of ​​the received image that corresponds to at least one pedestrian, and assigning a second intensity value to a second area of ​​the received image that does not correspond to at least one pedestrian, where the first intensity value is greater than the second intensity value.

[0006] In other features, combining the attention weights and the machine learning model feature inputs involves generating a weighted sum of the machine learning model feature inputs according to the attention weights.

[0007] In other features, the machine learning model includes a multi-layered perceptron, and the execution of the machine learning model includes generating the output of a prediction of a crossing intention according to an output of the multi-layered perceptron.

[0008] In other features, extracting machine learning model feature inputs based on the obtained image involves feeding the obtained image to several visual transformer layers to generate the machine learning model feature inputs.

[0009] In other features, executing the machine learning model includes obtaining multiple key values ​​according to the machine learning model feature inputs, executing a classification query according to the machine learning model feature inputs, and correlating a classification query output with the multiple key values.

[0010] In other features, the execution of the machine learning model involves combining the attention weights with a correlation of the classification query output and the multiple key values, and running a normalized exponential function on a combination of the attention weights and the correlation of the classification query output and the multiple key values ​​to generate the output of a prediction of crossing intention.

[0011] In other features, the execution of the machine learning model includes combining an output of the normalized exponential function with the multiple key values ​​to generate an embedding vector, and feeding the embedding vector to a multilayer perceptron to produce the output of a crossing intent prediction.

[0012] Other features include feeding training and test data to the machine learning model, comparing multiple outputs of a crossing intention prediction from the machine learning model based on the training data with flagged crossing intention outputs from the test data, determining whether the accuracy of a comparison is greater than or equal to a specified accuracy threshold, adjusting parameters of the machine learning model and retraining the machine learning model in response to the finding that the accuracy of the comparison is less than the specified accuracy threshold, and storing the machine learning model for use in generating an output of a crossing intention prediction in response to the finding that the accuracy of the comparison is greater than or equal to the specified accuracy threshold.

[0013] In other features, the image encompasses at least a 45-degree field of view from at least one vehicle camera. In other features, the one or more objects include at least one pedestrian crossing, traffic light, or another vehicle.

[0014] A vehicle control system for controlling vehicle braking based on image processing from a vehicle camera comprises at least one vehicle camera configured to receive an image of an area in front of a host vehicle, and a vehicle control module of the host vehicle.The vehicle control module is configured to extract machine learning model feature inputs based on the received image, detecting one or more objects in the received image, where the one or more objects include at least one pedestrian; assign attention weights to areas of the received image according to the positions of the one or more objects in the received image; combine the attention weights with corresponding machine learning model feature inputs according to the areas of the received image; execute a machine learning model to generate a crossing intention prediction output associated with the at least one pedestrian; and, in response to the crossing intention prediction output exceeding a crossing intention threshold, control the automatic braking of the host vehicle according to the position of the at least one pedestrian.

[0015] In other features, the vehicle control module is configured to assign attention weights by assigning a first intensity value to a first area of ​​the received image that corresponds to at least one pedestrian and a second intensity value to a second area of ​​the received image that does not correspond to at least one pedestrian, with the first intensity value being greater than the second intensity value.

[0016] For other features, the vehicle control module is configured to assign the attention weights and the machine learning model feature inputs by generating a weighted sum of the machine learning model feature inputs according to the attention weights.

[0017] In other features, the machine learning model includes a multi-layered perceptron, and the vehicle control module is configured to execute the machine learning model by generating the output of a crossing intention prediction according to an output from the multi-layered perceptron.

[0018] In other features, the vehicle control module is configured to extract the machine learning model feature inputs based on the received image by feeding the received image to several visual transformer layers to generate the machine learning model feature inputs.

[0019] In other features, the vehicle control module is configured to execute the machine learning model by receiving multiple key values ​​according to the machine learning model feature inputs, performing a classification query according to the machine learning model feature inputs, and correlating a classification query output with the multiple key values.

[0020] In other features, the vehicle control module is configured to execute the machine learning model by combining the attention weights with a correlation of the classification query output and the multiple key values, and by performing a normalized exponential function on a combination of the attention weights and the correlation of the classification query output and the multiple key values ​​to generate the output of a crossing intention prediction.

[0021] In other features, the vehicle control module is configured to execute the machine learning model by combining an output of the normalized exponential function with the multiple key values ​​to generate an embedding vector, and feeding the embedding vector to a multilayer perceptron to generate the intersection prediction output.

[0022] In other features, the image includes at least a forty-five degree field of view from the at least one vehicle camera, and the one or more objects include at least one pedestrian crossing, traffic light, or other vehicle.

[0023] Further applications of the present disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and specific examples serve only for illustrative purposes and are not intended to limit the scope of the disclosure. Brief description of the drawings

[0024] The present disclosure is more fully understood from the detailed description and the accompanying drawings, whereby: Fig. 1 is a representation of an exemplary vehicle which includes a front vehicle camera configured to take pictures of pedestrians and traffic objects in front of the vehicle; Fig. 2 is a block diagram of an exemplary system for predicting the crossing intention of a pedestrian based on image processing of a vehicle camera image; Fig. 3 is a block diagram that shows an exemplary architecture of a machine learning model of the system of Fig. 2 illustrated; Fig. 4 is a flowchart illustrating an exemplary process for predicting a pedestrian's intention to cross based on image processing of a vehicle camera image: Fig. 5A and Fig. 5B graphical representations of exemplary recurrent neural networks for predicting a pedestrian's intention to cross, based on image processing of a vehicle camera image; Fig. 6 is a graphical representation of layers of an exemplary long short-term memory (LSTM) machine learning model; and Fig. Figure 7 is a flowchart illustrating an exemplary process for training a machine learning model.

[0025] Reference symbols can be reused in the drawings to identify similar and / or identical elements. Detailed description

[0026] Pedestrians often stand and wait to cross a road on which a vehicle is currently traveling, which can pose a collision risk if a pedestrian crosses while the vehicle is moving. Some exemplary embodiments described herein include systems and methods for estimating pedestrians' crossing intentions using a vehicle camera image with a large or wide field of view (FOV). The wide FOV image effectively captures the interactions between the pedestrian and various traffic objects such as the road, a pedestrian crossing, traffic lights, other vehicles, bus stops, etc.

[0027] However, the wide-field view image also includes additional pedestrians and unrelated information. The exemplary implementations described here can focus a machine learning model (e.g., a neural network) on the pedestrian and the traffic objects that are relevant for estimating the pedestrian's intention to cross.

[0028] Referring now to Fig. 1 comprises a vehicle with 10 front wheels (12) and 13 rear wheels. Fig. 1 A drive unit 14 selectively transmits torque to the front wheels 12 and / or the rear wheels 13 via drive systems 16 and 18, respectively. The vehicle 10 can contain various types of drive units. For example, the vehicle can be an electric vehicle such as a battery electric vehicle (BEV), a hybrid vehicle or a fuel cell vehicle, an internal combustion engine vehicle (ICE), or another type of vehicle.

[0029] Some examples of the drive unit 14 can include any suitable electric motor, an inverter or power inverter, and a motor controller configured to control circuit breakers within the power inverter to adjust or set the motor speed and torque during drive and / or regeneration. A battery system supplies power to or receives power from the electric motor of the drive unit 14 via the power inverter during drive or regeneration.

[0030] While the vehicle 10 in Fig. While the vehicle has one drive unit 14, it can have 10 other configurations. For example, two separate drive units can power the front wheels 12 and the rear wheels 13, one or more individual drive units can power individual wheels, etc. As can be seen, other vehicle configurations and / or drive units can also be used.

[0031] The vehicle control module 20 can be configured to control the operation of one or more vehicle components, such as the drive unit 14 (e.g., by issuing commands regarding torque settings of an electric motor in the drive unit 14). The vehicle control module 20 can receive inputs for controlling vehicle components, such as signals received from a steering wheel, accelerator paddles, a vehicle camera, etc. The vehicle control module 20 can monitor vehicle telematics data, such as vehicle speed, vehicle location, vehicle braking and acceleration, etc., for safety purposes.

[0032] The vehicle control module 20 can receive signals from any suitable components to monitor one or more aspects of the vehicle, including one or more vehicle sensors (such as cameras, microphones, pressure sensors, wheel position sensors, location / position sensors like GPS antennas, etc.). Some sensors can be configured to monitor the vehicle's current movement, acceleration, steering torque, etc.

[0033] As in Fig. As shown in Figure 1, the vehicle 10 includes a front-facing vehicle camera 26 configured to capture images of a field of view in front of the vehicle 10. The field of view can be a large or wide field of view (such as at least a field of view of thirty degrees, a field of view of forty-five degrees, a field of view of sixty degrees, a field of view of ninety degrees, etc.) in order to detect or record objects at the sides of a road on which the vehicle 10 is traveling.

[0034] The vehicle 10 can include an optional rear vehicle camera 24 and an optional side vehicle camera 28. In different implementations, the vehicle 10 may contain more or fewer of these optional vehicle cameras. The vehicle 10 can include any suitable laser, lidar sensor, etc., used to detect objects in the vicinity of the vehicle 10.

[0035] In some exemplary embodiments, an object detector of the vehicle can be configured to detect a nearest vehicle in the path (CIPV) (e.g., another vehicle ahead of the current path of vehicle 10), an unprotected road user (VRU) (e.g., a pedestrian or cyclist), etc. The vehicle control module 20 can be configured to control the movement of vehicle 10 based on a detected CIP target vehicle, such as by automatically increasing or decreasing the automatic acceleration of vehicle 10, automatically applying the brakes of vehicle 10 (e.g., in response to an imminent braking event during a collision), etc.

[0036] The vehicle control module 20 can communicate with another device or apparatus via a wireless communication interface, which may include one or more wireless antennas for transmitting and / or receiving wireless communication signals. For example, the wireless communication interface can use any suitable wireless communication protocol, including, but not limited to, vehicle-to-everything (V2X) communication, Wi-Fi communication, wireless area network (WAN) communication, cellular communication, personal area network (PAN) communication, short-range wireless communication (e.g., Bluetooth), etc.The interface for wireless communication can communicate with a remote computer device via one or more wireless and / or wired networks. With regard to vehicle-to-vehicle (V2X) communication, the vehicle can include one or more V2X transceivers (e.g., V2X signal transmitting and / or receiving antennas).

[0037] Vehicle 10 also includes a user interface. The user interface can include any suitable displays (such as on a dashboard, console, or elsewhere), a touchscreen or other input devices, speakers for generating audio signals, etc.

[0038] Fig. Figure 2 is a block diagram of an example system 100 for predicting a pedestrian's crossing intention based on image processing of a vehicle camera image. The components of system 100 can, for example, be controlled by the vehicle control module 20. Fig. 1 will be executed.

[0039] The vehicle control module 20 is configured to receive a vehicle camera image 102 from a vehicle camera, such as the front vehicle camera 26. At 104, the vehicle control module 20 is configured to extract features for image processing using a machine learning model. As explained below, the vehicle camera image 102 can, for example, be fed to multiple visual transformer layers to generate multiple output features 106. In some examples, the multiple output features 106 can be arranged in a three-dimensional array with M by N rows and columns and C channels.

[0040] In a process separate from feature extraction, the vehicle camera image 102 is fed to a traffic object detection model 108 to detect traffic and pedestrian objects in the vehicle camera image 102. For example, one or more object detection algorithms, machine learning models, etc., can process the vehicle camera image 102 to identify pedestrians and traffic objects in the image. Examples of traffic objects include, but are not limited to, traffic lights, pedestrian crossings, sidewalks, vehicles, traffic signs, etc.

[0041] Fig. Figure 2 illustrates an object detection output 110 divided into several areas, with identified objects in hatched boxes. For example, a detected pedestrian 112 can be identified in a vertical bounding box corresponding to three areas of the object detection output 110, and a detected pedestrian crossing object 114 can be identified by a horizontal bounding box corresponding to six other areas of the object detection output 110.

[0042] The vehicle control module 20 is configured to assign attention weights to areas of the vehicle camera image 102 containing detected objects. For example, the vehicle camera image 102 can be divided into a grid of areas (e.g., squares, spots, or patches, etc.) in the object detection output 110, and areas in the grid that correspond to detected objects (such as the pedestrian crossing object 114) or pedestrians (such as the pedestrian 112) can be assigned higher attention weights than areas that do not contain corresponding detected objects.

[0043] The vehicle control module 20 is configured to combine the attention weights with the extracted features (e.g., the multiple output features 106) for each image area at 116. For example, a weighted sum can be generated by applying the assigned attention weight for each area to the extracted output features of the image area. In this example, a higher intensity due to the higher attention weights can steer or influence the machine learning model in a direction that focuses it on areas of the image where detected pedestrians or traffic objects are located.

[0044] The vehicle control module 20 is configured to execute a machine learning model to generate a crossing intention prediction output. For example, the multiple output features 106, as modified by the attention weights from the object detection output 110, can be fed to a multilayer perceptron (MLP) 118 or another suitable machine learning model to generate a pedestrian crossing intention prediction output 120.

[0045] The output 120 of a pedestrian crossing intention prediction can indicate the probability that an identified pedestrian (such as pedestrian 112) plans to cross a road on which the host vehicle is currently traveling. For example, the vehicle control module 20 can determine whether the output 120 of a pedestrian crossing intention prediction is greater than a specified threshold (such as a 30% probability of crossing intention, a 50% probability of crossing intention, an 80% probability of crossing intention, etc.).

[0046] The crossing intention can be used by various systems in the vehicle 10, such as displaying the identified pedestrian on a vehicle display along with an indicator of the predicted crossing intention (so that the driver recognizes a pedestrian who may be planning to cross the road in front of the vehicle), generating an audible or tactile warning for the driver, controlling automatic braking to apply the vehicle brakes to slow the vehicle as it approaches the identified pedestrian, etc.

[0047] For example, if automatic braking is activated in the vehicle, the vehicle control module 20 can be configured to actuate the brakes of the vehicle 10 according to the position of the detected pedestrian (e.g. based on the time until collision with the detected pedestrian 112 etc.).

[0048] Fig. Figure 3 is a block diagram showing an exemplary architecture of a machine learning model of the system. Fig. Figure 2 illustrates that a vehicle camera image 202 is fed to several visual transformer layers 204 to generate extracted output features 206 (which can be arranged in an array of M by N rows and columns, with each element having C channels).

[0049] The vehicle camera image 202 is also fed to a traffic object detection model 108 to generate an object detection output 210. The object detection output 210 contains a detected pedestrian 212 and a detected pedestrian crossing object 214, but may also contain other types of detected objects.

[0050] In 222, the vehicle control module 20 is configured to assign attention weights for crossing intention to different areas of the vehicle camera image 202. For example, areas of the vehicle camera image corresponding to detected traffic objects, such as the pedestrian crossing object 214, may have a higher probability of being relevant for an accurate prediction of whether the pedestrian 212 intends to cross the road.

[0051] As in Fig. As shown in Figure 3, multiple keys 224 can be obtained from the extracted output features 206, and a classification query (CLS) 226 can be executed on the extracted output features 206. The multiple keys 224 (which can be represented by (NM)xC parameters) can be correlated with the CLS query 226 (which can be represented by a channel Cx1) to generate a correlated output (NM).

[0052] The correlated output can be combined with the attention weights and fed to a normalized exponential function 230 (e.g., a normalized softmax function) to produce an output of dimension (NM). This output can be combined with values ​​228 of the extracted output features 206 to produce an embedding vector of dimension (C). The embedding vector is fed to a classification MLP head 218 to produce the output 220 of a crossing intention prediction.

[0053] Fig. Figure 4 is a flowchart illustrating an exemplary procedure for predicting a pedestrian's crossing intention based on image processing from a vehicle camera image. The diagram in Fig. The illustrated process 4 can be implemented, for example, by the vehicle control module 20. Fig. 1. At 404, the process begins by taking an image from a vehicle camera, such as the front vehicle camera 26. Fig. 1. will be received.

[0054] The vehicle control module is configured to extract 408 features for image processing using a machine learning model. For example, the image received from the vehicle camera can be fed to multiple visual transformer layers to generate multiple output features. In some examples, the multiple output features can be arranged in a three-dimensional array with M by N rows and columns and C channels.

[0055] The vehicle control module is configured to detect 412 traffic and pedestrian objects in the received image. For example, one or more object detection algorithms, machine learning models, etc., can process the received image to identify pedestrians and traffic objects. Examples of traffic objects include, but are not limited to, traffic lights, pedestrian crossings, sidewalks, vehicles, traffic signs, etc.

[0056] The vehicle control module is configured to assign an attention weight to 416 areas of the image containing detected objects. For example, the image can be divided into a grid of areas (e.g., squares, patches, etc.), and areas within the grid corresponding to detected objects or pedestrians can be assigned higher attention weights than areas without such detected objects.

[0057] The vehicle control module is configured to combine the attention weights with the extracted features for each image area at a value of 420. For example, a weighted sum can be generated by applying the assigned attention weight for each area to the extracted output features of the image area. In this example, a higher intensity due to the higher attention weights can steer or influence the machine learning model in a direction that focuses it on areas of the image where detected pedestrians or traffic objects are located.

[0058] The vehicle control module is configured to execute a machine learning model on command 424 to generate a pedestrian crossing intention prediction. For example, the output features, as modified by the attention weights, can be fed to a multilayer perceptron (MLP) or other suitable machine learning model to generate a pedestrian crossing intention prediction.

[0059] The output of a pedestrian crossing intention prediction can indicate the probability that an identified pedestrian plans to cross a road on which the host vehicle is currently traveling. At 428, the controller determines whether the output of a pedestrian crossing intention prediction is greater than a specified threshold (e.g., a 30% probability of crossing intention, a 50% probability of crossing intention, an 80% probability of crossing intention, etc.).

[0060] If the output of a pedestrian crossing intention prediction is not greater than the specified crossing intention threshold, the controller returns to 404 to obtain another image from the vehicle camera. If, at 428, the output of a pedestrian crossing intention prediction is greater than the specified crossing intention threshold, the controller proceeds to 432 to assign a crossing intention to the identified pedestrian.

[0061] The intention to cross can be used by various systems in the vehicle, such as displaying the identified pedestrian on a vehicle display along with an indicator of the predicted intention to cross (so that the driver recognizes a pedestrian who may be planning to cross the road in front of the vehicle), generating an audible or tactile warning for the driver, controlling automatic braking to apply vehicle brakes to slow the vehicle as it approaches the identified pedestrian, etc.

[0062] For example, the vehicle control module is configured to determine at 436 whether automatic braking is activated in the vehicle. If so, the control unit proceeds to 440 to apply the vehicle's brakes according to the position of the detected pedestrian (e.g., based on the time until collision with the identified pedestrian, etc.). The control unit then returns to 404 to obtain another image from the vehicle camera.

[0063] Various machine learning models can be used in different implementations to generate output predicting a pedestrian's crossing intention. Examples of different types of machine learning models that can be used for the automated processing of vehicle camera images are described below. Fig. 5 - 7 described and illustrated; however, the exemplary embodiments are not limited thereto (and may include transformer layers, multilayer perceptrons, etc.).

[0064] Fig. 5A and Fig. Figure 5B shows an example of a recurrent neural network used to generate models such as those described above, using machine learning techniques. Machine learning is a process used to design complex models and algorithms suitable for prediction (e.g., patient-provider matching predictions). Models generated using machine learning, such as those described above, can provide reliable, repeatable decisions and outcomes, and uncover hidden insights by learning from historical relationships and trends in the data.

[0065] The purpose of using a recurrent neural network-based model and training it using machine learning, as described above, can be to directly predict dependent variables without having to mathematically define the relationships between them. A neural network model comprises a large number of virtual neurons that operate in parallel and are arranged in layers. The first layer is the input layer and receives raw, unprocessed input data. Each subsequent layer modifies the outputs of a previous layer and sends them to the next layer. The final layer is the output layer and generates the system's output.

[0066] Fig. Figure 5A shows a fully connected neural network where every neuron in a given layer is connected to every neuron in the next layer. In the input layer, each input node is associated with a numerical value, which can be any real number. In each layer, every connection emanating from an input node has an associated weight, which can also be any real number (see Figure 5A). Fig. 5B). In the input layer, the number of neurons is equal to the number of features (columns) in a dataset. The output layer can have multiple continuous outputs.

[0067] The layers between the input and output layers are hidden layers. The number of hidden layers can be one or more (one hidden layer should be sufficient for most applications). A neural network without hidden layers can represent linearly separable functions or decisions. A neural network with one hidden layer can perform a continuous mapping from one finite space to another. A neural network with two hidden layers can approximate any smooth mapping with arbitrary accuracy.

[0068] The number of neurons can be optimized. At the beginning of training, a network configuration is more likely to have too many nodes. Some of the nodes that would not noticeably affect the network's performance can be removed during training. For example, nodes with weights approaching zero after training can be removed (this process is called pruning). The number of neurons can lead to underfitting (the inability to adequately capture signals in the dataset) or overfitting (insufficient information to train all neurons; the network operates on a training dataset but not a test dataset).

[0069] Various methods and criteria can be used to measure the performance of a neural network model. For example, the root mean squared error (RMSE) measures the average difference between observed values ​​and model predictions. The coefficient of determination (R²) measures the correlation (not accuracy) between observed and predicted outcomes. This method may not be reliable if the data exhibits high variance. Other performance measures include irreducible noise, model bias, and model variance. High model bias indicates that the model is unable to capture the true relationship between the predictors and the outcome. Model variance can indicate whether a model is stable (a small disturbance in the data significantly alters the model fit).The neural network can receive inputs, such as vectors, which can be used to generate models that can be used with vendor matching, risk model processing, or both, as described here.

[0070] Fig. Figure 6 illustrates an example of a long short-term memory (LSTM) neural network 602, which is used to generate models such as those described above using machine learning techniques, although other exemplary embodiments may also include other types of machine learning models, including transformer layers, multilayer perceptrons, etc. The generic or general exemplary LSTM neural network 602 can be used to implement a machine learning model, and various implementations can utilize other types of machine learning networks (such as transformer layers, multilayer perceptrons, etc.). The LSTM neural network 602 comprises an input layer 604, a hidden layer 608, and an output layer 612. The input layer 604 includes inputs 604a, 604b... 604n. The hidden layer 608 contains neurons 608a, 608b... 608n.The output layer 612 contains the outputs 612a, 612b... 612n.

[0071] Each neuron in the hidden layer 608 receives an input from the input layer 604 and outputs a value to the corresponding output in the output layer 612. For example, neuron 608a receives an input from input 604a and outputs a value to output 612a. Every neuron besides neuron 608a also receives an output from a previous neuron as an input. For example, neuron 608b receives inputs from input 604b and output 612a. In this way, the output of each neuron is passed on to the next neuron in the hidden layer 608. The final output 612n in the output layer 612 outputs a probability associated with inputs 604a-604n. Although the input layer 604, the hidden layer 608, and the output layer 612 are represented as each comprising three elements, each layer can contain any number of elements.

[0072] In various embodiments, each layer of the LSTM neural network 602 must contain the same number of elements as each of the other layers of the LSTM neural network 602. In some exemplary embodiments, a convolutional neural network can be implemented. Similar to LSTM neural networks, convolutional neural networks contain an input layer, a hidden layer, and an output layer. However, in a convolutional neural network, the output layer contains one fewer output than the number of neurons in the hidden layer, and each neuron is connected to every output. Furthermore, each input in the input layer is connected to every neuron in the hidden layer. In other words, input 604a is connected to each of neurons 608a, 608b, ..., 608n.

[0073] In various implementations, each input node in the input layer can be assigned a numerical value, which can be any real number. In each layer, every connection emanating from an input node has an associated weight, which can also be any real number. In the input layer, the number of neurons is equal to the number of features (columns) in a dataset. The output layer can have multiple continuous outputs.

[0074] As mentioned above, the layers between the input and output layers are hidden layers. The number of hidden layers can be one or more (one hidden layer may be sufficient for many applications). A neural network without hidden layers can represent linearly separable functions or decisions. A neural network with one hidden layer can perform a continuous mapping from one finite space to another. A neural network with two hidden layers can approximate any smooth mapping with arbitrary accuracy. The neural network of Fig. 6 can receive inputs, e.g. vectors, which can be used to generate models that can be used with vehicle camera images to generate outputs of a prediction of a pedestrian's crossing intention, as described here.

[0075] Fig. Figure 7 illustrates an example process for creating a machine learning model. In Figure 707, the controller receives data from a database (e.g., a data warehouse). The data can contain any suitable data for developing machine learning models.

[0076] At 711, the controller separates the data received from database 702 into training data 715 and test data 719. The training data 715 is used to train the model at 723, and the test data 719 is used to test the model at 727. Typically, depending on the desired model development parameters, the set of training data 715 is selected to be larger than the set of test data 719. For example, the training data 715 might comprise approximately seventy percent of the data obtained from database 702, approximately eighty percent, approximately ninety percent, and so on. The remaining thirty percent, twenty percent, or ten percent are then used as test data 719.

[0077] Separating a portion of the acquired data as test data (719) allows the trained model to be tested against the actual output data, enabling more accurate training and development of the model (723 and 727). The model can be trained (723) using any suitable machine learning techniques, including those described here, such as Random Forest, generalized linear models, decision trees, and neural networks.

[0078] At 731, the controller evaluates the model test results. For example, the trained model at 727 can be tested using test data 719, and the output data from the tested model can be compared with the actual outputs of test data 719 to determine a degree of accuracy. The model results can be evaluated using any suitable analysis for machine learning models, such as the exemplary techniques described below.

[0079] After evaluating the model test results at 731, the model can be deployed at 735 if the results are satisfactory. Deploying the model may involve using it to generate predictions for a large input dataset with unknown outputs. If the evaluation of the model test results at 731 is unsatisfactory, the model can be further developed using different parameters, different modeling techniques, different model types, etc. The procedure for machine learning models of Fig. 7 can receive inputs, e.g., vectors, which can be used to generate models that can be used with vendor matching, risk model processing, or both, as described herein.

[0080] The preceding description is merely illustrative and intended to limit the scope of the revelation, its application, or uses. The comprehensive doctrine of revelation can be implemented in a multitude of forms. Therefore, although this revelation contains particular examples, the true scope of the revelation should not be so limited, since other modifications will become apparent upon study of the drawings, the description, and the following claims. It should be understood that one or more steps within a process may be carried out in a different order (or simultaneously) without altering the principles of the present revelation.Furthermore, although each of the embodiments described above is characterized by certain features, one or more of these features described in relation to any embodiment of the disclosure may be implemented in one of the other embodiments and / or combined with features of one of the other embodiments, even if this combination is not expressly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with each other remain within the scope of this disclosure.

[0081] Spatial and functional relationships between elements (for example, between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "interlocked," "coupled," "adjacent," "near or beside," "on," "above," "below," and "arranged." Unless explicitly described as "direct," when a relationship between first and second elements is described in the above disclosure, this relationship may be a direct relationship, in which no other intervening elements exist between the first and second elements, or it may be an indirect relationship, in which one or more intervening elements (either spatial or functional) exist between the first and second elements.As used here, the phrase “at least one of A, B and C” should be understood as meaning a logical (A OR B OR C) using a non-exclusive logical OR, and should not be understood as meaning “at least one of A, at least one of B and at least one of C”.

[0082] In the diagrams, the direction of an arrow, as indicated by its tip, generally illustrates the flow of information (e.g., data or instructions) that is relevant to the illustration. For example, if Element A and Element B exchange a variety of information, but information transferred from Element A to Element B is important for the illustration, the arrow may point from Element A to Element B. This unidirectional arrow does not imply that no other information is transferred from Element B to Element A. Furthermore, Element B may send requests for, or acknowledgments of, information to Element A in connection with the information transferred from Element A to Element B.

[0083] In this application, including the definitions below, the term "module" or the term "controller" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above components, such as in a system-on-a-chip.

[0084] The module may contain one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the internet, a wide area network (WAN), or combinations thereof. The functionality of any given module of this disclosure may be distributed among multiple modules connected via interface circuits. For example, multiple modules may enable load balancing. In another example, a server module (also known as a remote or cloud module) may perform some functions for a client module.

[0085] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" refers to a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" refers to a processor circuit that, in combination with additional processor circuits, executes some or all of the code from one or more modules. References to multiple processor circuits include multiple processor circuits on separate chips, multiple processor circuits on a single chip, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above.The term shared memory circuit refers to a single memory circuit that stores some or all of the code from multiple modules. The term group memory circuit refers to a memory circuit that, in combination with additional memory, stores some or all of the code from one or more modules.

[0086] The term storage circuit is a subset of the term computer-readable medium. The term computer-readable medium, as used here, does not include transitory electrical or electromagnetic signals that propagate through a medium (such as on a carrier wave); the term computer-readable medium can therefore be considered material and non-transient.Non-restrictive examples of a non-transient, physical, computer-readable medium include non-volatile memory circuits (such as a flash memory circuit, a erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

[0087] The devices and methods described in this application can be partially or fully implemented by means of a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above serve as software specifications that can be translated into computer programs through the routine work of a person skilled in the art or a programmer.

[0088] Computer programs contain instructions executable by processors, stored on at least one non-transitory, physical, computer-readable medium. Computer programs may also contain or rely on stored data. Computer programs may include a basic input / output system (BIOS) that interacts with the computer's special-purpose hardware, device drivers that interact with specific devices of the computer for special purposes, one or more operating systems, user applications, background services, background applications, etc.

[0089] Computer programs can contain: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code for execution by an interpreter; (v) source code for compilation and execution by a just-in-time compiler; etc. For example, source code can be written using the syntax of languages ​​such as C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5th Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, and Visual Studio. Include Basic®, Lua, MATLAB, SIMULINK and Python®. legend

[0090] In the drawing figures, N stands for no and Y for yes.

Claims

[1] A method for controlling automatic acceleration and braking of a vehicle, the method comprising: Obtaining an image using at least one vehicle camera of a host vehicle; Extracting machine learning model feature inputs based on the obtained image; Detecting one or more objects in the obtained image, wherein the one or more objects comprise at least one pedestrian; Assigning attention weights to regions of the obtained image according to positions of the one or more objects in the obtained image; Combining the attention weights with corresponding ones of the machine learning model feature inputs according to the regions of the obtained image; Executing a machine learning model to generate an output of a prediction of a crossing intention associated with the at least one pedestrian; and in response to outputting a prediction of crossing intent that exceeds a crossing intent threshold, controlling automatic braking of the host vehicle according to a position of the at least one pedestrian. [2] The method of claim 1, wherein assigning the attention weights comprises: Assigning a first intensity value to a first region of the obtained image corresponding to the at least one pedestrian; and Assigning a second intensity value to a second region of the obtained image that does not correspond to the at least one pedestrian, wherein the first intensity value is greater than the second intensity value. [3] The method of claim 2, wherein combining the attention weights and the machine learning model feature inputs comprises generating a weighted sum of the machine learning model feature inputs according to the attention weights. [4] The method of claim 1, wherein: the machine learning model contains a multi-layer perceptron; and executing the machine learning model comprises generating the output of a prediction of a crossing intention according to an output of the multi-layer perceptron. [5] The method of claim 1, wherein extracting machine learning model feature inputs based on the obtained image comprises feeding the obtained image to a plurality of visual transformer layers to generate the machine learning model feature inputs. [6] The method of claim 1, wherein executing the machine learning model comprises: Obtaining multiple key values ​​according to the machine learning model feature inputs; Executing a classification query according to the machine learning model feature inputs; and Correlating a classification query output with the multiple key values. [7] The method of claim 6, wherein executing the machine learning model comprises: Combining the attention weights with a correlation of the classification query output and the multiple key values; and Executing a normalized exponential function on a combination of the attention weights and the correlation of the classification query output and the plurality of key values ​​to produce the output of a prediction of crossing intention. [8] The method of claim 7, wherein executing the machine learning model comprises: Combining an output of the normalized exponential function with the plurality of key values ​​to generate an embedding vector; and Feeding the embedding vector to a multi-layer perceptron to produce the output of a prediction of crossing intention. [9] The method of claim 1, further comprising: Feeding training data and test data to the machine learning model; Comparing multiple crossing intention prediction outputs of the machine learning model based on the training data with labeled crossing intention outputs of the test data; Determining whether an accuracy of a comparison is greater than or equal to a specified accuracy threshold; Adjusting parameters of the machine learning model and retraining the machine learning model in response to determining that the accuracy of the comparison is less than the specified accuracy threshold; and Storing the machine learning model for use in generating an output of a prediction of crossing intent in response to determining that the accuracy of the comparison is greater than or equal to the specified accuracy threshold. [10] A vehicle control system for controlling vehicle braking based on image processing of a vehicle camera, the vehicle control system comprising: at least one vehicle camera configured to obtain an image of an area in front of a host vehicle; and a vehicle control module of the host vehicle, the vehicle control module being configured to: Machine learning model feature inputs extracted based on the obtained image; detecting one or more objects in the obtained image, wherein the one or more objects comprise at least one pedestrian; assigning attention weights to regions of the obtained image according to positions of the one or more objects in the obtained image; combines the attention weights with corresponding machine learning model feature inputs according to the regions of the obtained image; executes a machine learning model to generate an output of a prediction of a crossing intention associated with the at least one pedestrian; and in response to outputting a prediction of crossing intent that exceeds a crossing intent threshold, controlling automatic braking of the host vehicle according to a position of the at least one pedestrian.

Images