Vehicle control system based on vehicle camera and pedestrian image processing

Abstract

A vehicle control system based on vehicle camera and pedestrian image processing is provided. A method for controlling automatic vehicle acceleration and braking includes: obtaining an image using at least one vehicle camera of a host vehicle; extracting machine learning model feature input based on the obtained image; detecting one or more objects in the obtained image, wherein the one or more objects comprise at least one pedestrian; assigning an attention weight to a region of the obtained image according to a location of one or more objects in the obtained image; according to the obtained region of the image, combining the attention weight with a corresponding machine learning model feature input in the machine learning model feature inputs; executing the machine learning model to generate a traverse intent prediction output associated with the at least one pedestrian; and in response to the traverse intent prediction output exceeding the traverse intent threshold, controlling automatic braking of the host vehicle in accordance with the position of the at least one pedestrian.

Description

Technical Field

[0001] The information provided in this section is for the purpose of generally presenting the background of the present disclosure. To the extent described in this section, the work of the presently named inventors, and aspects that may otherwise not qualify as prior art at the time of filing, are neither explicitly nor implicitly admitted to be prior art to the present disclosure.

[0002] The present disclosure generally relates to a vehicle control system for controlling automatic vehicle braking, including a machine learning model for predicting pedestrian crossing intent based on image processing of vehicle camera images. Background Art

[0003] Some vehicles include a vehicle camera that is configured to obtain images from in front of the vehicle along the vehicle's travel path. The autonomous driving system may be configured to control acceleration, braking, etc. based on objects in front of the vehicle. Pedestrians often stand and wait to cross the road that the vehicle is traveling on, and there may be a risk of collision if a pedestrian crosses the road while the vehicle is driving. Summary of the invention

[0004] A method for controlling automatic vehicle acceleration and braking includes: obtaining an image using at least one vehicle camera of a host vehicle; extracting a machine learning model feature input based on the obtained image; detecting one or more objects in the obtained image, the one or more objects including at least one pedestrian; assigning an attention weight to an area of ​​the obtained image based on the position of the one or more objects in the obtained image; combining the attention weight with a corresponding one of the machine learning model feature inputs based on the area of ​​the obtained image; executing the machine learning model to generate a crossing intention prediction output associated with at least one pedestrian; and in response to the crossing intention prediction output exceeding a crossing intention threshold, controlling automatic braking of the host vehicle based on the position of at least one pedestrian.

[0005] In other features, assigning the attention weight includes assigning a first intensity value to a first region of the obtained image corresponding to at least one pedestrian, and assigning a second intensity value to a second region of the obtained image that does not correspond to the at least one pedestrian, and the first intensity value is greater than the second intensity value.

[0006] In other features, combining the attention weights and the machine learning model feature inputs includes generating a weighted sum of the machine learning model feature inputs according to the attention weights.

[0007] In other features, the machine learning model includes a multilayer perceptron, and executing the machine learning model includes generating a crossing intent prediction output based on an output of the multilayer perceptron.

[0008] In other features, extracting a machine learning model feature input based on the obtained image includes providing the obtained image to a plurality of visual transformer layers to generate the machine learning model feature input.

[0009] In other features, executing the machine learning model includes: obtaining multiple key values ​​based on the machine learning model feature input, performing a classification query based on the machine learning model feature input, and associating the classification query output with the multiple key values.

[0010] In other features, executing the machine learning model includes: combining attention weights with correlations between the classification query output and multiple key values; and performing a normalized exponential function on the combination of the attention weights with correlations between the classification query output and multiple key values ​​to generate a crossing intention prediction output.

[0011] In other features, executing the machine learning model includes combining an output of the normalized exponential function with a plurality of key values ​​to generate an embedding vector, and providing the embedding vector to a multilayer perceptron to generate a traversing intent prediction output.

[0012] In other features, training data and test data are provided to a machine learning model; multiple crossing intention prediction outputs of the machine learning model are compared with labeled crossing intention outputs of the test data based on the training data; it is determined whether the accuracy of the comparison is greater than or equal to a specified accuracy threshold; in response to determining that the accuracy of the comparison is less than the specified accuracy threshold, the parameters of the machine learning model are adjusted and the machine learning model is retrained; and in response to determining that the accuracy of the comparison is greater than or equal to the specified accuracy threshold, the machine learning model is saved for generating crossing intention prediction outputs.

[0013] In other features, the image includes at least a 45 degree field of view from at least one vehicle camera. In other features, the one or more objects include at least one of a crosswalk, a traffic light, or another vehicle.

[0014] A vehicle control system for controlling vehicle braking based on vehicle camera image processing includes at least one vehicle camera and a vehicle control module of a host vehicle, wherein the at least one vehicle camera is configured to obtain an image from the front of the host vehicle. The vehicle control module is configured to: extract a machine learning model feature input based on the obtained image; detect one or more objects in the obtained image, the one or more objects including at least one pedestrian; assign attention weights to areas of the obtained image based on the positions of the one or more objects in the obtained image; combine the attention weights with corresponding machine learning model feature inputs in the machine learning model feature inputs based on the areas of the obtained image; execute the machine learning model to generate a crossing intention prediction output associated with at least one pedestrian; and in response to the crossing intention prediction output exceeding a crossing intention threshold, control automatic braking of the host vehicle based on the position of at least one pedestrian.

[0015] In other features, the vehicle control module is configured to assign attention weights by assigning a first intensity value to a first area of ​​the obtained image corresponding to at least one pedestrian, and assigning a second intensity value to a second area of ​​the obtained image that does not correspond to the at least one pedestrian, and the first intensity value is greater than the second intensity value.

[0016] In other features, the vehicle control module is configured to assign attention weights and machine learning model feature inputs by generating a weighted sum of the machine learning model feature inputs according to the attention weights.

[0017] In other features, the machine learning model includes a multi-layer perceptron, and the vehicle control module is configured to execute the machine learning model by generating a crossing intent prediction output based on an output of the multi-layer perceptron.

[0018] In other features, the vehicle control module is configured to extract machine learning model feature inputs based on the obtained image by providing the obtained image to multiple visual transformer layers to generate machine learning model feature inputs.

[0019] In other features, the vehicle control module is configured to execute the machine learning model by obtaining a plurality of key values ​​based on the machine learning model feature input, performing a classification query based on the machine learning model feature input, and associating the classification query output with the plurality of key values.

[0020] In other features, the vehicle control module is configured to execute the machine learning model by: combining the attention weights with the correlations between the classification query output and the multiple key values; and performing a normalized exponential function on the combination of the attention weights with the correlations between the classification query output and the multiple key values ​​to generate a crossing intention prediction output.

[0021] In other features, the vehicle control module is configured to execute the machine learning model by combining an output of the normalized exponential function with a plurality of key values ​​to generate an embedding vector, and providing the embedding vector to a multilayer perceptron to generate a crossing intent prediction output.

[0022] In other features, the image includes at least a 45 degree field of view from at least one vehicle camera, and the one or more objects include at least one of a crosswalk, a traffic light, or another vehicle.

[0023] The following options are provided:

[0024] 1. A method for controlling automatic vehicle acceleration and braking, the method comprising:

[0025] acquiring an image using at least one vehicle camera of a host vehicle;

[0026] Extracting feature inputs of a machine learning model based on the obtained images;

[0027] detecting one or more objects in the obtained image, the one or more objects including at least one pedestrian;

[0028] assigning attention weights to regions of the obtained image based on locations of one or more objects in the obtained image;

[0029] combining the attention weight with a corresponding one of the machine learning model feature inputs according to the obtained region of the image;

[0030] executing the machine learning model to generate a crossing intention prediction output associated with at least one pedestrian; and

[0031] In response to the crossing intent prediction output exceeding a crossing intent threshold, automatic braking of the host vehicle is controlled based on the position of the at least one pedestrian.

[0032] 2. The method according to claim 1, wherein assigning attention weights comprises:

[0033] assigning a first intensity value to a first region of the obtained image corresponding to at least one pedestrian; and

[0034] A second intensity value is assigned to a second region of the obtained image that does not correspond to the at least one pedestrian, and the first intensity value is greater than the second intensity value.

[0035] 3. A method according to Option 2, wherein combining attention weights and machine learning model feature inputs includes generating a weighted sum of the machine learning model feature inputs based on the attention weights.

[0036] 4. The method according to claim 1, wherein:

[0037] The machine learning model includes a multi-layer perceptron; and

[0038] Executing the machine learning model includes generating a crossing intention prediction output based on the output of the multi-layer perceptron.

[0039] 5. A method according to Option 1, wherein extracting machine learning model feature input based on the obtained image includes providing the obtained image to multiple visual transformer layers to generate machine learning model feature input.

[0040] 6. The method of claim 1, wherein executing the machine learning model comprises:

[0041] Obtain multiple key values ​​based on machine learning model feature input;

[0042] Perform classification queries based on machine learning model feature inputs; and

[0043] Correlate categorical query output with multiple key values.

[0044] 7. The method of claim 6, wherein executing the machine learning model comprises:

[0045] combining the attention weights with the relevance of the classification query output and the plurality of key values; and

[0046] A normalized exponential function is performed on the combination of the attention weights and the relevance of the classification query output and multiple key values ​​to generate a crossing intent prediction output.

[0047] 8. The method of claim 7, wherein executing the machine learning model comprises:

[0048] combining the output of the normalized exponential function with a plurality of key values ​​to generate an embedding vector; and

[0049] The embedding vector is fed to a multi-layer perceptron to generate the crossing intention prediction output.

[0050] 9. The method according to claim 1, further comprising:

[0051] Provide training and testing data to machine learning models;

[0052] comparing a plurality of crossing intent prediction outputs of the machine learning model based on the training data with labeled crossing intent outputs of the test data;

[0053] Determines whether the accuracy of the comparison is greater than or equal to the specified accuracy threshold;

[0054] In response to determining that the accuracy of the comparison is less than a specified accuracy threshold, adjusting parameters of the machine learning model and retraining the machine learning model; and

[0055] In response to determining that the accuracy of the comparison is greater than or equal to the specified accuracy threshold, the machine learning model is saved for use in generating a crossing intent prediction output.

[0056] 10. The method of claim 1, wherein the image includes at least a 45 degree field of view from at least one vehicle camera.

[0057] 11. The method of claim 10, wherein the one or more objects include at least one of a crosswalk, a traffic light, or another vehicle.

[0058] 12. A vehicle control system for controlling vehicle braking based on vehicle camera image processing, the vehicle control system comprising:

[0059] at least one vehicle camera configured to obtain images from the front of the host vehicle; and

[0060] A vehicle control module of a host vehicle, the vehicle control module being configured to:

[0061] Extracting feature inputs of a machine learning model based on the obtained images;

[0062] detecting one or more objects in the obtained image, the one or more objects including at least one pedestrian;

[0063] assigning attention weights to regions of the obtained image based on locations of one or more objects in the obtained image;

[0064] combining the attention weight with a corresponding one of the machine learning model feature inputs according to the obtained region of the image;

[0065] executing the machine learning model to generate a crossing intention prediction output associated with at least one pedestrian; and

[0066] In response to the crossing intent prediction output exceeding a crossing intent threshold, automatic braking of the host vehicle is controlled based on the position of the at least one pedestrian.

[0067] 13. The vehicle control system according to claim 12, wherein the vehicle control module is configured to assign attention weights by:

[0068] assigning a first intensity value to a first region of the obtained image corresponding to at least one pedestrian; and

[0069] A second intensity value is assigned to a second region of the obtained image that does not correspond to the at least one pedestrian, and the first intensity value is greater than the second intensity value.

[0070] 14. A vehicle control system according to Option 13, wherein the vehicle control module is configured to assign attention weights and machine learning model feature inputs by generating a weighted sum of the machine learning model feature inputs based on the attention weights.

[0071] 15. The vehicle control system according to claim 12, wherein:

[0072] The machine learning model includes a multi-layer perceptron; and

[0073] The vehicle control module is configured to execute the machine learning model by generating a crossing intention prediction output based on the output of the multi-layer perceptron.

[0074] 16. A vehicle control system according to Option 12, wherein the vehicle control module is configured to extract machine learning model feature input based on the obtained image by providing the obtained image to multiple visual transformer layers to generate machine learning model feature input.

[0075] 17. The vehicle control system of claim 12, wherein the vehicle control module is configured to execute the machine learning model by:

[0076] Obtain multiple key values ​​based on machine learning model feature input;

[0077] Perform classification queries based on machine learning model feature inputs; and

[0078] Correlate categorical query output with multiple key values.

[0079] 18. The vehicle control system of claim 17, wherein the vehicle control module is configured to execute the machine learning model by:

[0080] combining the attention weights with the relevance of the classification query output and the plurality of key values; and

[0081] A normalized exponential function is performed on the combination of the attention weights and the relevance of the classification query output and multiple key values ​​to generate a crossing intent prediction output.

[0082] 19. The vehicle control system of claim 18, wherein the vehicle control module is configured to execute the machine learning model by:

[0083] combining the output of the normalized exponential function with a plurality of key values ​​to generate an embedding vector; and

[0084] The embedding vector is fed to a multi-layer perceptron to generate the crossing intention prediction output.

[0085] 20. The vehicle control system according to claim 19, wherein:

[0086] The image includes at least a 45 degree field of view from at least one vehicle camera; and

[0087] The one or more objects include at least one of a crosswalk, a traffic light, or another vehicle.

[0088] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The present disclosure will become more fully understood from the detailed description and accompanying drawings, in which:

[0090] Figure 1 is a diagram of an example vehicle including a front vehicle camera configured to capture images of pedestrians and traffic objects in front of the vehicle.

[0091] Figure 2 is a block diagram of an example system for predicting pedestrian crossing intent based on image processing of vehicle camera images.

[0092] Figure 3 It is shown Figure 2 Block diagram of an example machine learning model architecture for a system.

[0093] Figure 4 is a flow chart depicting an example process for predicting pedestrian crossing intent based on image processing of vehicle camera images.

[0094] Figure 5A and 5B is a graphical representation of an example recurrent neural network for predicting pedestrian crossing intention based on image processing of vehicle camera images.

[0095] Figure 6 is a graphical representation of the layers of an example long short-term memory (LSTM) machine learning model.

[0096] Figure 7 is a flow chart illustrating an example process for training a machine learning model.

[0097] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0098] Pedestrians often stand and wait to cross a road while a vehicle is traveling on it, which may present a collision risk if the pedestrian crosses the road while the vehicle is driving. Some example embodiments herein include systems and methods for estimating a pedestrian's crossing intention using vehicle camera images with a wide field of view (FOV). The wide FOV images effectively capture the interaction between pedestrians and various traffic objects, such as roads, crosswalks, traffic lights, other vehicles, bus stops, etc.

[0099] However, the wide FOV image also contains additional pedestrians and irrelevant information.Example embodiments described herein can focus a machine learning model (eg, a neural network) on pedestrians and traffic objects of interest for estimating the pedestrian's crossing intention.

[0100] Reference now Figure 1 , the vehicle 10 includes front wheels 12 and rear wheels 13. Figure 1 , the drive unit 14 selectively outputs torque to the front wheels 12 and / or the rear wheels 13 via the transmissions 16 and 18, respectively. The vehicle 10 may include different types of drive units. For example, the vehicle may be an electric vehicle, such as a battery electric vehicle (BEV), a hybrid vehicle or a fuel cell vehicle, a vehicle including an internal combustion engine (ICE), or other types of vehicles.

[0101] Some examples of drive unit 14 may include any suitable electric motor, power inverter, and motor controller, the motor controller being configured to control the power switch within the power inverter to adjust the motor speed and torque during propulsion and / or regeneration. During propulsion or regeneration, the battery system provides power to or receives power from the electric motor of drive unit 14 via the power inverter.

[0102] Although in Figure 1 The vehicle 10 includes one drive unit 14, but the vehicle 10 can have other configurations. For example, two separate drive units can drive the front wheels 12 and the rear wheels 13, one or more separate drive units can drive separate wheels, etc. As can be appreciated, other vehicle configurations and / or drive units can be used.

[0103] The vehicle control module 20 may be configured to control the operation of one or more vehicle components, such as controlling the operation of the drive unit 14 (e.g., by commanding a torque setting of an electric motor of the drive unit 14). The vehicle control module 20 may receive inputs for controlling vehicle components, such as signals received from a steering wheel, an accelerator pedal, a vehicle camera, etc. The vehicle control module 20 may monitor the vehicle's telematics for safety purposes, such as vehicle speed, vehicle location, vehicle braking and acceleration, etc.

[0104] The vehicle control module 20 may receive signals from any suitable components for monitoring one or more aspects of the vehicle, including one or more vehicle sensors (e.g., cameras, microphones, pressure sensors, wheel position sensors, position sensors such as global positioning system (GPS) antennas, etc.). Some sensors may be configured to monitor the current motion of the vehicle, the acceleration of the vehicle, steering torque, etc.

[0105] like Figure 1 As shown, the vehicle 10 includes a front vehicle camera 26 configured to capture images of a field of view in front of the vehicle 10. The field of view may be a wide field of view (e.g., at least a 30 degree field of view, a 45 degree field of view, a 60 degree field of view, a 90 degree field of view, etc.) in order to capture objects on the side of the road on which the vehicle 10 is traveling.

[0106] The vehicle 10 may include an optional rear vehicle camera 24 and an optional side vehicle camera 28. In various embodiments, the vehicle 10 may include more or less of any of these optional vehicle cameras. The vehicle 10 may include any suitable laser, lidar sensor, etc., for detecting objects around the vehicle 10.

[0107] In some example embodiments, the vehicle object detector may be configured to detect a closest in-path vehicle (CIPV) (e.g., another vehicle ahead of the current driving path of the vehicle 10), a vulnerable road user (VRU) (e.g., a pedestrian or cyclist), etc. The vehicle control module 20 may be configured to control the movement of the vehicle 10 based on the detected CIP target vehicle, such as by increasing or decreasing automatic acceleration of the vehicle 10, automatically applying braking of the vehicle 10 (e.g., in response to a braking event in which a collision is imminent), etc.

[0108] The vehicle control module 20 can communicate with another device via a wireless communication interface, which may include one or more wireless antennas for transmitting and / or receiving wireless communication signals. For example, the wireless communication interface can communicate via any suitable wireless communication protocol, including but not limited to vehicle-to-everything (V2X) communication, Wi-Fi communication, wireless area network (WAN) communication, cellular communication, personal area network (PAN) communication, short-range wireless communication (e.g., Bluetooth), etc. The wireless communication interface can communicate with a remote computing device via one or more wireless and / or wired networks. With respect to vehicle-to-vehicle (V2X) communication, the vehicle 10 may include one or more V2X transceivers (e.g., V2X signal transmitting and / or receiving antennas).

[0109] The vehicle 10 also includes a user interface. The user interface may include any suitable display (eg, on a dashboard, console, or elsewhere), a touch screen or other input device, speakers for generating audio, and the like.

[0110] Figure 2 is a block diagram of an example system 100 for predicting pedestrian crossing intention based on image processing of vehicle camera images. The components of system 100 may be provided by, for example Figure 1 The vehicle control module 20 executes.

[0111] The vehicle control module 20 is configured to obtain a vehicle camera image 102 from a vehicle camera, such as the front vehicle camera 26. At 104, the vehicle control module 20 is configured to extract features for image processing through a machine learning model. For example, as further explained below, the vehicle camera image 102 can be provided to a plurality of visual transformer layers to generate a plurality of output features 106. In some examples, the plurality of output features 106 can be arranged in a three-dimensional array having M×N rows and columns and C channels.

[0112] In a process separate from feature extraction, the vehicle camera image 102 is provided to a traffic object detection model 108 to detect traffic and pedestrian objects in the vehicle camera image 102. For example, one or more object detection algorithms, machine learning models, etc. may process the vehicle camera image 102 to identify pedestrians and traffic objects in the image. Example traffic objects may include, but are not limited to, traffic lights, crosswalks, sidewalks, vehicles, traffic signs, etc.

[0113] Figure 2 The object detection output 110 is shown divided into a plurality of regions, with objects identified in shaded boxes. For example, a detected pedestrian 112 may be identified in vertical bounding boxes corresponding to three regions of the object detection output 110, and a detected crosswalk object 114 may be identified by horizontal bounding boxes corresponding to six other regions of the object detection output 110.

[0114] The vehicle control module 20 is configured to assign attention weights to regions having detected objects in the vehicle camera image 102. For example, the vehicle camera image 102 may be divided into a grid of regions (e.g., squares, patches, etc.) in the object detection output 110, and regions in the grid corresponding to detected objects (e.g., crosswalk object 114) or pedestrians (e.g., pedestrian 112) may be assigned higher attention weights than regions without corresponding detected objects.

[0115] At 116, the vehicle control module 20 is configured to combine the attention weight with the extracted features (e.g., the plurality of output features 106) of each image region. For example, a weighted sum may be generated by applying the attention weight assigned to each region to the extracted output features of the image region. In this example, the higher strength due to the higher attention weight may bias the machine learning model to focus on the image region where the detected pedestrian or traffic object is located.

[0116] The vehicle control module 20 is configured to execute a machine learning model to generate a crossing intention prediction output. For example, the plurality of output features 106 modified by the attention weights from the object detection output 110 may be provided to a multi-layer perceptron (MLP) 118 or other suitable machine learning model to generate a pedestrian crossing intention prediction output 120.

[0117] The pedestrian crossing intention prediction output 120 may indicate the likelihood that an identified pedestrian (e.g., pedestrian 112) is planning to cross a street on which the host vehicle is driving. For example, the vehicle control module 20 may determine whether the pedestrian crossing intention prediction output 120 is greater than a specified threshold (e.g., a 30% likelihood, a 50% likelihood, an 80% likelihood, etc. of intending to cross).

[0118] Various systems in vehicle 10 may use crossing intent to, for example, display an identified pedestrian on a display of the vehicle along with a predicted crossing intent indicator (so the driver is aware of a pedestrian who may be planning to cross the street in front of the vehicle), generate an audible or tactile alert to the driver, control automatic braking to apply the vehicle brakes to slow the vehicle when the vehicle approaches an identified pedestrian, and so on.

[0119] For example, if automatic braking is activated in the vehicle, the vehicle control module 20 may be configured to apply brakes of the vehicle 10 based on the location of the detected pedestrian (eg, based on the time of collision with the identified pedestrian 112 , etc.).

[0120] Figure 3 It is shown Figure 2 200 . A vehicle camera image 202 is provided to a plurality of visual transformer layers 204 to generate extracted output features 206 (which may be arranged into an array of M×N columns, with each element having C channels).

[0121] The vehicle camera image 202 is also provided to a traffic object detection model 208 to generate object detection output 210. The object detection output 210 includes detected pedestrians 212 and detected crosswalk objects 214, but may also include other types of detected objects.

[0122] At 222, the vehicle control module 20 is configured to assign crossing intention attention weights to various regions of the vehicle camera image 202. For example, regions of the vehicle camera image corresponding to detected traffic objects (e.g., crosswalk object 214) may have a higher likelihood associated with an accurate prediction of whether the pedestrian 212 is intending to cross the street.

[0123] like Figure 3As shown, a plurality of keys 224 may be obtained from the extracted output features 206, and a classification (CLS) query 226 may be run on the extracted output features 206. The plurality of keys 224 (which may be represented by (NM)xC parameters) may be correlated with the CLS query 226 (which may be represented by the channel Cx1) to generate a correlated output (NM).

[0124] The relevant output can be combined with the attention weights and provided to a normalized index function 230 (e.g., a normalized Softmax function) to generate an output of dimension (NM). The output can be combined with the value 228 of the extracted output feature 206 to generate an embedding vector of dimension (C). The embedding vector is provided to the classification MLP head 218 to generate a crossing intention prediction output 220.

[0125] Figure 4 is a flow chart depicting an example process for predicting pedestrian crossing intent based on image processing of vehicle camera images. Figure 4 The process shown in can be done by, for example Figure 1 The vehicle control module 20 is executed. At 404, the process begins by taking a picture from a vehicle camera, such as Figure 1 The front vehicle camera 26 obtains an image.

[0126] At 408, the vehicle control module is configured to extract features for image processing through a machine learning model. For example, an image obtained by a vehicle camera can be provided to a plurality of visual transformer layers to generate a plurality of output features. In some examples, the plurality of output features can be arranged in a three-dimensional array having M×N rows and columns and C channels.

[0127] At 412, the vehicle control module is configured to detect traffic and pedestrian objects in the obtained image. For example, one or more object detection algorithms, machine learning models, etc. can process the obtained image to identify pedestrians and traffic objects in the image. Example traffic objects can include, but are not limited to, traffic lights, crosswalks, sidewalks, vehicles, traffic signs, etc.

[0128] At 416, the vehicle control module is configured to assign attention weights to regions of the image having detected objects. For example, the image may be divided into a grid of regions (e.g., squares, patches, etc.), and regions of the grid corresponding to detected objects or pedestrians may be assigned higher attention weights than regions without corresponding detected objects.

[0129] At 420, the vehicle control module is configured to combine the attention weights with the extracted features of each image region. For example, a weighted sum may be generated by applying the attention weights assigned to each region to the extracted output features of the image region. In this example, the higher strength due to the higher attention weights may bias the machine learning model to focus on the image region where the detected pedestrian or traffic object is located.

[0130] At 424, the vehicle control module is configured to execute the machine learning model to generate a crossing intention prediction output. For example, the output features modified by the attention weights can be provided to a multi-layer perceptron (MLP) or other suitable machine learning model to generate a pedestrian crossing intention prediction output.

[0131] The pedestrian crossing intention prediction output may indicate the likelihood that the identified pedestrian is planning to cross the street on which the host vehicle is driving. At 428, control determines whether the pedestrian crossing intention prediction output is greater than a specified threshold (e.g., 30% likelihood, 50% likelihood, 80% likelihood, etc. of intending to cross).

[0132] If the pedestrian crossing intent prediction output is not greater than the specified crossing intent threshold, control returns to 404 to obtain another image from the vehicle camera. If the pedestrian crossing intent prediction output is greater than the specified crossing intent threshold at 428, control proceeds to 432 to assign crossing intent to the identified pedestrian.

[0133] Various systems in the vehicle may use crossing intent to, for example, display an identified pedestrian on the vehicle's display along with a predicted crossing intent indicator (so the driver is aware of a pedestrian who may be planning to cross the street in front of the vehicle), generate an audible or tactile alert to the driver, control automatic braking to apply the vehicle brakes to slow the vehicle as the vehicle approaches an identified pedestrian, and so on.

[0134] For example, at 436, the vehicle control module is configured to determine whether automatic braking in the vehicle is activated. If so, control proceeds to 440 to apply braking of the vehicle based on the position of the detected pedestrian (e.g., based on the time of collision with the identified pedestrian, etc.). Control then returns to 404 to obtain another image from the vehicle camera.

[0135] In various implementations, a machine learning model may be used to generate a pedestrian crossing intention prediction output. Examples of various types of machine learning models that may be used for autonomous vehicle camera image processing are described below and shown in FIGS. 5-7 , but example embodiments are not limited thereto (and may include transformer layers, multi-layer perceptrons, etc.).

[0136] Figure 5A and 5BAn example of a recurrent neural network for generating models such as those described above using machine learning techniques is shown. Machine learning is a method for designing complex models and algorithms that lend themselves to predictions (e.g., patient and provider matching predictions). Models generated using machine learning, such as those described above, can produce reliable, repeatable decisions and results and reveal hidden insights by learning from historical relationships and trends in data.

[0137] The purpose of using a recurrent neural network based model and using machine learning to train the model as described above can be to directly predict the dependent variable without projecting the relationship between the variables into a mathematical form. The neural network model includes a large number of virtual neurons that operate in parallel and are arranged in layers. The first layer is the input layer and receives the raw input data. Each subsequent layer modifies the output from the previous layer and sends them to the next layer. The last layer is the output layer and produces the output of the system.

[0138] Figure 5A A fully connected neural network is shown, where each neuron in a given layer is connected to each neuron in the next layer. In the input layer, each input node is associated with a numeric value, which can be any real number. In each layer, each connection leaving an input node has a weight associated with it, which can also be any real number (see Figure 5B ). In the input layer, the number of neurons is equal to the number of features (columns) in the dataset. The output layer can have multiple continuous outputs.

[0139] The layers between the input and output layers are hidden layers. The number of hidden layers can be one or more (for most applications, one hidden layer is probably enough). A neural network with no hidden layers can represent linearly separable functions or decisions. A neural network with one hidden layer can perform a continuous mapping from one finite space to another. A neural network with two hidden layers can approximate any smooth mapping with any accuracy.

[0140] The number of neurons can be optimized. At the beginning of training, the network configuration will most likely have redundant nodes. During training, some nodes can be removed from the network without significantly affecting the network's performance. For example, nodes whose weights approach zero after training can be removed (a process called pruning). The number of neurons can lead to underfitting (not adequately capturing the signal in the dataset) or overfitting (not enough information to train all the neurons; the network performs well on the training dataset, but poorly on the test dataset).

[0141] The performance of a neural network model can be measured using a variety of methods and criteria. For example, the root mean square error (RMSE) measures the average distance between observations and model predictions. The coefficient of determination (R2) measures the correlation (not accuracy) between observations and predicted outcomes. This approach may be unreliable if the data has large differences. Other performance measures include irreducible noise, model bias, and model variance. A high model bias of a model indicates that the model cannot capture the true relationship between predictors and outcomes. Model variance can indicate whether the model is stable (slight perturbations in the data will significantly change the model fit). A neural network can receive inputs, such as vectors, which can be used to generate a model that can be used with provider matching, risk model processing, or both, as described herein.

[0142] Figure 6 An example of a long short-term memory (LSTM) neural network 602 is shown for generating models such as those described above using machine learning techniques, although other example embodiments may include other types of machine learning models, including transformer layers, multi-layer perceptrons, etc. The general example LSTM neural network 602 can be used to implement a machine learning model, and various implementations may use other types of machine learning networks (e.g., transformer layers, multi-layer perceptrons, etc.). The LSTM neural network 602 includes an input layer 604, a hidden layer 608, and an output layer 612. The input layer 604 includes inputs 604a, 604b...604n. The hidden layer 608 includes neurons 608a, 608b...608n. The output layer 612 includes outputs 612a, 612b...612n.

[0143] Each neuron of hidden layer 608 receives input from input layer 604 and outputs a value to a corresponding output in output layer 612. For example, neuron 608a receives input from input 604a and outputs a value to output 612a. In addition to neuron 608a, each neuron also receives the output of the previous neuron as input. For example, neuron 608b receives input from input 604b and output 612a. In this way, the output of each neuron is fed forward to the next neuron in hidden layer 608. The last output 612n in output layer 612 outputs the probability associated with input 604a-604n. Although input layer 604, hidden layer 608 and output layer 612 are depicted as each including three elements, each layer can contain any number of elements.

[0144] In various implementations, each layer of the LSTM neural network 602 must include the same number of elements as each other layer of the LSTM neural network 602. In some example embodiments, a convolutional neural network may be implemented. Similar to the LSTM neural network, the convolutional neural network includes an input layer, a hidden layer, and an output layer. However, in the convolutional neural network, the output layer includes one less output than the number of neurons in the hidden layer, and each neuron is connected to each output. In addition, each input in the input layer is connected to each neuron in the hidden layer. In other words, the input 604a is connected to each neuron 608a, 608b...608n.

[0145] In various implementations, each input node in the input layer can be associated with a numerical value, which can be any real number. In each layer, each connection leaving an input node has a weight associated with it, which can also be any real number. In the input layer, the number of neurons is equal to the number of features (columns) in the dataset. The output layer can have multiple continuous outputs.

[0146] As mentioned above, the layers between the input layer and the output layer are hidden layers. The number of hidden layers can be one or more (for many applications, one hidden layer may be sufficient). A neural network with no hidden layers can represent linearly separable functions or decisions. A neural network with one hidden layer can perform a continuous mapping from one finite space to another finite space. A neural network with two hidden layers can approximate any smooth mapping with any accuracy. Figure 6 The neural network can receive input, such as a vector, which can be used to generate a model that can be used with vehicle camera images to generate pedestrian crossing intention prediction output, as described in this article.

[0147] Figure 7 An example process for generating a machine learning model is shown. At 707, control obtains data from a database 702 (e.g., a data warehouse). The data may include any suitable data for developing a machine learning model.

[0148] At 711, control separates the data obtained from the database 702 into training data 715 and test data 719. At 723, the training data 715 is used to train the model, and at 727, the test data 719 is used to test the model. Typically, the set of training data 715 is selected to be larger than the set of test data 719, depending on the desired model development parameters. For example, the training data 715 may include about seventy percent of the data, about eighty percent of the data, about ninety percent of the data, etc. obtained from the database 702. The remaining thirty percent, twenty percent, or ten percent is then used as the test data 719.

[0149] Separating a portion of the acquired data as test data 719 allows the trained model to be tested against actual output data to facilitate more accurate training and development of the model at 723 and 727. At 723, the model can be trained using any suitable machine learning model technique, including those described herein, such as random forests, generalized linear models, decision trees, and neural networks.

[0150] At 731, control evaluates the model test results. For example, the trained model can be tested at 727 using the test data 719, and the results from the output data of the tested model can be compared to the actual output of the test data 719 to determine the level of accuracy. The model results can be evaluated using any suitable machine learning model analysis, such as example techniques described further below.

[0151] After evaluating the model test results at 731, if the model test results are satisfactory, the model can be deployed at 835. Deploying the model can include using the model to make predictions on a large-scale input data set with unknown outputs. If the evaluation of the model test results at 731 is not satisfactory, the model can be further developed using different parameters, using different modeling techniques, using other model types, etc. Figure 7 The machine learning model method can receive an input, such as a vector, which can be used to generate a model that can be used with provider matching, risk model processing, or both, as described herein.

[0152] The foregoing description is merely illustrative in nature and is by no means intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in various forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, because after studying the drawings, the specification and the appended claims, other modifications will become apparent. It should be understood that one or more steps in the method can be performed in different orders (or concurrently) without changing the principles of the present disclosure. In addition, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure can be implemented in the features of any other embodiment and / or combined with the features of any other embodiment, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the replacement of one or more embodiments with each other is still within the scope of the present disclosure.

[0153] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "next to," "on top of," "above," "below," and "disposed." Unless explicitly described as "direct," when describing the relationship between a first and a second element in the above disclosure, the relationship can be a direct relationship between the first and second elements with no other intervening elements, but can also be an indirect relationship between the first and second elements with one or more intervening elements (spatially or functionally) present. As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A or B or C), using a non-exclusive logical or, and should not be construed to mean "at least one of A, at least one of B, and at least one of C."

[0154] In a diagram, the direction of the arrow indicated by the arrow generally indicates the flow of information (such as data or instructions) of interest to the diagram. For example, when component A and component B exchange various information, but the information transmitted from component A to component B is relevant to the diagram, the arrow may point from component A to component B. This one-way arrow does not mean that no other information is transmitted from component B to component A. In addition, for information sent from component A to component B, component B may send a request or a receipt confirmation of the information to component A.

[0155] In this application, including the definitions below, the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to part of or include: an application specific integrated circuit (ASIC); a digital, analog or mixed analog / digital discrete circuit; a digital, analog or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated or grouped) that executes code; a memory circuit (shared, dedicated or grouped) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0156] The module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functions of any given module of the present disclosure may be distributed between multiple modules connected by the interface circuit. For example, multiple modules may allow load balancing. In another example, a server (also referred to as a remote or cloud) module may perform some functions on behalf of a client module.

[0157] The term code used above may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules. The term group processor circuit encompasses such a processor circuit, which is combined with additional processor circuits, to execute some or all code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules. The term group memory circuit encompasses such a memory circuit, which is combined with additional memory, to store some or all code from one or more modules.

[0158] The term memory circuit is a subset of the term computer-readable medium. The term computer-readable medium as used herein does not cover transient electrical or electromagnetic signals propagated through a medium (e.g., on a carrier wave); thus, the term computer-readable medium may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits or dynamic random access memory circuits), magnetic storage media (e.g., analog or digital magnetic tape or hard drive), and optical storage media (e.g., CD, DVD, or Blu-ray disc).

[0159] The apparatus and methods described in this application may be implemented in part or in whole by a special-purpose computer, which is created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks, flow chart components and other elements described above act as software specifications, which can be translated into computer programs by the routine work of a skilled technician or programmer.

[0160] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0161] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated by a compiler from source code; (iv) source code executed by an interpreter; (v) source code compiled and executed by a real-time compiler, etc. By way of example only, the source code may be compiled using languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language Fifth Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK, and The grammar of the language is used to write it.

Claims

1. A method for controlling automatic vehicle acceleration and braking, the method comprising: acquiring an image using at least one vehicle camera of a host vehicle; Extracting feature inputs of a machine learning model based on the obtained images; detecting one or more objects in the obtained image, the one or more objects including at least one pedestrian; assigning attention weights to regions of the obtained image based on locations of one or more objects in the obtained image; combining the attention weight with a corresponding one of the machine learning model feature inputs according to the obtained region of the image; executing the machine learning model to generate a crossing intention prediction output associated with at least one pedestrian; as well as In response to the crossing intent prediction output exceeding a crossing intent threshold, automatic braking of the host vehicle is controlled based on the position of the at least one pedestrian.

2. The method of claim 1, wherein assigning attention weights comprises: assigning a first intensity value to a first region of the obtained image corresponding to at least one pedestrian; as well as A second intensity value is assigned to a second region of the obtained image that does not correspond to the at least one pedestrian, and the first intensity value is greater than the second intensity value.

3. A method according to claim 2, wherein combining attention weights and machine learning model feature inputs includes generating a weighted sum of the machine learning model feature inputs according to the attention weights.

4. The method according to claim 1, wherein: The machine learning model includes a multi-layer perceptron; and Executing the machine learning model includes generating a crossing intention prediction output based on the output of the multi-layer perceptron.

5. The method according to claim 1, wherein extracting machine learning model feature input based on the obtained image includes providing the obtained image to multiple visual transformer layers to generate machine learning model feature input.

6. The method of claim 1, wherein executing the machine learning model comprises: Obtain multiple key values ​​based on machine learning model feature input; Perform classification queries based on machine learning model feature input; as well as Correlate categorical query output with multiple key values.

7. The method of claim 6, wherein executing the machine learning model comprises: Combining the attention weights with the classification query output and the relevance of multiple key values; as well as A normalized exponential function is performed on the combination of the attention weights and the relevance of the classification query output and multiple key values ​​to generate a crossing intent prediction output.

8. The method of claim 7, wherein executing the machine learning model comprises: Combining the output of the normalized exponential function with a plurality of key values ​​to generate an embedding vector; as well as The embedding vector is fed to a multi-layer perceptron to generate the crossing intention prediction output.

9. The method according to claim 1, further comprising: Provide training and testing data to machine learning models; comparing a plurality of crossing intent prediction outputs of the machine learning model based on the training data with labeled crossing intent outputs of the test data; Determines whether the accuracy of the comparison is greater than or equal to the specified accuracy threshold; In response to determining that the accuracy of the comparison is less than a specified accuracy threshold, adjusting parameters of the machine learning model and retraining the machine learning model; as well as In response to determining that the accuracy of the comparison is greater than or equal to the specified accuracy threshold, the machine learning model is saved for use in generating a crossing intent prediction output.

10. A vehicle control system for controlling vehicle braking based on vehicle camera image processing, the vehicle control system comprising: at least one vehicle camera configured to obtain an image from the front of the host vehicle; as well as A vehicle control module of a host vehicle, the vehicle control module being configured to: Extracting feature inputs of a machine learning model based on the obtained images; detecting one or more objects in the obtained image, the one or more objects including at least one pedestrian; assigning attention weights to regions of the obtained image based on locations of one or more objects in the obtained image; combining the attention weight with a corresponding one of the machine learning model feature inputs according to the obtained region of the image; executing the machine learning model to generate a crossing intention prediction output associated with at least one pedestrian; as well as In response to the crossing intent prediction output exceeding a crossing intent threshold, automatic braking of the host vehicle is controlled based on the position of the at least one pedestrian.

Images