Systems and methods for reducing missed detections of tracked objects in the vehicle's surrounding environment.

Abstract

This disclosure provides a system and method for receiving, at a processor associated with a vehicle and via one or more image sensors associated with the vehicle, image data associated with the environment surrounding the vehicle and corresponding to a first image captured at a first time, and supplementary image data associated with the environment surrounding the vehicle and corresponding to a second image captured at a second time. The provided system and method can determine, based on the received supplementary image data and a machine learning model, that a tracked object not detected in the second image but identified in the first image, and can determine, based on vehicle data and tracking data of the tracked object, that the tracked object should be present in the second image and perform remedial actions on the supplementary image data to identify the tracked object in the second image.

Description

Technical Field

[0001] This application relates to a system and method for reducing missed detections of tracked objects in the environment surrounding a vehicle. Background Technology

[0002] In autonomous and semi-autonomous vehicles, it can be useful to determine the presence of objects in the vicinity of the vehicle and the distance between such objects and the vehicle. One approach employs a neural network machine learning model to detect objects around the vehicle. However, what is needed is to reduce the negative false recognitions and missed detections of objects by the neural network. Summary of the Invention

[0003] Therefore, this document discloses a system and method comprising processing circuitry configured to: receive, at a processor associated with a vehicle and via one or more image sensors associated with the vehicle, image data associated with the environment surrounding the vehicle, wherein the image data corresponds to a first image captured by the one or more image sensors at a first time; and receive, at the processor and via the one or more image sensors, additional image data associated with the environment surrounding the vehicle, wherein the additional image data corresponds to a second image captured by the one or more image sensors at a second time following the first time. The processing circuitry can determine, based on the received additional image data and a machine learning model, that a tracked object identified in the first image is not detected in the second image. The provided system and method can also determine, based on vehicle data and tracking data of the tracked object, that the tracked object should be present in the image, and perform remedial actions on the additional image data to identify the object in the second image.

[0004] In some embodiments, the processing circuitry is configured to determine, based on received additional image data and the machine learning model, that the tracked object not detected in the second image is not identified in the first image by: inputting the image data into the machine learning model, and the machine learning model outputting a first output based on the input image data. Alternatively, the additional image data may be input into the machine learning model, and the machine learning model outputting a second output based on the input additional image data, and the provided system and method may determine, based on the second output of the machine learning model, that the second image does not include a depiction of the tracked object.

[0005] In some implementations, the vehicle data includes the vehicle's speed, the tracking data includes a first distance between the vehicle and the tracked object, and the processing circuitry is configured to determine, based on the vehicle data and the tracking data of the tracked object, that the tracked object should be present in the second image by: determining, based on the first output of the machine learning model, that the tracked object is located at the first distance from the vehicle at the first time; and determining, based on the first distance and the vehicle's speed, that the second output should indicate that the tracked object should be depicted in the second image.

[0006] In some implementations, the first output includes a bounding box associated with the location of the tracked object in the first image, and the processing circuitry is configured to perform the remedial action on the additional image data to identify the tracked object in the second image by determining a region of interest associated with the predicted location of the tracked object in the second image based on the bounding box, the first distance, and the vehicle's speed.

[0007] In some embodiments, the processing circuitry is also configured to perform the remedial action on the additional image data to identify the tracked object in the second image by: performing a computer vision convolution in the region of interest to identify the tracked object in the second image; and determining, based on the computer vision convolution, that the tracked object is located at a second distance from the vehicle at the second time.

[0008] In some implementations, the machine learning model is associated with a threshold confidence level for identifying objects in image data input to the machine learning model, and the processing circuitry is configured to perform the remedial action on the additional image data to identify the tracked object in the second image by reducing the threshold confidence level of a portion of the second image associated with the region of interest.

[0009] In some embodiments, the processing circuitry is configured to perform the remedial action on the supplementary image data to identify the tracked object in the second image by: feeding the supplementary image data to the machine learning model associated with the reduced threshold confidence level of the portion of the second image associated with the region of interest; and determining, based on a third output of the machine learning model received in response to feeding the supplementary image data to the machine learning model, that the tracked object is located at a second distance from the vehicle at the second time.

[0010] In some implementations, the vehicle is a first vehicle, the environment around the vehicle is defined by a predetermined distance from the sensor of the first vehicle, and the tracked object is a second vehicle within the predetermined distance.

[0011] This document discloses a non-transitory computer-readable medium having non-transitory computer-readable instructions encoded thereon. When executed by a processor associated with a vehicle, the non-transitory computer-readable instructions cause the processor to: receive image data associated with the environment surrounding the vehicle via one or more image sensors associated with the vehicle, wherein the image data corresponds to a first image captured by the one or more image sensors at a first time; and receive additional image data associated with the environment surrounding the vehicle via the one or more image sensors, wherein the additional image data corresponds to a second image captured by the one or more image sensors at a second time after the first time. The instructions can also be executed to cause the processor to determine, based on the received additional image data and a machine learning model, that a tracked object is not detected in the image, and, based on vehicle data and tracking data of the tracked object, determine that the tracked object should be present in the second image and perform remedial actions to identify the tracked object in the second image. Attached Figure Description

[0012] The present disclosure is described in detail with reference to the following accompanying drawings, which illustrate one or more various embodiments. The drawings are provided for illustrative purposes only and show only typical or exemplary embodiments. These drawings are provided to facilitate understanding of the concepts disclosed herein and should not be considered as limitations on the breadth, scope, or applicability of these concepts. It should be noted that these drawings are not necessarily drawn to scale for clarity and ease of illustration.

[0013] Figure 1 A block diagram of components of a system configured to perform remedial actions to identify objects in an image, according to some embodiments of the present disclosure, is shown.

[0014] Figure 2 An exemplary machine learning model for detecting objects in an image is shown according to some embodiments of the present disclosure;

[0015] Figure 3 A block diagram of components of a system configured to perform remedial actions to identify objects in an image, according to some embodiments of the present disclosure, is shown.

[0016] Figure 4 A flowchart illustrating an exemplary process for performing remedial actions to identify objects in an image, according to some embodiments of the present disclosure, is shown.

[0017] Figure 5 A flowchart illustrating an exemplary process for performing remedial actions to identify objects in an image, according to some embodiments of the present disclosure, is shown; and

[0018] Figure 6A flowchart illustrating an exemplary process for performing remedial actions to identify objects in an image, according to some embodiments of the present disclosure, is shown. Detailed Implementation

[0019] Figure 1 A block diagram of components of a system 100 according to some embodiments of the present disclosure is shown. The system is configured to perform remedial actions to identify objects in an image. System 100 may include a vehicle 101 that can communicate with a server 136 and user equipment 138 via one or more networks 105. Vehicle 101 may be an automobile (e.g., a sedan, car, truck, SUV, bus), a motorcycle, an aircraft (e.g., a drone), a vessel (e.g., a boat), or any other type of vehicle. In some embodiments, vehicle 101 may be configured to operate autonomously or semi-autonomously.

[0020] Vehicle 101 may include processing circuitry 102, which may include processor 104 and memory 106. Processor 104 may include a hardware processor, a software processor (e.g., a processor emulated using a virtual machine), or any combination thereof. In some embodiments, the combination of processor 104 and memory 106 may be referred to as processing circuitry 102 of vehicle 101. In some embodiments, processor 104 alone may be referred to as processing circuitry 102 of vehicle 101. Memory 106 may include hardware elements for non-transitory storage of commands or instructions that, when executed by processor 104, cause processor 104 to operate vehicle 101 according to the embodiments discussed above and below. Processing circuitry 102 may be communicatively connected to components of vehicle 101 via one or more wires or via a wireless connection.

[0021] Processing circuitry 102 may be communicatively connected to input interface 116 (e.g., steering wheel, touchscreen display, button, knob, microphone, or other audio capture device) via input circuitry 108. In some embodiments, the driver of vehicle 101 may be permitted to select specific settings in conjunction with the operation of vehicle 101 (e.g., to indicate whether vehicle 101 should operate autonomously). In some embodiments, processing circuitry 102 (and / or processing circuitry of one or more servers 136) may be configured to train and / or utilize machine learning models to detect tracked objects in the environment surrounding vehicle 101 in images (e.g., captured by one or more sensors 124, 126, 128, 130) and to perform remedial actions to identify objects in the images based on the determination that no tracked object was detected in the images.

[0022] In some embodiments, processing circuitry 102 may be communicatively connected to a GPS system 134 or other positioning device of vehicle 101, wherein the driver may interact with the GPS system via input interface 116. GPS system 134 may communicate with multiple satellites and / or server 136 located remotely from vehicle 101 to determine the driver's location and provide navigation directions to processing circuitry 102. Alternatively, positioning devices may operate on terrestrial signals (such as cellular phone signals, Wi-Fi signals, or ultra-wideband signals) to determine the location of electric vehicle 101. The determined location may be in any suitable form, such as geographic coordinates, street address, nearby landmarks (such as identification of the nearest charging station), or a marked location associated with the vehicle (e.g., the location of the user's home stored in memory 106). In some embodiments, processing circuitry 102 may use the determined location to determine whether vehicle 101 is within a threshold range of the marked location.

[0023] Processing circuitry 102 can be communicatively connected to display 112 and speaker 114 via output circuitry 110. Display 112 may be located at a head-up display on the dashboard and / or windshield of vehicle 101. For example, it may be used to generate an interface for GPS system 134 or an infotainment system for display, and display 112 may include an LCD display, OLED display, LED display, or any other type of display. Speaker 114 may be located anywhere within the cabin of vehicle 101, such as on the dashboard, on the interior of a door, etc.

[0024] Processing circuitry 102 may be communicatively connected (e.g., via sensor interface 113) to sensors (e.g., front sensor 124, rear sensor 126, left-side sensor 128, right-side sensor 130, orientation sensor 118, and speed sensor 120). Orientation sensor 118 may be an inclinometer, accelerometer, slew rate sensor, any other pitch sensor, or any combination thereof, and may be configured to provide processing circuitry 102 with vehicle orientation values ​​(e.g., vehicle pitch and / or vehicle roll). Speed ​​sensor 120 may be a speedometer, GPS sensor, or any combination thereof, and may be configured to provide processing circuitry 102 with a reading of the vehicle's current speed. Front sensor 124, rear sensor 126, left-side sensor 128, and / or right-side sensor 130 may be located at multiple positions on vehicle 101 and may be one or more of a variety of types, such as image sensors, ultrasonic sensors, radar sensors, LED sensors, LiDAR sensors, etc., configured to measure the distance between vehicle 101 and objects in the environment surrounding the vehicle (e.g., by outputting light or radio wave signals and measuring the time and / or intensity of the return signal to be detected, and / or performing image processing on images of the environment surrounding vehicle 101 captured by image sensors). In some embodiments, processing circuitry 102 may, when determining the predicted distance between vehicle 101 and an object, consider, for example, the acceleration of vehicle 101 based on sensor data generated by orientation sensor 118.

[0025] Processing circuitry 102 is communicatively connected to battery system 132, which can be configured to provide power to one or more components of vehicle 101 during operation. In some embodiments, vehicle 101 may be an electric vehicle or a hybrid electric vehicle.

[0026] In some embodiments, communication circuitry 135 (e.g., including a network transceiver) and / or user equipment 138 (e.g., a mobile device) may communicate with one or more servers 136 (e.g., via communication network 105 such as the Internet), which may be configured to perform any suitable portion of the processes described above and below. Server 136 may include multiple servers configured to remotely provide cloud computing resources (e.g., storage, processing, software applications, etc.) to other components of system 100 (e.g., user equipment 138 and / or vehicle 101). In some embodiments, server 136 may be associated with the manufacturer of vehicle 101. Components of system 100 may be configured to communicate via any suitable wireless (or wired) communication path. For example, user equipment 138 may be configured to communicate with vehicle 101 using short-range wireless communication technologies such as, for example, Bluetooth Low Energy (BLE) or Near Field Communication (NFC), and user equipment 138 and vehicle 101 may communicate with server 136 via network 105 (e.g., the Internet, local area network, wide area network, satellite network, cellular network, etc.). In some implementations, server 136 may include one or more processors configured to perform processing-intensive tasks to assist vehicle 101 in autonomous or semi-autonomous navigation.

[0027] It should be understood that Figure 1 Only some components of vehicle 101 are shown, and it should be understood that vehicle 101 also includes other components commonly found in vehicles (e.g., electric vehicles), such as motors, brakes, wheels, wheel controls, turn signals, windows, doors, etc.

[0028] Figure 2An exemplary machine learning model for detecting objects in an image is illustrated according to some embodiments of the present disclosure. Machine learning model 208 may be, for example, a convolutional neural network (CNN), or any other suitable machine learning model trained to accept an input image 212 of the environment surrounding vehicle 101 and output a determination 214 of the location and / or classification of objects of a certain type or class depicted in image 212. Training data 202 may include images of the environment surrounding the vehicle and has been assigned labels. For example, each training image may be associated with a vector of any suitable number of dimensions encoding information indicating the presence of one or more objects in the training image, specifying the class or type of the object (e.g., car, pedestrian, motorcycle, animal, train, bicycle, road sign, terrain, background, etc.) and parameters of the bounding box around the object (e.g., x-coordinate, y-coordinate, midpoint, height, width) and / or indicating the distance from the vehicle to the object annotation if one or more objects are present in the training image. Training framework 204 may train an untrained machine learning model 206 using the processing resources described herein to generate a trained machine learning model 208. In some implementations, initial weights can be selected randomly or through pre-training using a deep belief network. Training can be performed in a supervised, partially supervised, or unsupervised manner.

[0029] A trainable machine learning model 208 can output a prediction of whether input image data (e.g., input image 212) contains an object and one or more parameters (e.g., position and coordinates) of the bounding box around the object. In some embodiments, object predictions associated with probabilities below a certain threshold (e.g., 0.4) can be discarded. In some embodiments, the input image data (e.g., image 212) can be divided into cells or regions according to a grid (e.g., forming an array of regions that aggregate to constitute the image), and analysis can be performed on each region of the image to output a prediction of the presence of an object and the predicted bounding box coordinates within that region. For example, filters or kernels of any suitable size (e.g., 3×3 pixels) can be overlaid on each region of the image to perform convolution, e.g., multiplying each overlapping pixel and summing each product, and input into the machine learning model that outputs the prediction.

[0030] In some implementations (e.g., such as if using a regression classifier), supervised learning can be used to train an untrained machine learning model 206, where the training dataset 202 includes inputs paired with the desired output, or where the training dataset 202 includes inputs with known outputs, and the outputs of the neural network are manually graded. In some implementations, the untrained machine learning model 206 can be trained in a supervised manner. The training framework 204 can process inputs from the training dataset 202 and compare the resulting outputs with a set of expected or desired outputs. In some implementations, backpropagation errors can then be propagated through the untrained machine learning model 206. The training framework 204 can adjust the weights controlling the untrained machine learning model 206. The training framework 204 may include tools for monitoring how well the untrained machine learning model 206 converges toward a model (such as a trained machine learning model 208) that is suitable for generating the correct answer in result 214 based on known input data (such as new data 212). In some implementations, training framework 204 repeatedly trains an untrained neural network 206 while adjusting weights to refine the output of the untrained neural network 206 using a loss function and adjustment procedures such as stochastic gradient descent. In some implementations, training framework 204 trains the untrained machine learning model 206 until the untrained neural network 206 reaches the desired accuracy. The trained machine learning model 208 can then be deployed to implement any number of machine learning operations. In some implementations, the machine learning model can be trained to classify pixels of input image data into drivable and non-drivable spaces for applications such as autonomous navigation.

[0031] In some implementations, unsupervised learning can be used to train an untrained machine learning model 206, where the untrained machine learning model 206 attempts to train itself using unlabeled data. In some implementations, the unsupervised learning training dataset 202 may include input data without any associated output data or "ground truth" data. The untrained machine learning model 206 can learn groupings within the training dataset 202 and can determine how each input relates to the untrained dataset 202. In some implementations, unsupervised training can be used to generate self-organizing graphs, which are types of trained machine learning models 208 capable of performing operations that can be used to reduce the dimensionality of new data 212. Unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the new dataset 212 that deviate from normal or existing patterns in the new dataset 212. In some implementations, semi-supervised learning can be used, which is a technique where the training dataset 202 includes a mixture of labeled and unlabeled data. Therefore, training frameworks 204 such as transfer learning techniques can be used to perform incremental learning. Such incremental learning enables a trained machine learning model 208 to adapt to new data 212 without forgetting the knowledge instilled in the network during initial training.

[0032] In some implementations, model 208 may be configured to employ a softmax layer (e.g., to obtain a normalized probability distribution), such as multiple probabilities output by the model combining one or more categories or classes of the object (regardless of whether the object is present) and the location of the bounding box. In some implementations, model 208 may be configured to apply an argmax function to such probabilities (e.g., to set each probability to zero or one).

[0033] Figure 3 A block diagram of components of a system 300 configured to perform remedial actions to identify objects in an image, according to some embodiments of the present disclosure, is shown. Image data, including one or more images of the environment surrounding a vehicle 301, may be generated by, for example, the vehicle 301 (which may correspond to...). Figure 1 Image sensors 324, 328 and / or 330 of vehicle 101 (each of which may correspond to) Figure 1One of the sensors 124, 126, 128, and 130 is captured at 302. Such image sensors can be mounted or positioned at any suitable part of vehicle 301 to facilitate the capture of images of the entire area or environment (or any suitable part thereof) around vehicle 301 when vehicle 301 is stationary or in motion. For example, one or more of image sensors 324, 328, and 330 can be located at one or more locations on the exterior and / or interior of vehicle 301, such as sensor 324 at the windshield, front bumper, and roof of vehicle 301, and sensors 324, 328, and 330 at the rear bumper, roof, windshield, corresponding side mirrors, or any combination thereof. A series of images, including any suitable number of images, can be captured by image sensors 324, 328, and / or 330. In some implementations, images can be captured repeatedly, for example, at a predetermined frequency, to capture the surrounding environment of vehicle 301 over time.

[0034] At 304, the processing circuitry 102 of vehicle 101 (and / or the processing circuitry of server 136) may be configured to perform preprocessing, such as extracting suitable features from the captured image 302, and / or converting the identified features of the captured image 302 into a vector or matrix representation, and / or matching the formatting of the identified features of the captured image with the formatting of the training data 202, normalizing, resizing, minimizing, etc. In some embodiments, preprocessing may include brightening an image or a portion thereof, darkening an image or a portion thereof, color shifting an image (e.g., in a color scheme, from color to grayscale or other mapping), cropping an image, scaling an image, adjusting the aspect ratio of an image, adjusting the contrast of an image, performing any other suitable processing for preparing an image, or any combination thereof.

[0035] At 306, the processing circuitry 102 of vehicle 101 (and / or the processing circuitry of server 136) may cause the captured image 302 (and / or its features extracted during preprocessing 304) to be input into machine learning model 306 (which may correspond to trained machine learning model 208). Machine learning model 306 may output (at 308) an indication at 311 of a class of identified object 310 (e.g., a car) and a probability (e.g., a confidence score) associated with the presence of object 310 in the captured image and / or the detected object class, as well as parameters of bounding box 312, based on one or more input images (e.g., including a first image, and received by the processor 104 of vehicle 101 and / or machine learning model 306 from one or more image sensors 324, 328, 330 at a first time 313) or image data corresponding to the one or more input images. Vehicle 301 may travel on roads (e.g., highways or local roads) or off-road. It should be understood that system 300 can be implemented with any type of vehicle on any type of terrain. In some embodiments, notifications associated with output 308 may be provided to the user (e.g., to generate a representation of object 310 to be displayed at display 112 to depict the real-time position of object 310 relative to vehicle 301). In some embodiments, specific actions may be suggested or automatically performed based on the output 308 of machine learning model 306 at a first time 313, such as providing specific warning indications or suggested navigation routes related to the position of object 310, performing autonomous driving based on the position of object 310, etc.

[0036] The output of machine learning model 306 can be configured to specify and / or plot 2D prediction parameters of the boundary shapes (e.g., bounding boxes, bounding polygons, bounding triangles, bounding ellipses, bounding circles, etc.) surrounding candidate objects (e.g., vehicles, people, animals, or other obstacles) located in front of, behind, or to the side of vehicle 301. In some embodiments, multiple images of the surrounding environment of vehicle 101 captured by sensors 324, 328, 330 can be input into machine learning model 306, for example, in a first-in-first-out manner. In some embodiments, machine learning model 306 can be a neural network (e.g., CNN) and / or implemented as a classification algorithm (e.g., primal Bayesian algorithm, support vector machine, logistic regression, linear regression, random forest, nearest neighbor algorithm, and / or any other suitable classifier). The classifier is discussed in more detail in conjunction with co-owned U.S. Application No. 17 / 225,518, filed April 8, 2021, which is incorporated herein by reference in its entirety.

[0037] The computation of bounding boxes or other boundary mechanisms can be performed in any suitable manner, such as through known computer vision-based methods and processes used to identify objects and fit boxes to their outer edges. For example, any method such as edge detection methods, feature search methods, probabilistic object models, graph matching, or histograms of gradient orientations (HOG) fed into a classifier (such as a support vector machine, Haar Cascade classifier, etc.) can be used to identify and locate objects and their outer boundaries within an image. Consider any suitable method. In some implementations, bounding boxes can be fitted to the identified objects in any manner, such as by drawing rectangular boxes whose edges are parallel to the camera coordinate system's axis orientation and each tangent to the outer edge of the identified object. In some implementations, bounding boxes can be fitted to the identified objects by drawing rectangular boxes whose vertical edges are parallel to the axis orientation of vehicle 301, the orientation of which is determined by an orientation sensor of vehicle 301. Neural networks and bounding boxes are discussed in more detail in co-owned application number 17 / 384,510, filed July 23, 2021, which is incorporated herein by reference in its entirety.

[0038] Machine learning model 306 may receive additional image data at a second time 315 (e.g., within a predefined time period such as a few seconds from the first time 313) and may output (at 314) indication 311 based on such input additional image data. The additional image data may include an image (including a second image) captured at the second time 315 by one or more image sensors 324, 328, 330, or a representation of that image, and may be associated with the environment surrounding vehicle 301, for example, an environment similar to the environment associated with the first image captured at the first time 313, but at a future time rather than the first time 313. Indication 311 may indicate that the probability of the tracked object 310 or any other object associated with the additional image data captured at the second time 315 being present in the environment surrounding vehicle 301 in the captured image is below a specific predefined threshold (e.g., 0.4), and therefore, it is determined that the object is not present in the environment surrounding vehicle 301.

[0039] In some implementations, a tracker portion 316, which may be implemented by the processing circuitry 102 of vehicle 101 (and / or the processing circuitry of server 136), may be used to combine or stitch together the outputs from machine learning model 306 to track objects in the environment surrounding vehicle 301 over time. In some implementations, the tracker portion 316 may be used to examine each output of machine learning model 306 for whether it is a missed detection (e.g., an object such as tracked object 310 identified in a first image captured at a first time 313 should have been present in the input supplementary image data captured at a second time 315, but the output of machine learning model 306 does not indicate the presence of such an object in the input supplementary image data) or a false positive (e.g., the output of machine learning model 306 indicates the presence of an object in the input supplementary image data captured at a second time 315, but the object should not have been present in that input supplementary image data). For example, tracker portion 316 may determine or otherwise receive information indicating a first distance between vehicle 301 and detected object 310 at a first time 313 (e.g., this distance may be determined based on output 308 and / or other image processing techniques, such as, for example, by determining the number of pixels between vehicle 301 and the bounding box 312 associated with object 310), and tracker portion 316 may determine or otherwise receive information indicating the speed of vehicle 301 at the first time 313 (e.g., from speed sensor 120). While tracker portion 316 is described as receiving information from images (e.g., video frames) to track objects over time, it should be understood that tracker portion 316, as described herein, may also receive information from other sensor modalities to generate a fused list of tracked objects. For example, tracker portion 316 may also receive information from one or more of radar sensors, LiDAR sensors, or ultrasonic sensors. In some embodiments, tracker portion 316 is a first tracker, and a separate second tracker is used for other sensor modalities. The outputs of the first tracker (i.e., tracker section 316) and the second tracker can then be combined to produce a fused list of tracked objects. Alternatively, the output of the first tracker (i.e., tracker section 316) can be input to the second tracker (or vice versa) to produce a fused list of tracked objects.

[0040] Tracker section 316 can determine vehicle data and tracking data, such as, for example, a determined speed of vehicle 301 at time 313 (e.g., absolute or relative speed between vehicle 301 and object 310) and the position of vehicle 301, and the tracking data, for example, the position of object 310 at time 313 and a determined distance between vehicle 301 and object 310 at the first time 313. Based on such vehicle data and tracking data, a predicted position of object 310 at a second time 315 associated with additional image data can be determined. In some embodiments, multiple images captured at multiple times prior to the first time 313 can be analyzed to determine the speed of vehicle 301 at time 313, the positions of vehicle 301 and object 310, the first distance between vehicle 301 and object 310, and / or the speed of object 310 (e.g., by comparing multiple frames depicting object 310 within a predefined time period prior to the current time). For example, if at time 315 the distance between object 310 and vehicle 301 is determined to be 5 meters, and each of vehicle 301 and object 310 is determined to be traveling at approximately the same speed, then tracker section 316 can determine that the output 314 of machine learning model 306 should indicate the presence of tracked object 310 at a specific location near vehicle 301. On the other hand, if no object is detected at time 313, but at time 315 an object is detected adjacent to vehicle 301, tracker section 316 can examine whether such an object might be a false identification (e.g., perform a check to verify that the object identified at 315 includes features matching objects of a known class). At 320, tracker section 316 can be configured to specify or draw an indication of a region of interest (ROI) 318 corresponding to the location that tracker section 316 has determined should be at the second time 315, associated with output 314.

[0041] At 322, processing circuitry 102 may perform one or more remedial actions on additional image data captured at a second time 315 based on the determination of ROI 318 at time 315. Remedial actions may be performed to reduce potential missed detections or false positives associated with output 314 by pulling information from a previous frame (e.g., associated with time t) into the current frame (e.g., associated with time t+1) or the next frame. In some embodiments, remedial actions may include performing one or more conventional computer vision techniques to search for missed-detection tracking objects 310 within ROI 318. For example, computer vision convolution techniques, such as kernels or filters or masks (e.g., any suitable number of pixels, e.g., 3×3, and convolved with a portion of the image), may be passed through portions of the image during edge detection (e.g., to identify vertical and horizontal edges) to detect the location of missed-detection tracking objects 310. In some embodiments, the image may be converted to grayscale to facilitate image processing thereon.

[0042] In some implementations, when performing remedial actions on additional image data captured at the second time point 315, features of the image within the ROI 318 can be extracted (or such features can be acquired by the tracker section 316 based on preprocessing 304), and / or boundaries within the ROI 318 can be identified (e.g., by identifying changes in pixel intensity or gradient between adjacent pixels that exceed a predefined threshold) to identify the location of the tracked object 310. In some implementations, the extracted features within the ROI 318 can be compared with features stored in a database (e.g., locally at vehicle 301 or at a remote server 136) in association with a corresponding object class (e.g., car, motorcycle, truck, pedestrian) to detect the location of the tracked object 310 within the ROI 318 or in another part of the image. For example, pattern matching can be performed between the extracted features and features stored at server 136. In some implementations, missed tracked objects 310 can be identified based on a color threshold. In some implementations, a hybrid approach including conventional computer vision techniques as well as deep learning and machine learning techniques can be employed.

[0043] In some implementations, remedial action may include feeding back the image associated with output 314 along with an indication to temporarily reduce the threshold confidence level of the ROI 318 of the machine learning model 306 relative to such images to the machine learning model 306. For example, if the threshold confidence level is typically 0.4 (e.g., discarding object recognition associated with confidence levels below 0.4), the tracker portion 316 may instruct the processing circuitry 102 of vehicle 101 to reduce the confidence level to, for example, 0.3 based on the ROI 318 of the image fed back to machine learning model 306. Thus, if the threshold confidence level is reduced to 0.3, the updated output 314 may indicate the presence of tracked object 310, and processing may be performed accordingly (e.g., autonomous navigation or notification to the operator of vehicle 301). In some implementations, the processing circuit 102 may determine that a lower confidence level may be due to object occlusion, and a lower threshold confidence level may be appropriate when the expected vehicle is occluded, for example, if the tracker 316 determines that the object has overtaken another object, and therefore one of the expected objects is occluded in the captured image.

[0044] In some implementations, the threshold confidence level can be dynamically adjusted over inference time, for example, before or during the computation performed by the machine learning model 306 to calculate the prediction made at output 314, within any suitable number of subsequent frames based on the calculations performed by the tracker section 316 that indicate the presence of the tracked object 310. In other words, remedial actions can be performed in system 300 using on-the-fly inference (e.g., to understand what a particular vehicle corresponding to the detected object looks like). It should be understood that, although Figure 3 The output 314 from the machine learning model 306 is depicted prior to the output 320 from the tracker section 316, but it should be understood that... Figure 3 The steps can occur in any suitable order (e.g., tracker 316 can determine ROI 318 before the output 314 from machine learning model 306 is generated).

[0045] Figure 4 A flowchart of an exemplary process 400 for performing remedial actions to identify objects in an image, according to some embodiments of the present disclosure, is shown. Process 400 may be executed at least in part by processing circuitry 102 of vehicle 101 and / or processing circuitry of server 136, which may implement machine learning model 406 and tracker portion 416 (which may respectively correspond to...) Figure 3 Machine learning model 306 and tracker part 316).

[0046] At 402, processing circuitry 102 may determine the location of one or more objects or actors at a specific time based on the output of machine learning model 406. For example, processing circuitry 102 may determine that at a first time (time t indicated at 402), the output of machine learning model 406 indicates the presence of multiple objects or actors in the environment surrounding vehicle 301, such as object 310 and one or more other additional objects (e.g., additional vehicles, pedestrians, etc.). Processing circuitry 102 may cause such determined locations of objects (e.g., vehicle 301, object 310, and additional objects) to be received from machine learning model 406 at tracker section 416.

[0047] Processing circuitry 102 can use the motion of vehicle 301 (e.g., a speed reading indicated by speed sensor 120) and the determined speed of vehicle 310 (and the distance between them) at 404 to predict the respective positions of vehicle 301 and object 310 in a frame following time t indicated at 402 (time t+1 indicated at 404), as depicted in a second image not yet captured. Additionally, processing circuitry 102 can predict at 404 the respective position of vehicle 301 and each of one or more other objects, i.e., detected objects (e.g., additional objects), in a frame following time t+1 indicated at 404. This prediction can be based on the speeds of vehicle 301 and such additional vehicles, and the distance between them. In some embodiments, the speeds of object 310 and additional objects can be determined by tracker section 416 based on the number of pixels separating vehicle 301 from object 310 and the number of pixels separating vehicle 301 from additional objects, respectively, in the image associated with time t indicated at 402, or using any other suitable technique.

[0048] At 406, processing circuitry 102 can use the predicted positions at 404 to draw a Region of Interest (ROI) 318 associated with object 310 and an additional ROI associated with an additional object, where each of ROI 318 and the additional ROI represents a predicted position of object 310 and the additional object at a future time t+1 indicated at 404. For example, based on the speed of vehicle 301 at time t indicated at 402 and the distance between vehicle 301 and object 310, processing circuitry 102 can predict that object 310 should be located at the position corresponding to ROI 318 at a future time t+1 indicated at 404.

[0049] At 408, processing circuitry 102 may search the output of machine learning model 406 to identify missed detections or detections associated with low confidence levels (e.g., below a predefined threshold confidence level). For example, one or more images corresponding to time t+1 indicated at 404 may be input to machine learning model 406, and machine learning model 406 may output an indication of the presence of one or more objects in one of the more input images, and if so, an indication of the corresponding class for each object and the corresponding bounding box associated with the location for each object. At 408, processing circuitry 102 may determine that such output from the machine learning model associated with time 315 is associated with one or more relatively low confidence levels (e.g., 0.3 and 0.2 respectively relative to object 310 associated with ROI 318 and additional objects associated with additional ROIs), otherwise it is a candidate for a missed detection or a false positive. That is, although object 310 and additional objects are present in one or more input images corresponding to time t+1 indicated at 404, the output of machine learning model 406 may not be able to detect their presence with sufficient confidence, for example, if such objects are occluded or only a small portion is depicted in the input images. In some implementations, the output of machine learning model 406 may be binary to indicate whether an object was detected or not in a specific region of interest, and / or such binary decisions may be obtained using the argmax function.

[0050] At 410, processing circuitry 102 can determine a difference between the ROI identified at 406 associated with time t+1 and the output of machine learning model 406 at 408 indicating that the image associated with time t+1 does not contain any objects (or contains more or fewer objects than the number identified at 406). In response to determining the existence of such a difference, processing circuitry 102 can determine the possibility that one or more objects were missed, and processing circuitry 102 can specify that in the next one or more frames (e.g., at time t+1 and the associated time t+2 after time t), machine learning model 406 should allow the detected objects in ROI 318 and additional ROIs to be forwarded to tracker section 316, even if a lower confidence level is associated with one or more detected objects. For example, processing circuitry 102 may cause machine learning model 306 to reduce the threshold confidence level (e.g., from 0.4 to 0.3 or 0.2 probability or confidence score) associated with determining the presence of an object in an input image, so that future outputs that can be associated with a confidence score of 0.3 can indicate the presence of one or more objects in the image, rather than discarding the image if the initial threshold of 0.4 is to be utilized. In some embodiments, in addition to reducing the confidence level of machine learning model 306 or as an alternative, processing circuitry 102 may perform conventional computer vision convolutional techniques to identify the location of object 310 within ROI 318 and the location of additional objects within additional ROIs. Based on one or more such remedial actions, the threshold used to determine the presence of object 310 and additional objects (and any other objects) in subsequent images associated with time t+2 and with a suitable number of future times may be reduced, and processing may be performed accordingly, such as by considering the location of object 310 during autonomous navigation.

[0051] Figure 5 A flowchart of an exemplary process 500 for performing remedial actions to identify an object in an image, according to some embodiments of the present disclosure, is shown. Process 500 may be performed at least in part by the processing circuitry 102 of vehicle 101 and / or the processing circuitry of server 136.

[0052] At 502, the processing circuitry 102 of vehicle 101 may receive image data associated with the environment surrounding vehicle 301 via one or more image sensors 324, 328, 330. Such image data may correspond to a first image captured by one or more image sensors 324, 328, 330 at a first time 313.

[0053] At 504, the processing circuitry 102 of vehicle 101 may receive image data associated with the environment surrounding vehicle 301 via one or more image sensors 324, 328, 330. Such image data may correspond to a second image captured by one or more image sensors 324, 328, 330 at a second time 315.

[0054] At 506, the processing circuitry 102 of vehicle 101 can determine, based on supplementary image data corresponding to the second image captured at the second time 315 and on machine learning model 306, that the tracked object 310 was not detected in the second image. For example, the tracked object 310 may have been tracked in a previous image or frame captured before the second time 315 (e.g., at the first time 313), but the output 314 associated with the input supplementary image data captured at the second time 315 may indicate that the tracked object 310 is not present in the second image.

[0055] At 508, the processing circuitry 102 of vehicle 101 can determine, based on vehicle data and tracking data of object 310, whether the tracked object 310 should be present in the second image captured at 302 and associated with the second time 315 (identified in the first image captured at time 313). In some embodiments, tracker portion 316 can determine the ROI 318 corresponding to the predicted location of the tracked object 310 based on the speed of vehicle 301 at time 313 and / or time 315 and based on the distance between the earlier tracked object 310 and vehicle 301. Such determination can be made even if the output of machine learning model 306 associated with time 315 indicates a low probability of the presence of object 310 (e.g., below a confidence threshold). In some embodiments, 508 can be performed prior to 506, for example, before capturing additional image data associated with the environment around vehicle 301 at the second time 315, and a prediction about whether a tracked object should be present at the second time 315 can be made based on vehicle data and tracking data associated with time 313.

[0056] At 510, the processing circuitry 102 of vehicle 101 may perform remedial actions on the additional image data to identify the tracked object 310 in a second image associated with time 315. For example, the processing circuitry 102 of vehicle 101 may utilize computer vision convolution techniques to identify the location of the tracked object 310 within ROI 318. Additionally or alternatively, the processing circuitry 102 of vehicle 101 may cause the threshold confidence level of machine learning model 306 to be reduced in association with ROI 318, for example, so that output 314 can make an identification of the detected object and be used to perform subsequent processing, such as performing autonomous navigation. In some embodiments, the reduced threshold confidence level of machine learning model 306 may be applied to ROIs of one or more images input to machine learning model 306 after time 315.

[0057] Figure 6 A flowchart of an exemplary process 600 for performing remedial actions to identify an object in an image, according to some embodiments of the present disclosure, is shown. Process 600 may be performed at least in part by the processing circuitry 102 of vehicle 101 and / or the processing circuitry of server 136.

[0058] At 602, the processing circuitry 102 of vehicle 101 and / or the processing circuitry of server 136 can train a machine learning model 206 using training data 202 including images of the environment surrounding the vehicle. This training data can be annotated with labels, for example, regarding the presence of objects in the training images, and if objects are present, annotated with indicators of the class or type of the objects present in the images and the bounding box parameters surrounding the objects. In some embodiments, training the machine learning model 206 can produce a trained machine learning model 208, which can be, for example, a convolutional neural network.

[0059] At 604, image sensors 124, 126, 128 and / or 130 may capture image data corresponding to or including a first image of the environment surrounding vehicle 301 at a first time 313. For example, vehicle 301 may be stationary, or may be traveling on a road (e.g., a highway) or off-road.

[0060] At 606, image data corresponding to or including a first image of the environment surrounding vehicle 301 at the first time 313 may be input by processing circuitry 102 to a trained machine learning model 208 (which may correspond to machine learning model 306), and processing circuitry 102 may receive a first output of machine learning model 306 based on the input of image data to machine learning model 306. In some embodiments, preprocessing 304 may be performed on such images before inputting the first image into machine learning model 306, for example to extract relevant features and / or perform any other suitable processing.

[0061] At 608, processing circuitry 102 may determine, based on the first output of a trained machine learning model, that the tracked object 310 is located at a first distance from vehicle 301 at a first time 313. In some embodiments, the output of the machine learning model may indicate the probability of the presence of an object in the input image data, parameters of the bounding box 312 surrounding the detected object, and the identification of the object's class (e.g., car, pedestrian, truck, bicycle, etc.). At 610, processing circuitry 102 may perform navigation based on the position of tracked object 310. For example, processing circuitry 102 may cause vehicle 301 (which may be in autonomous driving mode) to change lanes in front of object 310 in response to determining that vehicle 301 is sufficiently far ahead of object 310.

[0062] At 612, image sensors 124, 126, 128, and / or 130 may capture additional image data corresponding to or including a second image of the environment surrounding vehicle 301 captured at a second time 315 after the first time 313. In some embodiments, the second time 315 may occur immediately after the first time 313. At 614, such additional image data corresponding to or including the second image may be input to a trained machine learning model 306, and a second output may be received from the machine learning model 306 indicating the presence of one or more objects in the second image and / or indicating bounding box parameters associated with one or more objects and / or the type or class of one or more detected objects.

[0063] At 616, processing circuitry 102 may determine, based on the received additional image data and the second output of machine learning model 306, whether the tracked object 310 was not detected in the second image. For example, if processing circuitry 102 determines that the output probability associated with a potential object in the second image is below a predefined confidence level threshold, processing may proceed to 620. In some embodiments, if an object is detected, processing may proceed to 618.

[0064] At 618, processing circuitry 102 may perform navigation based on the position of the tracked object 310, as indicated in a second output 314, for example, indicating that the confidence level associated with the detected object 310 exceeds a predefined threshold. In some embodiments, the processing at 618 may determine whether the output of machine learning model 306 is a false positive or false negative. For example, even if the probability associated with the detected object and output by machine learning model 306 exceeds a predefined threshold, processing circuitry 102 may perform one or more checks regarding whether such a detected object corresponds to a false positive or false negative (e.g., the object is not depicted in other images among several recently captured images), and then processing may proceed to 620.

[0065] At 620, processing circuitry 102 can identify a first distance (output by machine learning model 306) between vehicle 301 and tracked object 310 in a first image associated with time 313 and determined at 608, and the speed of vehicle 301 (e.g., indicated by speed sensor 120). Processing circuitry 102 can determine, based on such determined first distance and vehicle speed, whether an object should be present in a second image. If so, processing can proceed to 622. Otherwise, processing can proceed to 618. In some embodiments, determining that an object should be present in a second image associated with a second time 315 may include identifying or drawing a Region of Interest (ROI) 318 associated with the predicted location of tracked object 310 in the second image.

[0066] At 622, processing circuitry 102 may determine a Region of Interest (ROI) 318 associated with the predicted location of the tracked object in the second image. For example, processing circuitry 102 may determine that the tracked object 310 is expected to be within ROI 318 at time 315 based on the speed of one or more of the vehicle 101 and the tracked object 310 at time 313 and the distance between the vehicle 101 and the object 310 at time 313.

[0067] At 624, processing circuitry 102 may perform computer vision convolution to identify the location of the tracked object 310 in ROI 318 and / or reduce the threshold confidence level of machine learning model 306 for a second image (e.g., relative to a portion of the image associated with ROI 318) and feed such a second image back to machine learning model 306. For example, one or more edge or boundary detection techniques may be used to identify the location of the tracked object 310. Additionally or alternatively, the threshold confidence level (e.g., 0.4) may be reduced (e.g., reduced to 0.3) so that output 314 can be considered a valid detection of object 310, and thus, output 314 can be used to perform processing at vehicle 301, such as performing autonomous navigation and / or outputting notifications to the occupants of vehicle 301.

[0068] The foregoing is merely illustrative of the principles of this disclosure, and various modifications can be made by those skilled in the art without departing from the scope of this disclosure. The above embodiments are presented for illustrative purposes and not for limitation. This disclosure may also take many forms other than those expressly described herein. Therefore, it should be emphasized that this disclosure is not limited to the methods, systems, and instruments expressly disclosed, but is intended to include variations and modifications thereof, which are within the spirit of the following claims.

Claims

1. A method for reducing missed detections of tracked objects in the environment surrounding a vehicle, the method comprising: Image data associated with the environment surrounding the vehicle is received at a processor associated with the vehicle and via one or more image sensors associated with the vehicle, wherein the image data corresponds to a first image captured by the one or more image sensors at a first time. Additional image data associated with the environment around the vehicle is received at the processor and via the one or more image sensors, wherein the additional image data corresponds to a second image captured by the one or more image sensors at a second time after the first time. The processor determines, based on the received additional image data and a machine learning model, that the tracked object identified in the first image is not detected in the second image; The processor determines, based on vehicle data and tracking data of the tracked object, that the tracked object should be present in the second image; as well as The processor performs a remedial action on the additional image data to identify the tracked object in the second image.

2. The method according to claim 1, wherein: Determining, based on the received additional image data and the machine learning model, that the tracked object identified in the first image was not detected in the second image includes: The image data is input into the machine learning model, wherein the machine learning model outputs a first output based on the input image data; The additional image data is input into the machine learning model, wherein the machine learning model outputs a second output based on the input additional image data; and Based on the second output of the machine learning model, it is determined that the second image does not include a depiction of the tracked object.

3. The method according to claim 2, wherein: The vehicle data includes the vehicle's speed; The tracking data includes a first distance between the tracked object and the vehicle at the first time; and The processor determines, based on the vehicle data and the tracking data of the tracked object, that the tracked object should be present in the second image, including: Based on the first output of the machine learning model, it is determined that the tracked object is located at the first distance from the vehicle at the first time. as well as Based on the first distance and the vehicle's speed, the second output should indicate the depiction of the tracked object in the second image.

4. The method according to claim 3, wherein: The first output includes a bounding box associated with the location of the tracked object in the first image; as well as The processor performing the remedial action on the additional image data to identify the tracked object in the second image includes: A region of interest associated with the predicted location of the tracked object in the second image is determined based on the bounding box, the first distance, and the speed of the vehicle.

5. The method of claim 4, wherein performing the remedial action on the additional image data to identify the tracked object in the second image further comprises: Perform computer vision convolution within the region of interest to identify the tracked object in the second image; as well as The computer vision convolution determines that the tracked object is located at a second distance from the vehicle at the second time point.

6. The method according to claim 4, wherein: The machine learning model is associated with a threshold confidence level for identifying objects in image data input to the machine learning model; as well as Performing the remedial action on the additional image data to identify the tracked object in the second image further includes: Reduce the threshold confidence level of the portion of the second image associated with the region of interest.

7. The method of claim 6, wherein performing the remedial action on the additional image data to identify the tracked object in the second image comprises: The additional image data is fed back to the machine learning model associated with the reduced threshold confidence level of the portion of the second image associated with the region of interest; as well as The tracking object is determined to be at a second distance from the vehicle at the second time point based on a third output of the machine learning model received in response to feeding the additional image data back to the machine learning model.

8. The method according to claim 1, wherein: The vehicle in question is the first vehicle; The environment surrounding the first vehicle is defined by a predetermined distance from the one or more image sensors of the first vehicle; and The object being tracked is a second vehicle within the predetermined distance.

9. A system for reducing missed detections of tracked objects in the environment surrounding a vehicle, the system comprising: One or more image sensors, said one or more image sensors being associated with the vehicle; and Processing circuitry, associated with and configured to: Image data associated with the environment surrounding the vehicle is received via the one or more image sensors, wherein the image data corresponds to a first image captured by the one or more image sensors at a first time. Additional image data associated with the environment around the vehicle is received via the one or more image sensors, wherein the additional image data corresponds to a second image captured by the one or more image sensors at a second time after the first time. Based on the received additional image data and machine learning models, it is determined that the tracked object identified in the first image was not detected in the second image; Based on vehicle data and tracking data of the tracked object, it is determined that the tracked object should be present in the second image; as well as A remedial action is performed on the additional image data to identify the tracked object in the second image.

10. The system according to claim 9, wherein: The processing circuitry is configured to determine, based on the received additional image data and the machine learning model, that the tracked object identified in the first image is not detected in the second image: The image data is input into the machine learning model, wherein the machine learning model outputs a first output based on the input image data; The additional image data is input into the machine learning model, wherein the machine learning model outputs a second output based on the input additional image data; as well as Based on the second output of the machine learning model, it is determined that the second image does not include a depiction of the tracked object.

11. The system according to claim 10, wherein: The vehicle data includes the vehicle's speed; The tracking data includes a first distance between the tracked object and the vehicle at the first time; and The processing circuit is configured to determine, based on the vehicle data and the tracking data of the tracked object, that the tracked object should be present in the second image by: Based on the first output of the machine learning model, it is determined that the tracked object is located at the first distance from the vehicle at the first time. as well as Based on the first distance and the vehicle's speed, the second output should indicate the depiction of the tracked object in the second image.

12. The system according to claim 11, wherein: The first output includes a bounding box associated with the location of the tracked object in the first image; and The processing circuit is configured to perform the remedial action on the additional image data to identify the tracked object in the second image by: A region of interest associated with the predicted location of the tracked object in the second image is determined based on the bounding box, the first distance, and the speed of the vehicle.

13. The system of claim 12, wherein the processing circuitry is further configured to perform the remedial action on the additional image data to identify the tracked object in the second image by: Perform computer vision convolution within the region of interest to identify the tracked object in the second image; and The computer vision convolution determines that the tracked object is located at a second distance from the vehicle at the second time point.

14. The system according to claim 12, wherein: The machine learning model is associated with a threshold confidence level for identifying objects in image data input to the machine learning model; and The processing circuit is configured to perform the remedial action on the additional image data to identify the tracked object in the second image by: Reduce the threshold confidence level of the portion of the second image associated with the region of interest.

15. The system of claim 14, wherein the processing circuitry is configured to perform the remedial action on the additional image data to identify the tracked object in the second image by: The additional image data is fed back to the machine learning model associated with the reduced threshold confidence level of the portion of the second image associated with the region of interest; and The tracking object is determined to be at a second distance from the vehicle at the second time point based on a third output of the machine learning model received in response to feeding the additional image data back to the machine learning model.

16. The system according to claim 9, wherein: The vehicle in question is the first vehicle; The environment surrounding the first vehicle is defined by a predetermined distance from the one or more image sensors of the first vehicle; and The object being tracked is a second vehicle within the predetermined distance.

17. A non-transitory computer-readable medium having non-transitory computer-readable instructions encoded thereon, the non-transitory computer-readable instructions causing the processor, when executed by a processor associated with a vehicle, to: Image data associated with the environment around the vehicle is received via one or more image sensors associated with the vehicle, wherein the image data corresponds to a first image captured by the one or more image sensors at a first time. Additional image data associated with the environment around the vehicle is received via the one or more image sensors, wherein the additional image data corresponds to a second image captured by the one or more image sensors at a second time after the first time. Based on the received additional image data and machine learning models, it is determined that the tracked object identified in the first image was not detected in the second image; Based on vehicle data and tracking data of the tracked object, it is determined that the tracked object should be present in the second image; as well as A remedial action is performed on the additional image data to identify the tracked object in the second image.

18. The non-transitory computer-readable medium according to claim 17, wherein: The execution of the instructions to determine, based on the additional image data and the machine learning model, that the tracked object was not detected in the image also causes the processor to: The image data is input into the machine learning model, wherein the machine learning model outputs a first output based on the input image data; Additional image data is input into the machine learning model, wherein the machine learning model outputs a second output based on the input additional image data; as well as Based on the second output of the machine learning model, it is determined that the second image does not include a depiction of the tracked object.

19. The non-transitory computer-readable medium according to claim 18, wherein: The vehicle data includes the vehicle's speed; The tracking data includes a first distance between the tracked object and the vehicle at the first time; and Determining that the tracked object should be present in the image based on the vehicle data and the tracking data of the tracked object includes determining, based on the first distance and the speed of the vehicle, that the second output should indicate that the tracked object should be depicted in the second image.

20. The non-transitory computer-readable medium according to claim 19, wherein: The first output includes a bounding box associated with the location of the tracked object in the first image; and Executing the instructions to perform the remedial action on the additional image data to identify the tracked object in the second image also causes the processor to: A region of interest associated with the predicted location of the tracked object in the second image is determined based on the bounding box, the first distance, and the speed of the vehicle.

Images