Machine learning and data classification for operating device such as vehicle

Abstract

The present disclosure provides machine learning and data classification for operating devices such as vehicles. A computer includes a processor and a memory including instructions executable by the processor to input an image to a first neural network to generate a first detected object, and to input the image to a second neural network to generate a reconstructed image, the reconstructed image is input to a third neural network to generate a second detected object. The image may be divided into a plurality of portions, and the plurality of portions are input to respective fourth neural networks to generate a plurality of portions of a third detected object. The first detected object, the second detected object, the plurality of portions of the third detected object, and contextual data may be input to a partially observable Markov decision process to generate a high confidence detected object.

Description

Technical Field

[0001] The present disclosure relates to machine learning and data classification for operating a device such as a vehicle. Background Art

[0002] The computer can operate systems and / or devices including vehicle, robot, drone, and / or object tracking systems. Data including images can be acquired by sensors and processed using a computer to determine the position of the system with respect to objects in the environment surrounding the system. The computer can use the position data to determine a trajectory for moving the system in the environment. The computer can then determine control data to be transmitted to system components to control the system components to move the system according to the determined trajectory. Summary of the invention

[0003] Sensing systems (including vehicles, robots, drones, etc.) can operate by acquiring sensor data about the environment around the system and processing the sensor data to determine a path to operate the system or part of the system. The sensor data can be processed to determine the location of objects in the environment. Objects can include roads, buildings, conveyors, vehicles, pedestrians, manufactured parts, etc. A computer included in the system can use the detected objects to operate the system. For example, a robot can use data about the detected objects to determine the path for a gripper to pick up an object. A vehicle or drone can use data about the detected objects to determine a path for operating the vehicle or drone based on locating and identifying objects in the environment around the vehicle or drone.

[0004] This document uses a vehicle as a non-limiting example of a system that includes a sensing subsystem. A computing device included in the vehicle can acquire sensor data including video, lidar, radar, and ultrasonic data. The sensor data can be processed by a computer to locate and identify objects in the environment around the vehicle. For example, a vehicle can acquire video sensor data as an image and process the image to determine data about a traffic sign. Traffic signs can include data about the operation of the vehicle, such as stop signs, speed limit signs, no entry signs, and curve arrows. The vehicle can also acquire video sensor data about objects in the environment (such as pedestrians, bicycles, animals, and other vehicles).

[0005] Vehicle sensors such as satellite-based global positioning systems (GPS) and accelerometer-based inertial measurement units (IMUs) can provide vehicle attitude data that can be used to locate the vehicle relative to an aerial image that includes position data in global coordinates. The vehicle can be operated on a road based on the path by determining commands for directing the vehicle's powertrain, braking, and steering components to operate the vehicle so as to travel along the path. The vehicle path is typically a polynomial function on which the vehicle can operate. Sometimes referred to as a path polynomial, the polynomial function can specify the vehicle position (e.g., in terms of x, y, and z coordinates) and / or attitude (e.g., roll, pitch, and yaw) over time. That is, the path polynomial can be a polynomial function of three or less degrees that describes the movement of the vehicle on the ground. The movement of the vehicle on the road is described by a multidimensional state vector, which includes the vehicle position, orientation, velocity, and acceleration. Specifically, the vehicle motion vector may include position in x, y, z, yaw, pitch, roll, yaw rate, pitch rate, roll rate, heading velocity, and heading acceleration, which may be determined, for example, by fitting a polynomial function to the continuous 2D position relative to the ground included in the vehicle motion vector.

[0006] The polynomial function may be used to guide the vehicle from a current position indicated by the vehicle sensors to another position in the environment surrounding the vehicle while maintaining minimum and maximum limits on lateral and longitudinal acceleration. The vehicle may be operated along the vehicle path by transmitting commands to a vehicle controller for controlling vehicle propulsion, steering, and braking. A computing device in the vehicle may detect object locations and identities and use data about the detected objects to determine the vehicle path. For example, detecting a stop sign may be used to determine a vehicle path that causes the vehicle to stop before reaching the stop sign. In another example, detecting a speed limit sign may be used to determine a maximum velocity of the vehicle.

[0007] The technology described herein uses a two-step object detection system to enhance the ability of a computing device in a vehicle to accurately and reliably detect an object. Detecting an object in the context of the present application includes determining the location and identity of an object in the environment surrounding the vehicle. The machine learning system can rely on obtaining a clear and flawless image of the object to be detected. For example, the image of the object may be partially obscured by lighting, atmospheric phenomena such as fog or dust, pollutants such as mud, snow or ice, or other objects in the environment (such as foliage or vehicles). The first step of the object detection technology described herein uses multi-path object detection to overcome partially obscured objects. Multi-path object detection uses more than one data path to process a single data input. In this example, three data paths are used to process an input image using three different techniques before combining the results. In addition to the first step object detection results input to a partially observable Markov decision process with memory, the second step of the object detection technology described herein also uses context data to determine the most likely location and identity of the object.

[0008] A method is disclosed herein, comprising: inputting an image into a first neural network to generate a first detected object; inputting the image into a second neural network to generate a reconstructed image, which is input into a third neural network to generate a second detected object; and dividing the image into a plurality of parts and inputting the plurality of parts into a corresponding fourth neural network to generate a plurality of parts of a third detected object. The first detected object, the second detected object, the plurality of parts of the third detected object, and context data may be input into a partially observable Markov decision process to generate a high confidence detected object. The first detected object, the second detected object, and the plurality of parts of the third detected object may be combined to determine a first object label, and the first object label and the context data may be input into a partially observable Markov decision process. The first neural network, the third neural network, and the fourth neural network may be convolutional neural networks. The second neural network may be a generative adversarial network, which includes a decoder and a generator and is trained using a discriminator.

[0009] A partially observable Markov decision process may include states, actions, and transitions, wherein inputs of a first detected object, a second detected object, a plurality of portions of a third detected object, and context data result in one or more transitions from a first state to a second state. The partially observable Markov decision process may include a learning process, which includes previous states, actions, and transitions. The high-confidence detected object output by the partially observable Markov decision process may be based on previous decisions. The context data may include one or more of: sensor data, map data, and the speed and direction of objects in the environment surrounding the vehicle. The high-confidence detected object may be used to determine a vehicle path. The vehicle path may be used to operate the vehicle. The learning process inputs the high-confidence detected object, the first detected object, the second detected object, a plurality of portions of a third detected object, and context data, and may update the partially observable Markov decision process. The partially observable Markov process may include a reward based on previous actions. The partially observable Markov process may include a discount factor that determines the number of time periods over which the reward will be summed. The partially observable Markov process may include a conditional observation probability that includes a confidence value.

[0010] A computer-readable medium is also disclosed, which stores program instructions for performing some or all of the above method steps. A computer programmed to perform some or all of the above method steps is also disclosed, the computer comprising a computer device, the computer device being programmed to: input an image to a first neural network to generate a first detected object; input the image to a second neural network to generate a reconstructed image, the reconstructed image is input to a third neural network to generate a second detected object; and divide the image into a plurality of parts and input the plurality of parts to a corresponding fourth neural network to generate a plurality of parts of a third detected object. The first detected object, the second detected object, the plurality of parts of the third detected object, and the context data may be input to a partially observable Markov decision process to generate a high-confidence detected object. The first detected object, the second detected object, and the plurality of parts of the third detected object may be combined to determine a first object label, and the first object label and the context data may be input to a partially observable Markov decision process. The first neural network, the third neural network, and the fourth neural network may be convolutional neural networks. The second neural network may be a generative adversarial network, which includes a decoder and a generator and is trained using a discriminator.

[0011] The instructions may include further instructions, wherein the partially observable Markov decision process may include states, actions, and transitions, wherein the input of a first detected object, a second detected object, a plurality of portions of a third detected object, and context data results in one or more transitions from a first state to a second state. The partially observable Markov decision process may include a learning process, which includes previous states, actions, and transitions. The high confidence detected object output by the partially observable Markov decision process may be based on previous decisions. The context data may include one or more of: sensor data, map data, and the speed and direction of objects in the environment surrounding the vehicle. The high confidence detected object may be used to determine a vehicle path. The vehicle path may be used to operate the vehicle. The learning process inputs the high confidence detected object, the first detected object, the second detected object, a plurality of portions of a third detected object, and context data, and may update the partially observable Markov decision process. The partially observable Markov process may include a reward based on previous actions. The partially observable Markov process may include a discount factor that determines the number of time periods over which the reward will be summed. A partially observable Markov process may include conditional observation probabilities that include confidence values. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an example vehicle sensing system.

[0013] Figure 2 is a diagram of an example convolutional neural network.

[0014] Figure 3 is a diagram of an example multi-path object detection system.

[0015] Figure 4 is a diagram of an example active inference object detection system.

[0016] Figure 5 is a graph of example objects.

[0017] Figure 6 is a flow chart of an example active inference object detection system.

[0018] Figure 7 is a flow chart of an example process for operating a vehicle based on a two-stage object detection system. DETAILED DESCRIPTION

[0019] Figure 11 is a diagram of a vehicle computing system 100. The vehicle computing system 100 includes a vehicle 110, a computing device 115 included in the vehicle 110, and a server computer 120 remote from the vehicle 110. The computing device 115 of one or more vehicles 110 may receive data regarding the operation of the vehicle 110 from sensors 116. The computing device 115 may operate the vehicle 110 based on the data received from the sensors 116 and / or the data received from the remote server computer 120. The server computer 120 may communicate with the vehicle 110 via a network 130.

[0020] The computing device 115 includes a processor and memory such as are known. In addition, the memory includes one or more forms of computer-readable media and stores instructions that can be executed by the processor to perform various operations including as disclosed herein. For example, the computing device 115 may include programming to operate one or more of vehicle braking, propulsion (i.e., controlling the acceleration of the vehicle 110 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior lights and / or exterior lights, etc., and determine whether and when the computing device 115 (rather than a human operator) controls such operations.

[0021] The computing device 115 may include more than one computing device (i.e., controllers included in the vehicle 110 for monitoring and / or controlling various vehicle components, etc. (i.e., propulsion controller 112, brake controller 113, steering controller 114, etc.)), or be communicatively coupled to the more than one computing device via a vehicle communication bus as further described below. The computing device 115 is typically arranged for communication over a vehicle communication network (i.e., including a bus in the vehicle 110, such as a controller area network (CAN), etc.); the vehicle 110 network may additionally or alternatively include, for example, known wired or wireless communication mechanisms, such as Ethernet or other communication protocols.

[0022] The computing device 115 may transmit and / or receive messages to and from various devices in the vehicle (i.e., controllers, actuators, sensors (including sensor 116), etc.) via the vehicle network. Alternatively or additionally, where the computing device 115 actually includes multiple devices, the vehicle communication network may be used to communicate between the devices represented in this disclosure as the computing device 115. Furthermore, as mentioned below, various controllers or sensing elements (such as sensor 116) may provide data to the computing device 115 via the vehicle communication network.

[0023] Additionally, computing device 115 may be configured to communicate with remote server computer 120 (i.e., cloud server) via network 130 through vehicle-to-infrastructure (V2X) interface 111, as described below, which interface includes hardware, firmware, and software that permit computing device 115 to communicate with remote server computer 120 (i.e., cloud server) via network 130, such as wireless Internet. The V2X interface 111 may thus include a computer configured to utilize various wired and / or wireless networking technologies (i.e., cellular, The computing device 115 may be configured to communicate with other vehicles 110 through a V2X (vehicle to the outside world) interface 111 using a vehicle-to-vehicle (V2V) network formed between neighboring vehicles 110 on the basis of a mobile ad hoc network or through an infrastructure-based network (i.e., based on cellular communication (C-V2X) wireless communication cellular, dedicated short-range communication (DSRC) and / or similar communications). The computing device 115 also includes a non-volatile memory such as known. The computing device 115 can record data by storing the data in the non-volatile memory for later retrieval and transmission to the server computer 120 or the user mobile device 160 via the vehicle communication network and the vehicle to infrastructure (V2X) interface 111.

[0024] As already mentioned, programming for operating one or more vehicle 110 components (i.e., braking, steering, propulsion, etc.) is typically included in instructions stored in memory and executable by a processor of the computing device 115. Using data received in the computing device 115 (i.e., sensor data from sensors 116, server computer 120, etc.), the computing device 115 can make various determinations and / or control various vehicle 110 components and / or operations without the need for a driver to operate the vehicle 110. For example, the computing device 115 may include programming to adjust vehicle 110 operating behavior (i.e., the physical manifestation of vehicle 110 operation), such as speed, acceleration, deceleration, steering, etc., as well as strategic behavior (i.e., controlling operating behavior in a manner that is generally intended to achieve efficient traversal of a route), such as the distance between vehicles and / or the amount of time between vehicles, lane changes, minimum gaps between vehicles, left turn crossing path minimums, arrival times at specific locations, and the minimum time from arrival at an intersection to crossing the intersection (without a signal light).

[0025] A controller, as that term is used herein, includes a computing device that is typically programmed to monitor and / or control a particular vehicle subsystem. Examples include a propulsion controller 112, a brake controller 113, and a steering controller 114. The controller may be an electronic control unit (ECU), such as is known, and may include additional programming as described herein. The controller may be communicatively connected to a computing device 115 and receive instructions from the computing device to actuate a subsystem in accordance with the instructions. For example, a brake controller 113 may receive instructions from a computing device 115 to operate the brakes of the vehicle 110.

[0026] One or more controllers 112, 113, 114 for the vehicle 110 may include known electronic control units (ECUs) and the like, including, as non-limiting examples, one or more propulsion controllers 112, one or more brake controllers 113, and one or more steering controllers 114. Each of the controllers 112, 113, 114 may include a corresponding processor and memory and one or more actuators. The controllers 112, 113, 114 may be programmed and connected to a vehicle 110 communication bus, such as a controller area network (CAN) bus or a local interconnect network (LIN) bus, to receive instructions from a computing device 115 and control actuators based on the instructions.

[0027] Sensors 116 may include a variety of devices known to provide data via a vehicle communication bus. For example, a radar fixed to a front bumper (not shown) of vehicle 110 may provide a distance from vehicle 110 to the next vehicle in front of vehicle 110, or a global positioning system (GPS) sensor disposed in vehicle 110 may provide geographic coordinates of vehicle 110. For example, the distance provided by radar and / or other sensors 116 and / or the geographic coordinates provided by a GPS sensor may be used by computing device 115 to operate vehicle 110 autonomously or semi-autonomously.

[0028] The vehicle 110 is typically a ground-based vehicle 110 (i.e., a passenger car, a light truck, etc.) capable of autonomous and / or semi-autonomous operation and having three or more wheels. The vehicle 110 includes one or more sensors 116, a V2X interface 111, a computing device 115, and one or more controllers 112, 113, 114. The sensor 116 may collect data related to the vehicle 110 and the operating environment of the vehicle 110. By way of example and not limitation, the sensor 116 may include, i.e., an altimeter, a camera, a lidar, a radar, an ultrasonic sensor, an infrared sensor, a pressure sensor, an accelerometer, a gyroscope, a temperature sensor, a pressure sensor, a Hall sensor, an optical sensor, a voltage sensor, a current sensor, a mechanical sensor (such as a switch), etc. The sensor 116 may be used to sense the operating environment of the vehicle 110, i.e., the sensor 116 may detect phenomena such as weather conditions (rainfall, outside temperature, etc.), road slope, road position (i.e., using road edges, lane markings, etc.), or the position of a target object (such as a neighboring vehicle 110). The sensors 116 may also be used to collect data, including dynamic vehicle 110 data related to the operation of the vehicle 110 , such as speed, yaw rate, steering angle, engine speed, brake pressure, oil pressure, power levels applied to controllers 112 , 113 , 114 in the vehicle 110 , connectivity between components, and accurate and timely performance of components of the vehicle 110 .

[0029] The server computer 120 generally has features in common with the V2X interface 111 and computing device 115 of the vehicle 110, such as a computer processor and memory and configuration for communicating via the network 130, and therefore these features will not be described further. The server computer 120 can be used to develop and train software that can be transferred to the computing device 115 in the vehicle 110.

[0030] Figure 2 2 is a diagram of an example convolutional neural network 200. The convolutional neural network 200 can input an image 202 and output a prediction 220. The convolutional neural network 200 includes convolutional layers 204, 206, 208, 210 (collectively referred to as convolutional layers 212) and fully connected layers 214, 216 (collectively referred to as fully connected layers 218). The convolutional layers receive image data as input and convolve the image data using a convolution kernel, which is typically a k×k neighborhood, where k is a small number such as 3, 5, 7, 9, etc. The operation performed by the convolution kernel is determined by the numbers included in the k×k neighborhood (called weights). The fully connected layers 218 calculate linear or nonlinear algebraic functions based on their inputs. They are called fully connected because any input value can be combined with any other input value. The linear or nonlinear algebraic function determined by the fully connected layer 218 is determined by parameters also called weights.

[0031] The convolutional neural network 200 can be trained by compiling a training data set including an image 202 and ground truth data indicating a user-selected prediction 220 to be output from the convolutional neural network 200 in response to the input image 202. In this example, the prediction 220 includes object detection data, such as the object location in global or pixel coordinates and an object label that identifies the object. The object label may include, for example, "vehicle", "bicycle", "traffic sign", "pedestrian", and "animal". The output from the neural network is referred to as a prediction in this article. The ground truth is determined by a process separate from the neural network and may include human inspection and measurement of the image data and the imaged scene. Training the convolutional neural network 200 may include processing each image 202 in the training data set hundreds or thousands of times, each time comparing the output prediction 220 to the ground truth to determine a loss function. The loss function is back-propagated from back to front through the fully connected layer 218 and the convolutional layer 212, thereby changing the weights included in the fully connected layer 218 and the convolutional layer 212 to minimize the loss function. When the loss function is sufficiently minimized, e.g., when changing the weights does not make the loss function smaller, the convolutional neural network 200 can be considered trained, and the current weights are saved.

[0032] The output from each convolutional layer 212 and each fully connected layer 218 to the next layer in the convolutional neural network 200 is called a tensor. The tensor is output from the layer via an activation function that can adjust the output. For example, a ReLu activation adjusts the output to be positive. The output from the convolutional layer 212 or the fully connected layer 218 via the activation function is referred to herein as an activation tensor. The activation tensor output by the layers 212, 218 of the trained convolutional neural network in response to a particular input image 202 can be used to characterize the convolutional neural network 200, and will be used herein to determine, for example, the similarity between two or more convolutional neural networks.

[0033] Figure 3 306. The multipath object detection system 300 may be a software program resident in memory and executed by the computing device 115. The multipath object detection system increases the likelihood that an object will be correctly detected by inputting an image 302 including an object into three different data paths that process the input image 302 using three different techniques. The first data path 304 receives the image 302 as an input to a first neural network 306. The first neural network 306 may be as described above with respect to Figure 2The convolutional neural network 200 is trained to detect objects in the image 302. The first neural network 306 outputs a first prediction 308, which includes the location of the object in real-world coordinates or pixel coordinates and an object label that identifies the object, as discussed above. In the example where the label is "traffic sign", the prediction may include data about the type of traffic sign, such as "stop" or "speed limit 70MPH". The first prediction 308 may also include a confidence value determined by the first neural network 306, which estimates the probability that the label is correct.

[0034] As described above, a problem with detecting an object in an image 302 is that the image 302 may be acquired under less than ideal environmental conditions. For example, the image 302 may be acquired under dim or cloudy lighting or in the presence of image "noise" such as rain, snow, fog, or dust. The second data path 310 included in the multi-path object detection system 300 may include a generative adversarial network (GAN) 312. The GAN 312 is a neural network including a decoder that receives the image 302 as input and decodes the image 302 into latent variables. The latent variables are input to a generator that reconstructs the image based on the latent variables while adding or subtracting image noise and simulating different lighting conditions. The GAN is trained by outputting the processed results to a discriminator, which is a neural network trained to determine a real image from a reconstructed image. The GAN 312 is considered trained when it can output a reconstructed image with simulated noise that fools the discriminator into classifying the image as real. At inference time, the GAN 312 receives the image 302 as input and adds noise to the image and outputs a reconstructed image. The second neural network 314 has been trained to detect objects in noisy images. The second neural network 314 receives the reconstructed image from the GAN 312 as input and outputs a second prediction 316.

[0035] The third data path 318 includes an image preprocessor 320 that divides the image 302 into a plurality of portions. The third data path 318 may process images 302 that are corrupted by a different type of noise than the second data path 310. The third data path 318 processes image 302 data that has been corrupted by localized noise. For example, a traffic sign may have mud or snow that obscures a portion of the sign. In other examples, a traffic sign may be tampered with to obscure portions of the sign. For example, Figure 5A traffic sign 500 is shown segmented into four portions 502, 504, 506, 508. One of the portions 506 is partially obscured by a snowball that has been thrown at the sign. The working principle of the third data path 318 is that in an example where a portion of the object is obscured by noise, dividing the image 302 into multiple segments can permit the fourth neural network 322 to process the image segments individually. Processing the image segments individually can advantageously limit blurry and / or low confidence results to the obscured image segments while permitting the unobscured segments to return high confidence results.

[0036] In examples where image segments are processed separately by fourth neural network 322, unoccluded portions of the object will be processed with high confidence. The image segments are processed by fourth neural network 322 to determine a plurality of object predictions 324 including an object prediction for each segment, where each object prediction includes a confidence value. In examples where a portion of image 302 is occluded, object predictions 324 including high confidence predictions may be combined to infer predictions for portions of image 302 including low confidence predictions. Figure 5 In the example of FIG. 5 , despite the low confidence prediction for segment 506 , the high confidence predictions for segments 502 , 504 , 508 may be combined to infer that the image of traffic sign 500 includes a speed limit sign.

[0037] The predictions 308, 316, 324 from the three data paths 304, 310, 318 may be combined at a combiner 326. The combiner 326 may include instructions for comparing the predictions 308, 316, 324 to determine whether the predictions 308, 316, 324 are consistent. In this context, the predictions 308, 316, 324 are consistent when they include the same object label at the same location. In an example where the atmospheric conditions are clear, the lighting is good, and the object is not obscured, the three data paths 304, 310, 318 may all derive the same predictions 308, 316, 324 for the object location and object label. In an example where there are one or more of poor atmospheric conditions, dim lighting, or object obscuration, one or more predictions 308, 316, 324 may have low or uncertain confidence values, while one or more predictions 308, 316, 324 have high confidence values. In an example, for example, a low confidence value may be less than 50% probability that the prediction 308, 316, 324 is correct, and a high confidence value may be greater than 90% probability that the prediction 308, 316, 324 is correct. In these examples, confidence values ​​between 50% and 90% may be uncertain.

[0038] In examples where the predictions 308, 316, 324 all agree and the mutually agreed predictions 308, 316, 324 all have high confidence values, the combiner 326 can select the high confidence prediction as a result, and the high confidence mutually agreed predictions 308, 316, 324 are output as predictions 328 and passed to the computing device 115 included in the vehicle 110 for use in operating the vehicle. The predictions 328 can also be passed to the active inference object detection system 400 to help train the active inference object detection system 400. In examples where the predictions 308, 316, 324 do not agree and / or one or more of the predictions 308, 316, 324 have low or uncertain confidence values, the predictions 308, 316, 324 are output as predictions 328 and passed to the computing device 115 as described below with respect to Figure 4 The described active inference object detection system 400 is further processed.

[0039] Figure 4 4 is a diagram of an active inference object detection system 400. The active inference object detection system 400 may be a software program resident in a memory and executed by a computing device 115. The active inference object detection system 400 receives as input the prediction 328 output by the multipath object detection system 300 and also receives as input context data 402. The prediction 328 includes the predictions 308, 316, 324 and the confidence values ​​output by the data paths 304, 310, 318. The context data 402 is data about the environment surrounding the vehicle 110 determined by the computing device 115 based on data acquired by the sensor 116 and data transmitted to the computing device 115 from the remote server computer 120 via the network 130. The data acquired by the sensor 116 may include GPS data about the real-world location of the vehicle 110. For example, the data acquired by the sensor 116 may include data about objects included in the environment, such as other vehicles. For example, the data about the other vehicles may be processed using a software program such as a neural network to determine the location, velocity, and direction of the other vehicles. The context data 402 received from the remote server computer 120 may include map data downloaded from a remote device via the Internet, for example. The map data may include data about roads, including, for example, traffic signs, traffic signals, intersections, and crosswalks.

[0040] The prediction 328 and the context data 402 are input to a partially observable Markov decision process (POMDP) ​​404 and a learning processor 406. The memory includes a previous refined prediction 408 output by the POMDP 404, the previous prediction 328 including a confidence value, and the context data 402. The POMDP 404 receives as input the prediction 328 including the confidence value, the context data 402, and the previous data from the learning processor 406, and outputs a refined prediction 408 based on the input prediction 328. The active inference object detection system 400 can enhance the ability to detect objects by combining the prediction 328 with the context data 402 and the previous refined prediction 408. The refined prediction 408 can include an object location and an object label as well as confidence values ​​for the object location and the object label.

[0041] The POMDP 404 is defined by a 7-tuple (S, A, T, R, Ω, O, γ), where S is a set of states, A is a set of actions, T is a set of conditional transition probabilities between states, R is a reward function that maps a set of states S and actions A to a real number R, and Ω is a set of observations. A state S is an element that indicates a computation performed by the POMDP 404. Each state s∈S may include variables and instructions that determine the transformations to be applied to the variables, the actions a∈A to be output as a refined prediction 408, and the next state s∈S upon receiving an input observation ω∈Ω. new t∈T. In some examples, the new state can be the same as the previous state. At any time period, one and only one state s is active. In this example, the variables included in the state S can be object positions and labels, confidence values, and the amount of available context data 402. In this example, the observation set Ω includes the predictions 328 and the context data 402. In this example, the observation set Ω also includes hidden observations generated internally by the POMDP 404. O is a set of conditional observation probabilities, which in this example can be confidence values ​​included in the predictions 328. γ is a discount factor that can determine the number of time periods over which the reward function will be summed.

[0042] At each time period, POMDP 404 is in state s∈S. At the current time period t0, POMDP 404 may receive observation ω∈Ω and current conditional observation probability o∈O via input prediction 328. Inputting observation ω conditioned on observation probability o causes POMDP 404 to transition from the current state s to a new state s new , and outputs action a to computing device 115 in the form of refined prediction 408. Refined prediction 408 may be a new object position and label including a new confidence value determined by POMDP 404 based on context data 402. POMDP 404 may be used to generate a new object position and label in time period t n Receiving a reward r based on computing device 115 evaluating the success of the previous action an ∈R. Reward r is a numerical value between 0 and 1 determined by computing device 115 that indicates how useful refined prediction 408 was in determining the vehicle path of vehicle 110. The discount factor γ determines how many time periods the rewards r are summed over, where a value of γ=0 means that only the current reward r is considered, and γ=1 means that all future rewards are maximized. Intermediate values ​​of γ will include rewards for intermediate time periods.

[0043] The POMDP 404 may output a refined prediction 408 including high confidence object locations and labels even though the input prediction 328 may be missing, incorrect, or low confidence. For example, the prediction 328 input to the POMDP 404 may include three predictions 308, 316, 324 with three different confidence values. The POMDP 404 may select between the different predictions 308, 316, 324 and determine the confidence value to output as the refined prediction 328 based on the previous prediction 328 and the context data 402 inputs included in the updated POMDP 404 including the current context data 402 and the input from the learning processor 406.

[0044] For example, you can Figure 5 Traffic sign 500 of FIG. 1 is input to multi-path object detection system 300. In an example, predictions 308, 316 may include low confidence object labels. Prediction 324 may include predictions for four segments, three of which have high confidence labels, but the segment for which the speed limit value can be determined has low confidence. Context data 402 included in the input to POMDP 404 may indicate a road on which vehicle 110 is currently traveling with a 70 MPH speed limit. In this example, POMDP 404 may be in state s indicating that a speed limit sign has been detected but the rate is uncertain. Context data 402 including data about the current road speed limit input based on map data and GPS data may indicate that an action a is output including a refined prediction 408, the refined prediction including an object label equal to "70 MPH speed limit sign" with high confidence.

[0045] The learning processor 406 updates the POMDP 404 using the previous predictions 328 and the context data 402 input to the POMDP 404. The learning processor 406 receives as input the current refined prediction 408, the input prediction 328, and the context data 402. The learning processor 406 updates the POMDP 404 based on the current refined prediction 408, the input prediction 426, and the context data 402, as well as the previous refined predictions 408, the previous predictions 328, and the previous context data 402 stored in a memory included in the learning processor 406. In examples where the input is an incomplete or low confidence prediction 328, the updated POMDP 404 can make a correct refined prediction 408 based on the previously determined refined prediction 408 and the previously input prediction 328 and the context data 402, which can result in a positive reward function r.

[0046] Another example of contextual data 402 for assisting POMDP 404 may include an example in which the input prediction 328 indicates an object in the field of view of sensor 116 that is detected with low confidence. For example, contextual data 402 may indicate that the object is located in a crosswalk that may be occupied by pedestrians. Additional contextual data 402 may indicate that other vehicles in the environment are stopped at the crosswalk. POMDP 404 may output a refined prediction 408 that assumes that the detected object has a high probability of being a pedestrian based on being located in the crosswalk and other vehicles stopped. In addition, refined prediction 408 is input to learning processor 406 along with input prediction 328 and contextual data 402 to update POMDP 404 and enhance object detection based on incomplete or missing data.

[0047] Figure 5 3 is a diagram of an image of a traffic sign 500. The traffic sign 500 is a speed limit sign and is segmented into four segments 502, 504, 506, 508 by the image preprocessor 320. The segment 506, which includes the number "7" indicating that the speed limit is 70 miles per hour (MPH) along with the number "0", is obscured by snow 510 in this example. The image of the traffic sign 500 input to the first data path 304 and the second data path 310 of the multi-path object detection system 300 can output inconclusive or low confidence predictions 308, 316 about the label to be attached to the traffic sign 500. The third data path 318 can output a high confidence prediction about the segments 502, 504, 506 and a low confidence prediction about the segment 506. Outputting the prediction 328 to the active inference object detection system 400 along with the context data 402 can enhance the prediction included in the refined prediction 408. For example, map data and vehicle location data included in context data 402 may indicate that vehicle 110 is currently traveling in a 70 MPH speed limit zone, making traffic sign 500 likely to be a 70 MPH speed limit sign.

[0048] Figure 6 1 is a flow chart of a process 600 for determining a refined prediction 408 including an object location and label indicated with high confidence based on vehicle sensor 116 data. The process 600 may be implemented in a computing device 115 included in a vehicle 110. The process 600 includes a number of blocks that may be performed in the order shown. Alternatively or in addition, the process 600 may include fewer blocks, or may include blocks that are performed in a different order.

[0049] Process 600 begins at block 602, where computing device 115 in vehicle 110 acquires image 302 from sensor 116, which may be a camera included in vehicle 110. Image 302 includes data about the environment surrounding vehicle 110. Image 302 may include objects that, when detected, may assist computing device 115 in directing vehicle 110.

[0050] At box 604, the computing device 115 receives the image 302 as input to the multi-path object detection system 300. The multi-path object detection system 300 includes three data paths 304, 310, 318. The first data path 304 processes the image 302 using the first neural network 306 to determine a first prediction 308 regarding the location and identity (label) of the object. The second data path 310 receives the image 302 as input to the GAN 312, which adds image noise to the image to simulate atmospheric and lighting conditions, and then processes the image 302 with the second neural network 314 to determine a second prediction 316. The third data path 318 includes a segmenter that segments the image into a plurality of segments. The plurality of segments are processed by one or more third neural networks 322 that output a plurality of predictions 324. The prediction 328 combines the first prediction, the second prediction, and the third prediction and the confidence value, as described above with respect to Figure 3 As described.

[0051] At block 606 , the prediction 328 and confidence value are output to the active inference object detection system 400 . In examples where the predictions agree and have a high confidence, the inference object detection system 400 may be skipped and the object location and label may be output directly to the computing device 115 .

[0052] At block 608, the output predictions 328, confidence values, and context data 402 from the computing device 115 are input to the POMDP 404 in the active inference object detection system 400. The output predictions 328, confidence values, and context data 402 are also input to the learning processor 406 portion of the active inference object detection system 400.

[0053] At block 610, the POMDP 404 processes the output predictions 328, the confidence values, the context data 402, and the data from the learning processor 406. The POMDP 404 receives as input the output predictions 328, the confidence values, the context data 402, and the data from the learning processor 406, and determines a refined prediction 408. The refined prediction 408 may include an updated prediction and confidence value for an object.

[0054] At block 612, the active inference object detection system 400 outputs the refined prediction 408 including the object prediction and the confidence value to the computing device 115. The active inference object detection system 400 also stores the refined prediction 408 in the learning processor 406 to assist in making future refined predictions 408. After block 612, the process 600 ends.

[0055] Figure 7 1 is a flow chart of a process 700 for operating a vehicle 110 based on objects detected by a multi-path object detection system 300 and an active inference object detection system 400. The process 700 may be implemented by a computing device 115 included in the vehicle 110. The process 700 includes a plurality of blocks that may be performed in the order shown. Alternatively or additionally, the process 700 may include fewer blocks, or may include blocks that are performed in a different order.

[0056] Process 700 begins at block 702, where computing device 115 in vehicle 110 acquires image 302 from sensor 116, which may be a camera included in vehicle 110. Image 302 includes data about the environment surrounding vehicle 110. Image 302 may include objects that, when detected, may assist computing device 115 in directing vehicle 110.

[0057] At block 704, computing device 115 performs the following steps by inputting image 302 into the Figure 3 The depicted multi-path object detection system 300 determines a prediction 326. The prediction may include a location, a label, and a confidence value for an object included in the image 302.

[0058] At block 706, computing device 115 performs the following steps by inputting prediction 326 and context data 402 into the Figure 4 The described active inference object detection system 400 is used to determine a refined prediction 408 .

[0059] At box 708, the computing device 115 operates the vehicle 110 based on the refined prediction 408 output by the active reasoning object detection system 400. For example, when the refined prediction 408 includes an object with a label "pedestrian", if the location of the object is on the road, the vehicle path may include stopping. In other examples, when the refined prediction 408 includes an object with a label "animal", the vehicle path may include turning or slowing the vehicle. In examples where the refined prediction 408 includes a traffic sign, the vehicle path (or trajectory, for example, including rate or speed) may be determined based on traffic regulations or information displayed on the traffic sign. Similarly, the vehicle path may be determined based on the prediction 408 about road lane markings, etc. In general, the vehicle 110 may be operated by determining a path polynomial function to determine the vehicle path, which maintains minimum and maximum limits on lateral and longitudinal acceleration. The vehicle 110 may be operated along the vehicle path by transmitting commands to the controllers 112, 113, 114 to control vehicle propulsion, steering, and braking. After box 708, the process 700 ends.

[0060] Computing devices such as those described herein typically each include commands that can be executed by one or more computing devices such as those identified above and used to implement the blocks or steps of the processes described above. For example, the process blocks described above may be embodied as computer executable commands.

[0061] The computer executable instructions may be compiled or interpreted by a computer program created using a variety of programming languages ​​and / or technologies, including but not limited to the following in single or combined form: Java TM , C, C++, Python, Julia, SCALA, Visual Basic, Java Script, Perl, HTML, etc. Typically, a processor (i.e., a microprocessor) receives commands from a memory, a computer-readable medium, etc., and executes these commands to perform one or more processes including one or more of the processes described herein. Such commands and other data may be stored in files and transmitted using a variety of computer-readable media. Files in a computing device are typically collections of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0062] Computer-readable media (also referred to as processor-readable media) include any non-transitory (i.e., tangible) media that participate in providing data (i.e., instructions) that can be read by a computer (i.e., by a processor of a computer). Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, including internals that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0063] Unless otherwise expressly indicated herein, all terms used in the claims are intended to be given their ordinary and customary meanings as understood by those skilled in the art. Specifically, unless a claim recites an express limitation to the contrary, the use of singular articles such as "a," "an," "the," and "said" should be interpreted as reciting one or more of the indicated elements.

[0064] The term "exemplary" is used herein in the sense of referring to an example, ie, reference to "exemplary widget" should be interpreted as referring merely to an example of a widget.

[0065] The adverb “approximately” modifying a value or result means that the shape, structure, measurement, value, determination, calculation, etc. may vary from the exactly described geometry, distance, measurement, value, determination, calculation, etc. due to imperfections in materials, machining, manufacturing, sensor measurement, calculation, processing time, communication time, etc.

[0066] In the accompanying drawings, the same candidate marks indicate the same elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, etc. described herein, it should be understood that although the steps or frames of such processes, etc. have been described as occurring in a sequence according to a specific order, such processes may be practiced by the described steps performed in an order other than the order described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the process herein is provided for the purpose of illustrating certain embodiments and should never be interpreted as limiting the claimed invention.

[0067] According to the present invention, a system is provided, which has: a computer, the computer including a processor and a memory, the memory including instructions, which can be executed by the processor to: input an image to a first neural network to generate a first detected object; input the image to a second neural network to generate a reconstructed image, and the reconstructed image is input to a third neural network to generate a second detected object; divide the image into multiple parts and input the multiple parts to corresponding fourth neural networks to generate multiple parts of a third detected object; and input the first detected object, the second detected object, the multiple parts of the third detected object, and scene data into a partially observable Markov decision process to generate a high-confidence detected object.

[0068] According to an embodiment, the first detected object, the second detected object, and portions of the third detected object are combined to determine a first object label, and the first object label and context data are input to a partially observable Markov decision process.

[0069] According to an embodiment, the first neural network, the third neural network and the fourth neural network are convolutional neural networks.

[0070] According to an embodiment, the second neural network is a generative adversarial network that includes a decoder and a generator and is trained using a discriminator.

[0071] According to an embodiment, a partially observable Markov decision process includes states, actions, and transitions, wherein inputs of a first detected object, a second detected object, multiple parts of a third detected object, and context data result in one or more transitions from a first state to a second state.

[0072] According to an embodiment, the partially observable Markov decision process includes a learning process that includes previous states, actions, and transitions.

[0073] According to an embodiment, the high confidence detected objects output by the partially observable Markov decision process are based on previous decisions.

[0074] According to an embodiment, the contextual data includes one or more of: sensor data, map data, and the speed and direction of objects in the environment surrounding the system.

[0075] According to an embodiment, high confidence detected objects are used to determine the vehicle path.

[0076] According to an embodiment, the vehicle path is used to operate the vehicle.

[0077] According to the present invention, a method includes: inputting an image into a first neural network to generate a first detected object; inputting the image into a second neural network to generate a reconstructed image, which is input into a third neural network to generate a second detected object; dividing the image into multiple parts and inputting the multiple parts into corresponding fourth neural networks to generate multiple parts of a third detected object; and inputting the first detected object, the second detected object, the multiple parts of the third detected object, and scene data into a partially observable Markov decision process to generate a high-confidence detected object.

[0078] In one aspect of the invention, a first detected object, a second detected object, and portions of a third detected object are combined to determine a first object label, and the first object label and context data are input to a partially observable Markov decision process.

[0079] In one aspect of the invention, the first neural network, the third neural network and the fourth neural network are convolutional neural networks.

[0080] In one aspect of the invention, the second neural network is a generative adversarial network that includes a decoder and a generator and is trained using a discriminator.

[0081] In one aspect of the invention, a partially observable Markov decision process comprises states, actions, and transitions, wherein inputs of a first detected object, a second detected object, multiple parts of a third detected object, and context data result in one or more transitions from a first state to a second state.

[0082] In one aspect of the invention, a partially observable Markov decision process includes a learning process that includes previous states, actions, and transitions.

[0083] In one aspect of the invention, high confidence detected objects output by a partially observable Markov decision process are based on previous decisions.

[0084] In one aspect of the invention, the context data includes one or more of: sensor data, map data, and the velocity and direction of objects in the environment surrounding the vehicle.

[0085] In one aspect of the invention, high confidence detected objects are used to determine a vehicle path.

[0086] In one aspect of the invention, a vehicle path is used to operate a vehicle.

Claims

1. A method comprising: inputting the image into a first neural network to generate a first detected object; inputting the image into a second neural network to generate a reconstructed image, the reconstructed image being input into a third neural network to generate a second detected object; dividing the image into a plurality of portions and inputting the plurality of portions into corresponding fourth neural networks to generate a plurality of portions of a third detected object; as well as The first detected object, the second detected object, the portions of the third detected object, and context data are input to a partially observable Markov decision process to generate high confidence detected objects.

2. The method of claim 1 , wherein the multiple parts of the first detected object, the second detected object, and the third detected object are combined to determine a first object label, and the first object label and context data are input into the partially observable Markov decision process.

3. The method of claim 1, wherein the first neural network, the third neural network, and the fourth neural network are convolutional neural networks.

4. The method of claim 1, wherein the second neural network is a generative adversarial network that includes a decoder and a generator and is trained using a discriminator.

5. The method of claim 1 , wherein the partially observable Markov decision process comprises states, actions, and transitions, wherein inputting the first detected object, the second detected object, the multiple parts of the third detected object, and the context data results in one or more transitions from a first state to a second state.

6. The method of claim 5, wherein the partially observable Markov decision process comprises a learning process, the learning process comprising previous states, actions, and transitions.

7. The method of claim 6, wherein the high confidence detected object output by the partially observable Markov decision process is based on previous decisions.

8. The method of claim 1, wherein the context data comprises one or more of: sensor data, map data, and velocity and direction of objects in the environment surrounding the vehicle.

9. The method of claim 1, wherein the high confidence detected objects are used to determine a vehicle path.

10. The method of claim 9, wherein the vehicle path is used to operate a vehicle.

11. The method of claim 5, wherein the learning process inputs the high-confidence detected object, the first detected object, the second detected object, the multiple parts of the third detected object, and context data, and updates the partially observable Markov decision process.

12. The method of claim 1, wherein the partially observable Markov process includes rewards based on previous actions.

13. The method of claim 12, wherein the partially observable Markov process includes a discount factor that determines the number of time periods over which the rewards are to be summed.

14. The method of claim 13, wherein the partially observable Markov process comprises a conditional observation probability, the conditional observation probability comprising a confidence value.

15. A system comprising a computer programmed to perform the method of any one of claims 1 to 14.

Images