Anomaly classification in imaging sensor data

Abstract

The present disclosure provides for "Anomaly classification in imaging sensor data." A computer includes a processor and a memory, and the memory stores instructions executable by the processor to: generate an anomaly map by executing a first machine learning procedure that takes as input sensor data from an imaging sensor; and classify anomalies in the sensor data as particular types by executing a second machine learning procedure. The anomaly map includes a set of likelihood values at respective pixel locations. Each likelihood value represents a probability that any type of anomaly is present at the respective pixel location. The second machine learning procedure takes as input the anomaly map.

Description

Technical Field

[0001] This disclosure describes techniques for classifying anomalies in sensor data from imaging sensors. Background Technology

[0002] Vehicles typically include a variety of sensors. Sensors can provide data about vehicle operation, such as wheel speed, wheel orientation, and engine and transmission data (e.g., temperature, fuel consumption, etc.). Sensors can detect the vehicle's position and / or orientation. Sensors can include Global Positioning System (GPS) sensors; accelerometers, such as piezoelectric or microelectromechanical systems (MEMS); gyroscopes, such as rate gyroscopes, ring laser gyroscopes, or fiber optic gyroscopes; inertial measurement units (IMUs); and magnetometers. Sensors can detect the external world, including objects and / or characteristics of the vehicle's surrounding environment, such as other vehicles, road lane markings, traffic lights and / or signs, road users, etc. Sensors can include radar sensors, ultrasonic sensors, scanning laser rangefinders, light detection and ranging (LiDAR) devices, and image processing sensors (such as cameras). Summary of the Invention

[0003] For example, the imaging sensor could be a camera mounted on the vehicle, the sensor data could be an image, and the anomaly could be various types of obstructions, such as rain, fog, or debris on the camera lens. Classifying the anomaly can help mitigate it. For example, vehicle components can be actuated based on this classification, such as activating the windshield wipers in response to rain, or activating the cleaning nozzles in response to debris on the camera lens.

[0004] A computer is programmed to generate anomaly maps by executing a first machine learning program and to classify anomalies in sensor data into specific types by executing a second machine learning program. The first machine learning program takes sensor data from an imaging sensor as input, and the second machine learning program takes the anomaly map as input. The anomaly map comprises a set of likelihood values ​​for corresponding pixel locations. Each likelihood value represents the probability that any type of anomaly exists at the corresponding pixel location. In other words, the anomaly map does not indicate what type of anomaly exists; instead, the probability is for a binary result, that is, whether any type of anomaly exists. Because the probability is associated with pixel location, the anomaly map also indicates where the anomaly is located within the field of view of the imaging sensor. Once the first machine learning program generates the anomaly map, the second machine learning program uses the anomaly map to classify anomalies in the sensor data into specific types.

[0005] The use of this style of anomaly map offers several advantages. This disclosure is computationally more efficient compared to multiple networks, each recognizing a single type of anomaly. Compared to both single-level classifiers and multiple networks, this disclosure uses less training data, thus reducing both the computational workload during training and the labeling workload. This disclosure contributes to the robustness of the output by capturing common elements of multiple types of occlusions in a first machine learning procedure. Since the detection and classification levels can be fine-tuned independently, it is easier to train a two-level classifier to high accuracy.

[0006] A computer includes a processor and a memory, the memory storing instructions executable by the processor to: generate an anomaly map by executing a first machine learning program, the first machine learning program taking sensor data from an imaging sensor as input; and classify anomalies in the sensor data into specific types by executing a second machine learning program. The anomaly map includes a set of likelihood values ​​at corresponding pixel locations. Each likelihood value represents the probability that any type of anomaly exists at the corresponding pixel location. The second machine learning program takes the anomaly map as input.

[0007] In the example, the imaging sensor may be mounted on the vehicle, and the instructions may further include instructions for actuating vehicle components based on a specific type of anomaly.

[0008] In the example, the imaging sensor may be mounted on the vehicle, and the instructions may further include instructions for actuating vehicle components based on the location of the anomaly indicated by the anomaly map.

[0009] In the example, the first machine learning program can be trained on a training sensor dataset with binary labels indicating whether an anomaly is present or not. In a further example, each binary label can be applied holistically to the corresponding training sensor data recorded at a specific time step.

[0010] In another further example, the training sensor dataset may be a first set of first training sensor data, and the first machine learning program may be trained on a second set of second training sensor data with labels indicating the locations of anomalies. In an even further example, the first set may be the majority of the training data used to train the first machine learning program.

[0011] In the example, the instructions for classifying anomalies may include instructions for selecting a specific type from a predetermined set of potential specific types. In a further example, each likelihood value in the anomaly graph may represent the probability that an anomaly of at least one potential specific type exists at the corresponding pixel location.

[0012] In another further example, a potential specific type can be the type of object that occludes.

[0013] In another further example, the output of the second machine learning program could be an anomaly type map, which includes a set of likelihood vectors at corresponding pixel locations. Each likelihood vector includes multiple entries, each representing the probability that an anomaly of a corresponding potential specific type exists at the corresponding pixel location.

[0014] In the example, the instructions may further include instructions for determining anomaly scores in the anomaly graph and, in response to an anomaly score exceeding a threshold, classifying the anomaly into a specific type by executing a second machine learning procedure.

[0015] In the example, the first machine learning program may include a fully convolutional data description.

[0016] In the example, the first machine learning program may include at least one upsampling layer.

[0017] In the example, the imaging sensor can be a camera, and the sensor data can be an image.

[0018] One method includes: generating an anomaly map by executing a first machine learning procedure, the first machine learning procedure taking sensor data from an imaging sensor as input; and classifying anomalies in the sensor data into specific types by executing a second machine learning procedure. The anomaly map includes a set of likelihood values ​​at corresponding pixel locations. Each likelihood value represents the probability that any type of anomaly exists at the corresponding pixel location. The second machine learning procedure takes the anomaly map as input.

[0019] In the example, the imaging sensor may be mounted on the vehicle, and the method may further include actuating a component of the vehicle based on at least one of a specific type of anomaly or the location of anomaly indicated by an anomaly map.

[0020] In the example, the first machine learning program can be trained on a training sensor dataset with binary labels that indicate whether an anomaly is present or not.

[0021] In the example, classifying anomalies may include selecting a specific type from a predetermined set of potential specific types, and the output of the second machine learning program may be an anomaly type map, which includes a set of likelihood vectors at the corresponding pixel locations. Each likelihood vector includes multiple entries, each entry representing the probability that an anomaly of a corresponding potential specific type exists at the corresponding pixel location.

[0022] In the example, the first machine learning procedure may include a fully convolutional data description, followed by at least one upsampling layer. Attached Figure Description

[0023] Figure 1 This is a block diagram of an example vehicle.

[0024] Figure 2 This is a rear view seen from inside the vehicle through the windshield.

[0025] Figure 3 This is a diagram of an example machine learning architecture installed on a vehicle for classifying anomalies.

[0026] Figure 4 This is a flowchart illustrating an example process for classifying anomalies and activating vehicles. Detailed Implementation

[0027] Referring to the accompanying drawings, wherein like reference numerals indicate like parts in several views, computer 105 includes a processor and a memory storing instructions executable by the processor to: generate an anomaly map 305 by executing a first machine learning program 310, the first machine learning program 310 taking sensor data 315 from imaging sensor 110 as input; and classify anomalies 205 in the sensor data 315 into specific types by executing a second machine learning program 320. Anomaly map 305 includes a set of likelihood values ​​at corresponding pixel locations. Each likelihood value represents the probability that any type of anomaly 205 exists at the corresponding pixel location. Second machine learning program 320 takes anomaly map 305 as input.

[0028] refer to Figure 1 Vehicle 100 can be any passenger or commercial vehicle, such as a sedan, truck, SUV, crossover, van, minivan, taxi, bus, etc. Alternatively, vehicle 100 can be a different mode of transportation, such as a boat, train, airplane, helicopter, flying taxi, unmanned aerial vehicle (UAV) (e.g., drone), etc. Vehicle 100 may include a computer 105, a communication network 115, an imaging sensor 110, and at least one actuated part 120.

[0029] Computer 105 is a microprocessor-based computing device, such as a general-purpose computing device (including a processor and memory, electronic controller, etc.), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a combination thereof. Typically, hardware description languages ​​such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) are used in electronic design to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, while the logic components inside an FPGA can be configured based on VHDL programming (e.g., stored in memory electrically connected to the FPGA circuitry). Therefore, computer 105 may include a processor, memory, etc. The memory of computer 105 may include media for storing instructions executable by the processor and for electronically storing data and / or databases, and / or computer 105 may include structures such as those providing programming. Computer 105 may be multiple computers coupled together.

[0030] Computer 105 can transmit and receive data via communication network 115. Communication network 115 can be a controller local area network (CAN) bus, Ethernet, WiFi, local area network (LIN), on-board diagnostic connector (OBD-II), and / or any other wired or wireless communication network. Computer 105 can be communicatively coupled to imaging sensor 110, actuable component 120, and other components via communication network 115.

[0031] Imaging sensor 110 detects objects and / or characteristics of the external world, including the environment surrounding vehicle 100, such as other vehicles, road lane markings, traffic lights and / or signs, road users, etc. For example, imaging sensor 110 can be a camera, radar sensor, light detection and ranging (LiDAR) device, ultrasonic sensor, etc. For example, imaging sensor 110 can be a camera and can detect electromagnetic radiation within a certain wavelength range. For example, a camera can detect visible light, infrared radiation, ultraviolet light, or a range of wavelengths including visible light, infrared light, and / or ultraviolet light. For example, a camera can be a charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), or any other suitable type. As another example, imaging sensor 110 can be radar. Radar emits radio waves and receives reflections of those radio waves to detect physical objects in the environment. Radar can use direct propagation (i.e., measuring the time delay between the transmission and reception of radio waves) and / or indirect propagation (i.e., frequency-modulated continuous wave (FMCW) methods, that is, measuring the frequency change between transmitted and received radio waves). As another example, imaging sensor 110 can be a LiDAR device. A lidar device detects the distance to an object by emitting laser pulses of a specific wavelength and measuring the time it takes for the pulses to travel to and return from the object. The lidar device can be any suitable type for providing lidar data to the computer 105, such as a spindle lidar, solid-state lidar, flash lidar, etc. For example, the imaging sensor 110 can be an ultrasonic sensor. An ultrasonic sensor measures the distance to features in the environment by emitting ultrasonic waves and converting the reflected sound into an electrical signal. The imaging sensor 110 is mounted on the vehicle 100. Therefore, the imaging sensor 110 can have a portion of the field of view of the vehicle 100's surrounding environment.

[0032] Component 120 is a part of vehicle 100 that can be actuated according to instructions from computer 105 via communication network 115. In the context of this disclosure, "actuation" is defined as setting an object to motion or action via mechanical or electromechanical stimulation. For example, component 120 may include windshield wipers, headlights, pumps and / or valves for supplying washer fluid or pressurized airflow to windshield 210 or imaging sensor 110, etc.

[0033] refer to Figure 2Anomaly 205 is something in the field of view of imaging sensor 110 that hinders or degrades the ability of imaging sensor 110 to detect objects and features around vehicle 100. Anomaly 205 is "abnormal" because it deviates from the depiction of the environment around vehicle 100. Anomaly 205 can be something physically attached to vehicle 100 (e.g., insects or dirt on imaging sensor 110 or windshield 210) or a non-solid feature of the environment through which vehicle 100 can navigate (e.g., rain, snow, fog, etc.). For example, anomaly 205 can be an obstruction (i.e., something blocking the field of view of imaging sensor 110). Anomaly 205 is not an object or environmental feature that vehicle 100 will navigate around.

[0034] Computer 105 may store a predetermined set of potential specific types of anomalies 205. Potential specific types may be types of obstructions. For example, potential specific types may include rain, fog, snow in the environment, water on imaging sensor 110 or windshield 210, dirt on imaging sensor 110 or windshield 210, insects on imaging sensor 110 or windshield 210, etc. The predetermined set may include multiple potential specific types from these. Potential specific types in the predetermined set may be selected to have different visual appearances and be resolvable by different interventions.

[0035] refer to Figure 3 Computer 105 can store machine learning architecture 300 and can be programmed to execute machine learning architecture 300. In general, machine learning architecture 300 includes a first machine learning program 310 and a second machine learning program 320. The first machine learning program 310 takes sensor data 315 from imaging sensor 110 as input and generates an anomaly map 305 as output. Execution of the second machine learning program 320 can be conditional upon detecting at least one anomaly 205 based on the anomaly map 305. The second machine learning program 320 takes the anomaly map 305 as input and classifies any anomalies 205 in the sensor data 315 as output. For example, the second machine learning program 320 generates an anomaly type map 325 as output.

[0036] Sensor data 315 is generated by imaging sensor 110. For example, sensor data 315 may be an image from a camera or distance data from a radar sensor, lidar device, or ultrasonic sensor. Computer 105 may be programmed to receive sensor data 315 from imaging sensor 110 (e.g., via communication network 115).

[0037] Each image is a two-dimensional matrix of pixels. The brightness or color of each pixel is represented as one or more numerical values, such as a scalar, unitless value of luminous intensity between 0 (black) and 1 (white), or a value for each of red, green, and blue (e.g., each using an 8-bit scale (0 to 255) or a 12-bit or 16-bit scale). Pixels can be a mixture of representations (e.g., a repeating pattern of scalar values ​​of intensity for three pixels and a fourth pixel with three numerical color values, or some other pattern). The position within an image frame (i.e., the position within the field of view of the imaging sensor 110 at the time the image frame is recorded) can be specified in pixel size or coordinates (e.g., a pair of ordered pixel distances), such as the number of pixels from the top edge of the image frame and the number of pixels from the left edge of the image frame.

[0038] Distance data can be, for example, a point cloud. Points in the point cloud specify the corresponding positions in the environment relative to the distance sensor's location. For example, distance data can be in spherical coordinates, where the distance sensor is located at the origin of the spherical coordinate system. Spherical coordinates can include: radial distance (i.e., the measured depth from the distance sensor to the point measured by the distance sensor); polar angle (i.e., the angle from the vertical axis passing through the distance sensor to the point measured by the distance sensor); and azimuth angle (i.e., the angle in the horizontal plane from the horizontal axis passing through the distance sensor to the point measured by the distance sensor). The horizontal axis can, for example, be along the vehicle's direction of travel. Alternatively, the distance sensor can return the points as Cartesian coordinates at the origin of the distance sensor or as coordinates in any other suitable coordinate system, or the computer 105 can convert the spherical coordinates to Cartesian coordinates or another coordinate system after receiving the distance data.

[0039] Sensor data 315 is time-series data. As is commonly understood, and for the purposes of this disclosure, time-series data is a discrete sample of data at consecutive time steps. In this disclosure, the data sample may be an image (i.e., one image at each time step) or a point cloud (i.e., one point cloud at each time step).

[0040] Computer 105 is programmed to generate anomaly maps 305 by executing a first machine learning program 310. The first machine learning program 310 takes sensor data 315 from imaging sensor 110 as input. The first machine learning program 310 generates anomaly maps 305 as output. For example, the first machine learning program 310 may take an image or point cloud as input, and the machine learning program may generate anomaly maps 305 corresponding to the image or point cloud. When an image or point cloud is received, the first machine learning program 310 may continuously generate anomaly maps 305 at each time step.

[0041] Anomaly map 305 includes a set of likelihood values ​​at corresponding pixel locations. For example, anomaly map 305 may include the likelihood value at each pixel location in the image or point cloud of sensor data 315, meaning that anomaly map 305 has the same size as the image or point cloud (e.g., pixel height and pixel width). Alternatively, anomaly map 305 may include likelihood values ​​associated with corresponding regions, where each region contains a range of pixel values ​​around the pixel location, meaning that anomaly map 305 is downsampled from the image or point cloud.

[0042] Each likelihood value represents the probability that any type of anomaly 205 exists at the corresponding pixel location. The probability represented by the likelihood value (i.e., whether anomaly 205 exists) reflects the underlying binary random variable. The binary random variable is "underlying" because it is not itself part of the anomaly map 305. For example, a value of 1 in the binary random variable can indicate the presence of anomaly 205 at that pixel location, and a value of 0 can indicate the absence of anomaly 205 at that pixel location. The likelihood value (which is part of the anomaly map 305) indicates the probability that the binary random variable is 1. The likelihood value can take any value from 0 to 1 (e.g., the likelihood value can be stored in memory as a floating-point number). More specifically, each likelihood value in the anomaly map 305 can represent the probability that at least one potential specific type of anomaly 205 of a potential specific type exists at the corresponding pixel location. The binary random variable can represent whether at least one potential specific type of anomaly 205 of a potential specific type exists at that pixel location (regardless of which potential specific type).

[0043] The first machine learning procedure 310 can be any suitable machine learning algorithm for single-class classification bound to locations in the input data. For example, the first machine learning procedure 310 can include a fully convolutional data description 330. The fully convolutional data description 330 is a neural network using convolutional layers and pooling layers (instead of fully connected layers). The convolutional and pooling layers can be alternated, with each layer outputting to another type of layer. This structure holds data associated with pixel locations.

[0044] The first machine learning procedure 310 may further include at least one upsampling layer 335 (e.g., a fully convolutional data description 330 followed by at least one upsampling layer 335). The fully convolutional data description 330 outputs an initial anomaly map of a size smaller than the image or point cloud input to the fully convolutional data description 330. The upsampling layer 335 may output an anomaly map 305 of the same size as the input sensor data 315. For example, the upsampling layer 335 may include any suitable algorithm for increasing resolution while preserving spatial data, such as dilated convolution, upsampling via skip-connect layers, or strided transposed convolution.

[0045] The first machine learning program 310 is trained on a first set X1 of first training sensor data and, possibly, a second set X2 of second training sensor data. The first set of first training sensor data includes data samples X11, ..., X12 of the same kind as the input data (e.g., images or point clouds) of the first machine learning program 310. n The set. The first training sensor data includes binary labels y11, ..., y1 indicating the presence or absence of an anomaly 205. n (i.e., the true value). Each binary label is applied holistically to the corresponding training sensor data recorded at a specific time step (e.g., holistically to an image or holistically to a point cloud), in other words, for data sample X1 i Tag y1 i = 0 indicates no abnormality 205, and y1 i = 1 indicates exception 205, where i is the index of time step from 1 to n. The first set can be selected as a data sample containing all potential exceptions 205 of a specific type and a data sample containing no exceptions 205 of any type.

[0046] The second set of training sensor data includes data samples X21, ..., X2 of the same kind as the input data (e.g., images or point clouds) of the first machine learning program 310. n The set. The second training data includes labels indicating the pixel locations of anomaly 205. For example, each label y2 i It can indicate whether an anomaly 205 exists at the corresponding pixel location; in other words, y2 i = (0, u, v) indicates that there is no anomaly at pixel coordinates (u, v) at time step i, and y2 i = (1, u, v) indicates an anomaly 205 at pixel coordinates (u, v) at time step i. The first set can be the majority of the training data used to train the first machine learning program 310. For example, the number of data samples in the first set X1 can be several orders of magnitude larger than the number of samples in the second set X2 (e.g., 2,000,000 vs. 50,000). The creation of the first set is significantly less labor-intensive, and the relatively small number of samples in the second set is sufficient to increase the accuracy of the output.

[0047] The first machine learning procedure 310 (e.g., fully convolutional data description 330) can be trained using any suitable training algorithm (such as backpropagation with an optimization algorithm—such as gradient descent applied to the objective function). For example, the objective function can be a pseudo-Huber loss, as given in the following expression:

[0048]

[0049] Where W is the first machine learning program 310 A(X; W) is the weight set of (X; W), where X is the training sensor dataset and A(X) is the first machine learning program 310. The output matrix is ​​(X; W), where u and v are the pixel height and width of the preliminary anomaly map (before upsampling), and ||A(X)||1 is the sum of the entries in the output matrix A(X). The output matrix is ​​given by the following expression:

[0050]

[0051] Computer 105 can be programmed to determine whether at least one anomaly 205 is detected in a data sample. For example, computer 105 can be programmed to determine an anomaly score for an anomaly map 305. The anomaly score can represent the overall probability that a data sample (e.g., an image or point cloud) contains at least one anomaly 205. For example, the anomaly score can be the sum of likelihood values ​​for the data sample (i.e., the sum of likelihood values ​​in a single image or point cloud). Computer 105 can be further programmed to determine whether the anomaly score exceeds a threshold. This threshold can be selected to indicate the possible presence of anomaly 205.

[0052] The execution of the second machine learning program 320 may be conditional upon an anomaly score exceeding a threshold. For example, computer 105 may be programmed to classify anomaly 205 into a specific type by executing the second machine learning program 320 in response to an anomaly score exceeding a threshold, as described below. Computer 105 may also be programmed to avoid executing the second machine learning program 320 in response to an anomaly score falling below a threshold (e.g., during that time step, or until the first machine learning program 310 has been executed again).

[0053] Computer 105 is programmed to classify anomalies 205 in sensor data 315 into specific types by executing a second machine learning program 320. The second machine learning program 320 takes anomaly maps 305 as input. The second machine learning program 320 classifies any anomalies 205 in the sensor data 315 as output. For example, the second machine learning program 320 generates anomaly type maps 325 as output. The second machine learning program 320 can continuously generate anomaly type maps 325 at each time step as anomaly maps 305 are received.

[0054] Anomaly type graph 325 includes a set of likelihood vectors at the corresponding pixel locations. For example, anomaly type graph 325 may include the likelihood vector at each pixel location in anomaly graph 305, meaning that anomaly type graph 325 has the same size as anomaly graph 305 (e.g., pixel height and pixel width), which may be the same size as the image or smaller than the image.

[0055] Each likelihood vector comprises multiple entries. Each entry represents the probability that an anomaly 205 of a corresponding latent specific type exists at the corresponding pixel location. For example, if the predetermined set of latent specific types includes six types, then the likelihood vectors each have six entries, one entry for each latent specific type. In other words, the anomaly type graph 325 has channels for each entry in the likelihood vector (e.g., six in this example), in... Figure 3 The values ​​are shown as separate plots. The presence of each potential specific type of anomaly 205 is represented by a binary random variable. For example, a value of 1 in the binary random variable can indicate the presence of anomaly 205 of that type at that pixel location, and a value of 0 can indicate the absence of anomaly 205 of that type at that pixel location (but a different potential specific type may exist). The value of each entry in the likelihood vector indicates the probability that the corresponding binary random variable is 1. The value of the entry can take any value from 0 to 1.

[0056] The second machine learning program 320 can be any suitable algorithm for selecting a specific type from a predetermined set of potential specific types. For example, computer 105 can execute a semantic segmentation algorithm. Semantic segmentation algorithms, for example, label each pixel in an image by determining the probability of each potential specific type in the predetermined set and selecting the potential specific type with the highest probability. Types of semantic segmentation algorithms include U-Net (such as...). Figure 3 (as shown in the image), deep neural networks, convolutional neural networks, etc.

[0057] The second machine learning procedure 320 is trained on a training dataset that includes anomaly maps paired with ground truth anomaly type maps. The ground truth anomaly type maps can be efficiently created by applying anomaly 205 for each potential specific type present at any location in the image frame across the entire ground truth anomaly type map (e.g., (0, 0, 1, 1, 0, 0) for all pixel locations if a third and fourth potential specific type are present), and then applying a mask over pixel locations without anomaly 205 according to the corresponding anomaly map (e.g., setting the masked pixel locations to (0, 0, 0, 0, 0, 0)).

[0058] The second machine learning program 320 can be trained using any suitable training algorithm (such as backpropagation with an optimization algorithm—such as gradient descent applied to the objective function). For example, the objective function could be L2 loss.

[0059] Computer 105 can be programmed to actuate component 120 based on the output of machine learning architecture 300. As an example, computer 105 can actuate component 120 of vehicle 100 based on a specific type of exception 205 (e.g., as indicated by exception type diagram 325). For example, computer 105 can actuate windshield wipers in response to a specific type of rain. For example, computer 105 can actuate a climate control system to blow hot air onto windshield 210 in response to a specific type of snow in the environment or ice on windshield 210. For example, computer 105 can actuate air nozzles aimed at imaging sensor 110 in response to a specific type of insect on imaging sensor 110. For example, computer 105 can actuate a pump or valve controlling the flow of washer fluid to fluid nozzles in response to a specific type of dirt or dust on imaging sensor 110.

[0060] Alternatively or additionally, computer 105 may actuate components 120 of vehicle 100 based on the location of anomaly 205 indicated by anomaly diagram 305. The location of anomaly 205 indicated in anomaly diagram 305 may be transferred to anomaly type diagram 325. For example, computer 105 may select one nozzle or subset of nozzles to be actuated from a plurality of nozzles targeting imaging sensor 110 based on the pixel location of anomaly 205 on imaging sensor 110. For example, computer 105 may actuate windshield wipers at a slower or less frequent speed in response to anomaly 205 covering a portion of windshield 210, and at a faster or more frequent speed in response to anomaly 205 covering the entire windshield 210.

[0061] Figure 4 This is a flowchart illustrating an example process 400 for classifying anomaly 205 and actuating component 120. The memory of computer 105 stores executable instructions for performing the steps of process 400, and / or can be programmed in a structure as mentioned above. As a general overview of process 400, computer 105 receives sensor data 315, generates an anomaly map 305, and determines an anomaly score in the anomaly map 305. In response to the presence of anomaly 205, computer 105 classifies anomaly 205 in the anomaly map 305 and actuates component 120. Process 400 continues as long as vehicle 100 remains open.

[0062] Process 400 begins in block 405, where computer 105 receives sensor data 315 from imaging sensor 110, as described above.

[0063] Next, in box 410, computer 105 generates anomaly graph 305 by executing first machine learning program 310, as described above.

[0064] Next, in box 415, computer 105 determines the anomaly score of anomaly graph 305 as described above.

[0065] Next, in decision box 420, computer 105 determines whether an anomaly exists 205 by determining whether the anomaly score exceeds a threshold, as described above. In response to the anomaly score exceeding the threshold, process 400 proceeds to box 425. In response to the anomaly score falling below the threshold, process 400 proceeds to decision box 435.

[0066] In box 425, computer 105 classifies anomalies 205 in sensor data 315 into specific types by executing a second machine learning program 320, thereby generating an anomaly type graph 325, as described above.

[0067] Next, in box 430, computer 105 actuates component 120 of vehicle 100 based on at least one of the specific type of anomaly 205 or the location of anomaly 205 indicated by anomaly diagram 305, as described above. After box 430, process 400 proceeds to decision box 435.

[0068] In decision box 435, computer 105 determines whether vehicle 100 is started (i.e., in the start state). For the purposes of this disclosure, "start state" is defined as a state in which all electrical power is supplied to the electrical components of vehicle 100 and vehicle 100 is ready to be driven (e.g., the engine is running); "off state" is defined as a state in which a small amount of electrical power is supplied to selected electrical components of vehicle 100 (typically used when vehicle 100 is stored); and "accessory energized state" is defined as a state in which all electrical power is supplied to more electrical components compared to the off state and vehicle 100 is not ready to be driven. Typically, the operator puts vehicle 100 in the start state when the operator is about to operate vehicle 100, puts vehicle 100 in the off state when the operator is about to leave vehicle 100, and puts vehicle 100 in the accessory energized state when the operator is about to sit in vehicle 100 but not operate vehicle 100. In response to vehicle 100 being in the start state, process 400 returns to block 405 to continue receiving sensor data 315. In response to vehicle 100 transitioning to the off state or accessory being powered on, process 400 ends.

[0069] Generally, the described computing system and / or device may employ any of a variety of computer operating systems, including, but not limited to, the following versions and / or types: Ford Sync® applications; AppLink / Smart Device Link middleware; Microsoft Automotive® operating system; Microsoft Windows® operating system; Unix operating system (e.g., Solaris® operating system released by Oracle Corporation of Redwood Coast, California); AIX UNIX operating system released by International Business Machines Corporation of Armonk, New York; Linux operating system; Mac OSX and iOS operating systems released by Apple Inc. of Cupertino, California; BlackBerry OS released by BlackBerry Ltd. of Waterloo, Canada; and Android operating system developed by Google and the Open Handset Alliance; or the QNX® CAR infotainment platform provided by QNX Software Systems. Examples of computing devices include, but are not limited to, in-vehicle computers, computer workstations, servers, desktop computers, laptop computers, handheld computers, or other computing systems and / or devices.

[0070] Computing devices typically include computer-executable instructions, which can be executed by one or more computing devices (such as those listed above). Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages ​​and / or technologies, which, individually or in combination, include, but are not limited to, Java™, C, C++, Matlab, Simulink, Stateflow, Visual Basic, JavaScript, Python, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as the Java Virtual Machine, the Dalvik Virtual Machine, etc. Generally, a processor (e.g., a microprocessor) receives instructions (e.g., from memory, computer-readable media, etc.) and executes those instructions to perform one or more processes, including one or more processes described herein. Such instructions and other data can be stored and transferred using a variety of computer-readable media. Files in a computing device are typically collections of data stored on computer-readable media (such as storage media, random access memory, etc.).

[0071] Computer-readable media (also known as processor-readable media) include any non-transitory (e.g., tangible) medium that contributes to providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, and internals that constitute a system bus coupled to the computer's processor. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or magnetic tape, or any other medium from which a computer can read.

[0072] The databases, data repositories, or other data stores described herein can include various mechanisms for storing, accessing / retrieving various types of data, including hierarchical databases, file sets in file systems, application databases in proprietary formats, relational database management systems (RDBMS), NoSQL databases, graph databases (GDBs), and so on. Each such data store is typically contained within a computing device employing a computer operating system (such as one of those mentioned above) and is accessed via a network in any one or more of various ways. File systems can be accessed from the computer operating system and can include files stored in various formats. In addition to languages ​​used to create, store, edit, and execute the stored programs (such as PL / SQL mentioned above), RDBMS typically employs Structured Query Language (SQL).

[0073] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) and stored on computer-readable media (e.g., disks, storage, etc.) associated therewith. Computer program products may include such instructions stored on computer-readable media for performing the functions described herein.

[0074] In the accompanying drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements may be changed. Regarding the media, processes, systems, methods, inspirations, etc., described herein, it should be understood that although the steps of such processes, etc., are described as occurring in a certain ordered order, such processes can be practiced by performing the steps in a different order than that described herein. It should also be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted.

[0075] This disclosure has been described in an illustrative manner, and it should be understood that the terminology used is intended to be descriptive in nature, not restrictive. The use of terms such as “in response to,” “after determining,” etc., indicates a causal relationship, not merely a temporal one. The adjectives “first,” “second,” “third,” and “fourth” are used throughout this document as identifiers, not intended to indicate importance, order, or quantity. In light of the foregoing teachings, numerous modifications and variations of this disclosure are possible, and this disclosure can be practiced in ways other than those specifically described.

[0076] According to the present invention, a computer is provided having a processor and a memory storing instructions executable by the processor to: generate an anomaly map by executing a first machine learning program, the first machine learning program taking sensor data from an imaging sensor as input, the anomaly map including a set of likelihood values ​​at corresponding pixel locations, each likelihood value representing the probability that any type of anomaly exists at the corresponding pixel location; and classify anomalies in the sensor data into specific types by executing a second machine learning program taking the anomaly map as input.

[0077] According to an embodiment, the imaging sensor is mounted on the vehicle, and the instructions further include instructions for actuating vehicle components based on a specific type of anomaly.

[0078] According to an embodiment, an imaging sensor is mounted on a vehicle, and the instructions further include instructions for actuating vehicle components based on the location of an anomaly indicated by an anomaly map.

[0079] According to an embodiment, the first machine learning program is trained on a training sensor dataset with binary labels that indicate whether an anomaly exists or not.

[0080] According to an embodiment, each binary tag is applied as a whole to the corresponding training sensor data recorded at a specific time step.

[0081] According to an embodiment, the training sensor dataset is a first set of first training sensor data, and the first machine learning program is trained on a second set of second training sensor data with labels indicating pixel locations of anomalies.

[0082] According to an embodiment, the first set is the majority of the training data used to train the first machine learning program.

[0083] According to an embodiment, the instructions for classifying anomalies include instructions for selecting a specific type from a predetermined set of potential specific types.

[0084] According to an embodiment, each likelihood value in the anomaly graph represents the probability that at least one potential specific type of anomaly exists at the corresponding pixel location.

[0085] According to an embodiment, the potential specific type is the type of obstruction.

[0086] According to an embodiment, the output of the second machine learning program is an anomaly type map, which includes a set of likelihood vectors at corresponding pixel locations. Each likelihood vector includes multiple entries, and each entry represents the probability that an anomaly of a corresponding potential specific type exists at the corresponding pixel location.

[0087] According to an embodiment, the instructions further include instructions for determining an anomaly score in an anomaly graph and, in response to an anomaly score exceeding a threshold, classifying the anomaly into a specific type by executing a second machine learning procedure.

[0088] According to an embodiment, the first machine learning program includes a fully convolutional data description.

[0089] According to an embodiment, the first machine learning program includes at least one upsampling layer.

[0090] According to an embodiment, the imaging sensor is a camera, and the sensor data is an image.

[0091] According to the present invention, a method includes: generating an anomaly map by executing a first machine learning program that takes sensor data from an imaging sensor as input, the anomaly map including a set of likelihood values ​​at corresponding pixel locations, each likelihood value representing the probability that any type of anomaly exists at the corresponding pixel location; and classifying anomalies in the sensor data into specific types by executing a second machine learning program that takes the anomaly map as input.

[0092] In one aspect of the invention, an imaging sensor is mounted to a vehicle, and the method further includes actuating a component of the vehicle based on at least one of a specific type of anomaly or the location of an anomaly indicated by an anomaly map.

[0093] In one aspect of the invention, a first machine learning program is trained on a training sensor dataset having binary labels that indicate whether an anomaly is present or not.

[0094] In one aspect of the invention, classifying anomalies includes selecting a specific type from a predetermined set of potential specific types, and the output of a second machine learning program is an anomaly type map, which includes a set of likelihood vectors at corresponding pixel locations, each likelihood vector including multiple entries, each entry representing the probability that an anomaly of a corresponding potential specific type exists at the corresponding pixel location.

[0095] In one aspect of the invention, the first machine learning procedure includes a fully convolutional data description, followed by at least one upsampling layer.

Claims

1. A method comprising: An anomaly map is generated by executing a first machine learning program that takes sensor data from an imaging sensor as input. The anomaly map includes a set of likelihood values ​​at corresponding pixel locations, each likelihood value representing the probability that any type of anomaly exists at the corresponding pixel location. as well as Anomalies in the sensor data are classified into specific types by executing a second machine learning program, which uses the anomaly map as input.

2. The method of claim 1, wherein the imaging sensor is mounted on the vehicle, the method further comprising: The vehicle's components are actuated based on the specific type of the anomaly.

3. The method of claim 1, wherein the imaging sensor is mounted on the vehicle, the method further comprising: The vehicle components are actuated based on the location of the anomaly indicated by the anomaly diagram.

4. The method of claim 1, wherein the first machine learning program is trained on a training sensor dataset having binary labels indicating whether an anomaly is present or not.

5. The method of claim 4, wherein each binary tag is applied as a whole to the corresponding training sensor data recorded at a specific time step.

6. The method of claim 4, wherein the training sensor dataset is a first set of first training sensor data, and the first machine learning program is trained on a second set of second training sensor data with labels indicating pixel locations of anomalies.

7. The method of claim 6, wherein the first set is the majority of training data used to train the first machine learning program.

8. The method of claim 1, wherein classifying the anomaly includes selecting the specific type from a predetermined set of potential specific types.

9. The method of claim 8, wherein each likelihood value in the anomaly graph represents the probability that at least one potential specific type of anomaly exists at the corresponding pixel location.

10. The method of claim 8, wherein the potential specific type is the type of obstruction.

11. The method of claim 8, wherein the output of the second machine learning program is an anomaly type graph, the anomaly type graph comprising a set of likelihood vectors at corresponding pixel locations, each likelihood vector comprising multiple entries, each entry representing the probability that an anomaly of a corresponding potential specific type exists at the corresponding pixel location.

12. The method of claim 1, further comprising: An anomaly score is determined for the anomaly graph, and in response to the anomaly score exceeding a threshold, the anomaly is classified into the specific type by executing the second machine learning program.

13. The method of claim 1, wherein the first machine learning procedure comprises a fully convolutional data description, followed by at least one upsampling layer.

14. The method of claim 1, wherein the imaging sensor is a camera and the sensor data is an image.

15. A computer comprising a processor and a memory storing instructions executable by the processor to perform the method as claimed in any one of claims 1 to 14.

Images