Sensor data occlusion positioning

Abstract

The present disclosure provides "sensor data occlusion localization." A computer includes a processor and a memory, and the memory stores instructions executable by the processor to: detect a feature in sensor data from an imaging sensor; and perform a transformer on the feature. The transformer transforms the feature to a bird's-eye view embedding space, classifies an occlusion from the feature, and outputs a spatial location of the occlusion. The transformer is trained to classify occlusions and is not trained to classify objects.

Description

Technical Field

[0001] This disclosure provides techniques for addressing occlusion that affects sensor data from imaging sensors. Background Technology

[0002] Vehicles typically include a variety of sensors. Sensors can provide data about vehicle operation, such as wheel speed, wheel orientation, and engine and transmission data (e.g., temperature, fuel consumption, etc.). Sensors can detect the vehicle's position and / or orientation. Sensors may include Global Positioning System (GPS) sensors; accelerometers, such as piezoelectric or microelectromechanical systems (MEMS); gyroscopes, such as rate gyroscopes, ring laser gyroscopes, or fiber optic gyroscopes; inertial measurement units (IMUs); and magnetometers. Sensors can detect the external world, including objects and / or characteristics of the vehicle's surrounding environment, such as other vehicles, road lane markings, traffic lights and / or signs, road users, etc. Sensors may include radar sensors, ultrasonic sensors, scanning laser rangefinders, light detection and ranging (LiDAR) devices, and image processing sensors (such as cameras). Summary of the Invention

[0003] For example, the imaging sensor may be a camera mounted on a vehicle, the sensor data may be an image, and the type of occlusion may include rain, fog, debris on the camera lens, etc. A computer may be programmed to detect features in the sensor data from the imaging sensor and perform a transformer on those features. The transformer transforms the features into a bird's-eye view embedding space, classifies the occlusion from the features, and outputs the spatial location of the occlusion. The transformer is trained to classify occlusions but not to classify objects.

[0004] Classifying anomalies can help mitigate them. Vehicle components can be actuated based on classification, such as activating windshield wipers in response to rain or cleaning nozzles in response to debris on camera lenses. Spatial location can also help mitigate anomalies, such as activating windshield wipers before entering the rain. Training only on occlusions, rather than objects, keeps the training load within reasonable limits. Furthermore, recognizing occlusions can be used to actuate different vehicle components compared to recognizing objects. Occlusion recognition can be used to actuate components related to cleaning and clearing the field of vision, such as windshield wipers or nozzles for imaging sensors. Object recognition can alternatively be used to control vehicle movement.

[0005] A computer includes a processor and a memory, the memory storing instructions executable by the processor to: detect features in sensor data from an imaging sensor; and perform a transformer on the features. The transformer transforms the features into a bird's-eye view embedding space, classifies occlusions from the features, and outputs the spatial location of the occlusions. The transformer is trained to classify occlusions but not to classify objects.

[0006] In the example, the instructions may further include instructions for actuating the component based on the spatial location of the occlusion. In a further example, the vehicle may include the imaging sensor and the component.

[0007] In the example, the instructions may further include instructions for actuating the component based on the classification of the occlusion.

[0008] In one example, the imaging sensor may be a camera, and the sensor data may be image data. In a further example, the transformer may be programmed to fuse the features from the image data with distance data from at least one distance sensor in the bird's-eye view embedding space. In yet another example, the instructions may further include instructions to perform: projecting the features onto the initial spatial position occluded in the bird's-eye view embedding space, and the transformer may be programmed to correct the occluded initial spatial position to the spatial position using the distance data.

[0009] In another, further example, the instructions may further include instructions for performing the following operation: rasterizing the distance data before the transformer fuses the feature and the distance data.

[0010] In another further example, the at least one distance sensor may include at least one of lidar or radar.

[0011] In another further example, the at least one distance sensor may include lidar and radar.

[0012] In the example, the spatial location can be specified in at least two horizontal dimensions.

[0013] In the example, the instructions for classifying the occlusion may include instructions for selecting a type from a predetermined set of potential types. In a further example, the predetermined set of potential types may include at least two of rain, fog, dirt on the imaging sensor, or insects on the imaging sensor.

[0014] One method includes detecting features in sensor data from an imaging sensor and performing a transformer on the features. The transformer transforms the features into a bird's-eye view embedding space, classifies occlusions from the features, and outputs the spatial location of the occlusions. The transformer is trained to classify occlusions but not to classify objects.

[0015] In the example, the method may further include actuating components based on the spatial location of the occlusion, wherein the vehicle includes the imaging sensor and the components.

[0016] In the example, the method may further include actuating components based on the classification of the occlusion, wherein the vehicle includes the imaging sensor and the components.

[0017] In one example, the transformer can be programmed to fuse the feature with distance data from at least one distance sensor into the bird's-eye view embedded space. In a further example, the transformer can be programmed to correct the spatial position of the occlusion using the distance data.

[0018] In another further example, the method may further include rasterizing the distance data before the transformer fuses the features and the distance data.

[0019] In the example, classifying the occlusion includes selecting a type from a predetermined set of potential types, and the predetermined set of potential types includes at least two of rain, fog, dirt on the imaging sensor, or insects on the imaging sensor. Attached Figure Description

[0020] Figure 1 This is a block diagram of an example vehicle.

[0021] Figure 2 It is a rear view of the vehicle from inside through the windshield.

[0022] Figure 3 It is an illustration of an obstruction embedded in a space in a bird's-eye view.

[0023] Figure 4 This is a diagram of an example machine learning architecture installed on a vehicle for analyzing occlusion.

[0024] Figure 5 This is a flowchart of an example process used to analyze occlusion and actuate a vehicle. Detailed Implementation

[0025] Referring to the accompanying drawings, where the same numbers indicate the same parts throughout several views, a computer 105 includes a processor and a memory, and the memory stores instructions executable by the processor to detect features in sensor data 405 from an imaging sensor 110 and to perform a transformer 410 on the features. The transformer 410 transforms the features into a bird's-eye view embedding space 300, classifies occlusions 205 from the features, and outputs the spatial location of the occlusions 205. The transformer 410 is trained to classify occlusions 205 and not to classify objects.

[0026] refer to Figure 1 Vehicle 100 can be any passenger or commercial vehicle, such as a sedan, truck, SUV, crossover, van, minivan, taxi, bus, etc. Alternatively, vehicle 100 can be a different mode of transportation, such as a ship, train, airplane, helicopter, flying taxi, unmanned aerial vehicle (UAV) (e.g., drone), etc. Vehicle 100 may include a computer 105, a communication network 115, an imaging sensor 110 (e.g., camera 120 and / or distance sensor 125), and actuated parts 130.

[0027] Computer 105 is a microprocessor-based computing device, such as a general-purpose computing device (including a processor and memory, electronic controller, etc.), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a combination thereof. Typically, hardware description languages ​​such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) are used in electronic design to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, while the logic components inside an FPGA can be configured based on VHDL programming (e.g., stored in memory electrically connected to the FPGA circuitry). Therefore, computer 105 may include a processor, memory, etc. The memory of computer 105 may include media for storing instructions executable by the processor and for electronically storing data and / or databases, and / or computer 105 may include structures such as those providing programming capabilities. Computer 105 may be multiple computers coupled together.

[0028] Computer 105 can transmit and receive data via communication network 115. Communication network 115 can be a controller area network (CAN) bus, Ethernet, WiFi, local area network (LIN), on-board diagnostic connector (OBD-II), and / or any other wired or wireless communication network. Computer 105 can be communicatively coupled to imaging sensor 110, actuable component 130, and other components via communication network 115.

[0029] Imaging sensor 110 detects objects and / or features of the external world, including the surrounding environment of vehicle 100, such as other vehicles, road lane markings, traffic lights and / or signs, road users, etc. For example, imaging sensor 110 may include at least one camera 120 and / or at least one distance sensor 125. Imaging sensor 110 is mounted to vehicle 100. Therefore, imaging sensor 110 can provide a field of view of a portion of the surrounding environment of vehicle 100.

[0030] Camera 120 can detect electromagnetic radiation within a certain wavelength range. For example, camera 120 can detect visible light, infrared radiation, ultraviolet light, or a range of wavelengths including visible light, infrared light, and / or ultraviolet light. For example, camera 120 may be a charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), or any other suitable type. As another example, camera 120 may be a time-of-flight (TOF) camera 120, which includes a modulated light source for illuminating the environment and detects both reflected light from the modulated light source and ambient light to sense the reflectivity amplitude and distance to the scene. As yet another example, camera 120 may be a stereo camera 120, where the parallax between two cameras 120 provides an estimate of the three-dimensional position of features in the environment.

[0031] The distance sensor 125 may include at least one lidar, at least one radar, and / or at least one ultrasonic sensor. The radar emits radio waves and receives reflections of those radio waves to detect physical objects in the environment. The radar may use direct propagation (i.e., measuring the time delay between the transmission and reception of radio waves) and / or indirect propagation (i.e., the frequency-modulated continuous wave (FMCW) method, that is, measuring the frequency change between the transmitted and received radio waves). The lidar detects the distance to an object by emitting laser pulses of a specific wavelength and measuring the time of flight of the pulses to and from the object. The lidar may be any suitable type for providing lidar data operable by the computer 105, such as a spindle lidar, a solid-state lidar, a flash lidar, etc. The ultrasonic sensor measures the distance to features in the environment by emitting ultrasonic waves and converting the reflected sound into an electrical signal.

[0032] Component 130 is a part of vehicle 100 that can be actuated according to instructions from computer 105 via communication network 115. In the context of this disclosure, "actuation" is defined as setting an object to motion or action via mechanical or electromechanical stimulation. For example, component 130 may include a windshield wiper, headlights, fog lights, a pump and / or valve for supplying washer fluid or pressurized airflow to windshield 210 or to one of the imaging sensors 110, etc.

[0033] refer to Figure 2 An obstruction 205 is something that hinders or reduces the ability of an imaging sensor 110 to detect objects and features in the field of view of an imaging sensor 110, such as something that blocks the field of view of the imaging sensor 110. An obstruction 205 can be something that deviates from the depiction of the environment around the vehicle 100, such as something that blocks the field of view of the imaging sensor 110. An obstruction 205 can be something that is temporarily physically attached to the vehicle 100 (e.g., insects or dirt on the imaging sensor 110 or the windshield 210), or a non-solid feature of the environment through which the vehicle 100 can navigate (e.g., rain, snow, fog, etc.). Figure 2 Rain is shown on the windshield 210. The obstruction 205 is not an object or environmental feature that the vehicle 100 will navigate around (i.e., a separate physical object separate from the vehicle 100).

[0034] Computer 105 may store a predetermined set of potential types of occlusion 205. For example, potential types may include rain, fog, snow, water on imaging sensor 110 or windshield 210, dirt on imaging sensor 110 or windshield 210, insects on imaging sensor 110 or windshield 210, etc. The predetermined set may include multiple (i.e., at least two) of these potential types. Potential types in the predetermined set may be selected to have visually different appearances and can be addressed through different interventions.

[0035] refer to Figure 3The bird's-eye view embedding space 300 is a virtual reference frame used to locate occlusion 205 relative to vehicle 100. The location of occlusion 205 in the bird's-eye view embedding space 300 indicates its location within the environment surrounding vehicle 100. The bird's-eye view embedding space 300 is defined relative to vehicle 100, so it moves with vehicle 100 as it moves through the environment. The bird's-eye view embedding space 300 is a "bird's-eye view" because it includes data indicating the position seen when looking straight down from vertically above. In other words, the bird's-eye view embedding space 300 includes two horizontal dimensions. For example, the bird's-eye view embedding space 300 could be a horizontal plane lacking a vertical dimension. Alternatively, the bird's-eye view embedding space 300 could additionally include a vertical dimension. In two or three dimensions, the bird's-eye view embedding space 300 could be a grid, in other words, subdivided into spaces along dimensions (i.e., squares or cubes). The bird's-eye view embedding space 300 is called an "embedding space" because the positions of features in the bird's-eye view embedding space 300 are generated by the operation of machine learning techniques described below, given sensor data 405 about the environment.

[0036] The occlusion 205 is specified by spatial location within the bird's-eye view embedding space 300. Each spatial location is specified in at least two horizontal dimensions. For example, if the bird's-eye view embedding space 300 is a horizontal plane, the spatial location can be specified in two horizontal dimensions. For example, the two spatial dimensions could be the longitudinal distance in front of vehicle 100 and the lateral distance to the left of vehicle 100 (where the distance to the right is indicated as a negative number). If the bird's-eye view embedding space 300 includes a vertical dimension, the spatial location can be specified in three spatial dimensions (including the two horizontal dimensions).

[0037] refer to Figure 4 The computer 105 can store the machine learning architecture 400 and can be programmed to execute the machine learning architecture 400. In general, the machine learning architecture 400 may include feature projectors 415 and transformers 410. Each feature projector 415 may receive sensor data 405 from one of the imaging sensors 110, detect features, and determine the initial spatial location corresponding to the features. The feature projectors 415 may include at least one image feature projector 415a and at least one distance feature projector 415b (e.g., Figure 4The example includes one image feature projector 415a and two distance feature projectors 415b. Image feature projector 415a acquires image data 420 from camera 120 as input. Each distance feature projector 415b acquires input from one of the distance sensors 125. For example, distance feature projectors 415b may include at least one distance feature projector 415b acquiring input from radar and at least one distance feature projector 415b acquiring input from lidar. Using both radar and lidar for distance data 425 is advantageous because radar typically detects rain, while lidar typically sees through rain. Transformer 410 transforms the features from feature projectors 415 into a bird's-eye view embedding space 300, classifies occlusions 205 from the features, and outputs the spatial location of occlusions 205.

[0038] Sensor data 405 is generated by imaging sensor 110. For example, sensor data 405 may include image data 420 from camera 120 and distance data 425 from distance sensor 125. Computer 105 may be programmed (e.g., via communication network 115) to receive sensor data 405. Each stream of sensor data 405 from one of the imaging sensors 110 may be an input to a corresponding feature projector 415.

[0039] Image data 420 includes one or more image frames. Each image frame is a two-dimensional pixel matrix. The brightness or color of each pixel is represented as one or more numerical values, such as a scalar, unitless value of luminance intensity between 0 (black) and 1 (white), or a value for each of red, green, and blue (e.g., each using an 8-bit scale (0 to 255) or a 12-bit or 16-bit scale). Pixels can be a mixture of representations (e.g., a repeating pattern of scalar values ​​of intensity for three pixels and a fourth pixel with three numerical color values, or some other pattern). The position within an image frame (i.e., the position in the field of view of camera 120 at the time the image frame is recorded) can be specified by pixel size or coordinates (e.g., a pair of ordered pixel distances), such as a number of pixels from the top edge of the image frame and a number of pixels from the left edge of the image frame.

[0040] The range data 425 can be, for example, a point cloud. Points in the point cloud can specify the corresponding positions in the environment relative to the location of the range sensor 125. For example, the range data 425 can be in spherical coordinates, with the range sensor 125 located at the origin of the spherical coordinate system. The spherical coordinates can include: radial distance (i.e., the measured depth from the range sensor to the point measured by the range sensor); polar angle (i.e., the angle from the vertical axis passing through the range sensor to the point measured by the range sensor); and azimuth angle (i.e., the angle in the horizontal plane from the horizontal axis passing through the range sensor to the point measured by the range sensor). The horizontal axis can, for example, be along the vehicle's direction of travel. Alternatively, the range sensor 125 can return the points as Cartesian coordinates of the range sensor 125 at the origin or as coordinates in any other suitable coordinate system, or the computer 105 can convert the spherical coordinates to Cartesian coordinates or another coordinate system after receiving the range data 425.

[0041] Computer 105 is programmed to detect features in sensor data 405 from imaging sensor 110. For the purposes of this disclosure, the term "feature" in its computer vision sense is used as a piece of information about the content of an image or point cloud, specifically about whether a certain region of sensor data 405 possesses certain properties. Types of features may include edges, corners, blobs, etc.

[0042] For example, computer 105 can detect features by executing a corresponding feature projector 415 for each stream of sensor data 405. Feature projector 415 can be used as a feature extractor. Each feature projector 415 may include one or more techniques suitable for that type of sensor data 405 (e.g., image, distance data 425) for feature extraction, such as low-level techniques like edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transform (SIFT), etc.; shape-based techniques like thresholding, blob extraction, template matching, Hough transform, generalized Hough transform, etc.; flexible methods like deformable parameterized shapes, active contours, etc.; and so on. Feature projector 415 may include machine learning operations. For example, feature projector 415 may include a residual network (ResNet) layer followed by a convolutional neural network.

[0043] Computer 105 can be programmed to project features to an initial spatial location of occlusion 205 within the bird's-eye view embedding space 300. For example, computer 105 can project features to the initial spatial location by executing a corresponding feature projector 415 for each stream of sensor data 405. Feature projector 415 can include any suitable machine learning technique for projection, such as a monocular depth estimator, multilayer perceptron (MLP), etc. Alternatively, for image data 420, computer 105 can apply a predefined geometric relation mapping an image plane to a ground plane. This geometric relation can be homography between the ground plane and the image plane of camera 120. The term "homography" is used in the sense of isomorphic projection geometry between projection spaces (in this case, between the projection space of the ground plane and the projection space of the image plane of camera 120). In either case, the initial spatial location can be assigned to a point on a grid of the bird's-eye view embedding space 300.

[0044] Computer 105 can be programmed to rasterize distance data 425 before transformer 410 fuses feature and distance data 425 from image data 420 (as described below). Computer 105 can rasterize distance data 425 for each stream from distance sensor 125 (e.g., for such...). Figure 4 The radar data and lidar data shown in the example are rasterized by the corresponding feature projector 415 to rasterize the range data 425. The term "rasterization" is used in its graphical sense as the process of converting spatial data into a pixel or voxel grid. In this case, the grid may be a bird's-eye view embedded in space 300. The computer 105 may use any suitable rasterization algorithm to sample the range data 425 to determine the pixel values ​​of the grid.

[0045] Computer 105 is programmed to perform transformer 410 on features returned by feature projector 415. Features may include corresponding initial spatial locations specified within a grid of the bird's-eye view embedding space 300. Transformer 410 transforms the features into the bird's-eye view embedding space 300, classifies any occlusions 205 from the features, and outputs the spatial location of the occlusions 205. As output, transformer 410 provides the classification of occlusions 205 and the spatial location of occlusions 205 within the bird's-eye view embedding space 300. Figure 3In the example, the output includes dirt on the windshield 210 (at the lower boundary of the bird's-eye view embedding space 300), rain on the left front of the vehicle 100, and fog on the right further ahead of the vehicle 100. Transformer 410 selects a type from a predetermined set of potential types for classifying occlusion 205. Transformer 410 is programmed to fuse features from image feature projector 415a with distance data 425 from at least one distance sensor 125 (e.g., as output by one or more of distance feature projectors 415b) in the bird's-eye view embedding space 300. The spatial location output by transformer 410 represents the fusion of the initial spatial locations of corresponding features and / or rasterized point clouds from the different feature projectors 415.

[0046] As a general overview of the structure of transformer 410, transformer 410 may include a fixed number of L A transformer layer (in) Figure 4 (Indicated by "L x"). The number of transformer layers can be selected. L To balance fast convergence and processing time with accurate feature detection. Each transformer layer may include (e.g., consist of) a first normalization layer 430, a multi-head attention layer 435, a first summing block 440, a second normalization layer 445, a multilayer perceptron (MLP) 450, and a second summing block 455 (e.g., in this order). Transformer 410 is programmed to use distance data 425 (e.g., from distance feature projector 415b) to correct the initial spatial position of occlusion 205 (e.g., received from image feature projector 415a) to the spatial position output by the final transformer layer. Each execution of one of the transformer layers refines (i.e., further corrects) the spatial position.

[0047] Features from feature projector 415 can be subdivided into multiple embedded patches 460, which are then provided as input to transformer 410. For example, features can be subdivided into regions based on corresponding initial spatial locations. The size of the embedded patches 460 can be selected to facilitate fast processing time while covering sufficient area to allow identification of the type of occlusion 205. Embedded patches 460 can accelerate the execution of transformer 410 and preserve spatial data during the execution of transformer 410.

[0048] The first normalization layer 430 receives features and initial spatial locations from one of the feature projectors 415 (e.g., for image data 420 from image feature projector 415a) or the output from the second summing block 455 of the previous transformer layer. The first normalization layer 430 performs normalization. The term "normalization" is used in the statistical and machine learning sense for transforming data to conform to specific statistical properties. The first normalization layer 430 applies layer normalization across the received data and outputs the transformed data to the multi-head attention layer 435.

[0049] The multi-head attention layer 435 can use either cross-attention or self-attention. The multi-head self-attention layer (MHSA) applies the attention mechanism to the input multiple times in parallel. In simple terms, MHSA makes elements of the input aware of other contextual information from the input. The multi-head cross-attention layer (MHCA) uses a function of the input (i.e., normalized data from image feature projector 415a) as a query and a function of distance data 425 from distance feature projector 415b as a candidate pool. The candidate pool acts as the key and value of MHCA. MHCA applies the attention mechanism multiple times in parallel between the query and the candidate pool. In simple terms, MHCA gathers information from the distance data 425 to update features from image data 420.

[0050] The first summation block 440 calculates the sum of the output of the multi-head attention layer 435 and the input to the first normalization layer 430. The first summation block 440 represents a skip connection.

[0051] The second normalization layer 445 receives the output of the first summing block 440. The second normalization layer 445 performs normalization. The second normalization layer 445 applies layer normalization across the received data and outputs the transformed data to the MLP 450.

[0052] MLP 450 is a fully connected feedforward artificial neural network (ANN). MLP 450 receives the output of the second normalized layer 445 as input, and MLP 450 outputs an updated version with updated features of the spatial location.

[0053] The second summing block 455 calculates the sum of the output of MLP 450 and the output of the first summing block 440. The second summing block 455 represents a skip connection. Before executing the final transformer layer, the output of the second summing block 455 is used as input to the first normalized layer 430 to execute the next transformer layer. After executing the final transformer layer, the output of the second summing block 455 is the output of transformer 410.

[0054] Transformer 410 is trained to classify occlusion 205 into types and determine the spatial location of occlusion 205. Transformer 410 can be trained on training data including a feature stream from feature projector 415 with initial spatial locations paired with the ground truth classification and ground truth spatial location. For example, labels indicating the ground truth classification and / or ground truth spatial location can be manually encoded. The ground truth classification is selected from a predetermined set of latent types. Therefore, transformer 410 is not trained to classify objects because the objects are not included in the predetermined set of latent types. Transformer 410 can be trained using any suitable training algorithm, such as backpropagation applied to an objective function and optimization algorithms such as gradient descent. For example, the objective function could be L2 loss.

[0055] Computer 105 can be programmed to actuate component 130 based on the output of transformer 410. As an example, computer 105 can actuate component 130 of vehicle 100 based on the classification of obstruction 205. For example, computer 105 can actuate a windshield wiper in response to the classification being rain. As another example, computer 105 can actuate a climate control system to blow hot air across windshield 210 in response to the classification being snow or ice on windshield 210 in the environment. As another example, computer 105 can actuate an air nozzle aimed at imaging sensor 110 in response to the classification being an insect on imaging sensor 110. As another example, computer 105 can actuate a pump or valve controlling washer fluid to a fluid nozzle in response to the classification being dirt or dust on imaging sensor 110.

[0056] Alternatively or additionally, computer 105 may actuate components 130 of vehicle 100 based on the spatial position of obstruction 205. For example, computer 105 may select one nozzle or subset of nozzles to be actuated from a plurality of nozzles targeting imaging sensor 110 based on the spatial position of obstruction 205 on imaging sensor 110. For another example, computer 105 may actuate a windshield wiper at a slower or less frequent rate in response to obstruction 205 covering a portion of windshield 210, and at a faster or more frequent rate in response to obstruction 205 covering the entire windshield 210. For another example, computer 105 may actuate a windshield wiper once vehicle 100 is within a threshold distance of obstruction 205 in the environment.

[0057] Figure 5This is a flowchart illustrating an example process 500 for analyzing occlusion 205 and actuating component 130. The memory of computer 105 stores executable instructions for performing the steps of process 500, and / or can be programmed using structures such as those mentioned above. As a general overview of process 500, computer 105 receives sensor data 405 from imaging sensor 110, detects features in the sensor data 405, projects the features into a bird's-eye view embedding space 300, rasterizes distance data 425, fuses sensor data 405, classifies occlusion 205, and actuates component 130 of vehicle 100. Process 500 continues as long as vehicle 100 remains open.

[0058] Process 500 begins at box 505, where computer 105 receives sensor data 405 from imaging sensor 110, as described above.

[0059] Next, in box 510, computer 105 detects features in sensor data 405, as described above regarding feature projector 415.

[0060] Next, in box 515, computer 105 projects the feature onto an initial spatial location in the bird's-eye view embedding space 300, as described above regarding feature projector 415.

[0061] Next, in box 520, computer 105 rasterizes the distance data 425 from distance feature projector 415b, as described above.

[0062] Next, in box 525, computer 105 fuses sensor data 405 by executing transformer 410, as described above.

[0063] Next, in box 530, computer 105 classifies occlusion 205 by executing transformer 410, as described above.

[0064] Next, in box 535, computer 105 actuates component 130 of vehicle 100 based on at least one of the classification of occlusion 205 or the spatial location of occlusion 205, as described above.

[0065] Next, in decision box 540, computer 105 determines whether vehicle 100 is started (i.e., in the start state). For the purposes of this disclosure, "start state" is defined as a state in which all electrical power is supplied to the electrical components of vehicle 100 and vehicle 100 is ready to be driven (e.g., the engine is running); "off state" is defined as a state in which a small amount of electrical power is supplied to selected electrical components of vehicle 100 (typically used when vehicle 100 is stored); and "accessory energized state" is defined as a state in which all electrical power is supplied to more electrical components compared to the off state and vehicle 100 is not ready to be driven. Typically, the operator puts vehicle 100 in the start state when the operator is about to operate vehicle 100, puts vehicle 100 in the off state when the operator is about to leave vehicle 100, and puts vehicle 100 in the accessory energized state when the operator is about to sit in vehicle 100 but not operate vehicle 100. In response to vehicle 100 being in the start state, process 500 returns to block 505 to continue receiving sensor data 405. In response to vehicle 100 transitioning to the off state or accessory power-on state, process 500 ends.

[0066] Generally, the described computing system and / or device may employ any of a variety of computer operating systems, including, but not limited to, the following versions and / or types: Ford Sync® applications; AppLink / Smart Device Link middleware; Microsoft Automotive® operating system; Microsoft Windows® operating system; Unix operating system (e.g., Solaris® operating system released by Oracle Corporation of Redwood Coast, California); AIX UNIX operating system released by International Business Machines Corporation of Armonk, New York; Linux operating system; Mac OSX and iOS operating systems released by Apple Inc. of Cupertino, California; BlackBerry OS released by BlackBerry Ltd. of Waterloo, Canada; and Android operating system developed by Google and the Open Handset Alliance; or the QNX® CAR infotainment platform provided by QNX Software Systems. Examples of computing devices include, but are not limited to, in-vehicle computers, computer workstations, servers, desktop computers, laptops, notebook computers, or handheld computers, or any other computing system and / or device.

[0067] Computing devices typically include computer-executable instructions, which can be executed by one or more computing devices such as those listed above. Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages ​​and / or technologies, which, individually or in combination, include, but are not limited to, Java™, C, C++, Matlab, Simulink, Stateflow, Visual Basic, JavaScript, Python, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as the Java Virtual Machine, the Dalvik Virtual Machine, etc. Generally, a processor (e.g., a microprocessor) receives instructions (e.g., from memory, computer-readable media, etc.) and executes those instructions to perform one or more processes, including one or more processes described herein. Such instructions and other data can be stored and transferred using a variety of computer-readable media. Files in a computing device are typically collections of data stored on computer-readable media such as storage media, random access memory, etc.

[0068] Computer-readable media (also known as processor-readable media) include any non-transitory (e.g., tangible) medium that contributes to providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, and internals that constitute a system bus coupled to the computer's processor. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, flash EEPROM, any other memory chip or magnetic tape, or any other medium from which a computer can read.

[0069] The databases, data repositories, or other data stores described herein can include various mechanisms for storing, accessing / retrieving various types of data, including hierarchical databases, file sets in file systems, application databases in proprietary formats, relational database management systems (RDBMS), non-relational databases (NoSQL), graph databases (GDB), and so on. Each such data store is typically contained within a computing device employing a computer operating system such as those mentioned above, and is accessed via a network in any one or more of various ways. File systems can be accessed from the computer operating system and can include files stored in various formats. In addition to languages ​​used to create, store, edit, and execute the stored programs (such as PL / SQL as described above), RDBMS typically employs Structured Query Language (SQL).

[0070] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) stored on one or more computing devices (e.g., servers, personal computers, etc.) and on computer-readable media (e.g., disks, storage, etc.) associated therewith. Computer program products may include such instructions stored on computer-readable media for performing the functions described herein.

[0071] In the accompanying drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements may be changed. Regarding the media, processes, systems, methods, inspirations, etc., described herein, it should be understood that although the steps of such processes, etc., are described as occurring in a certain ordered order, such processes can be practiced by performing the steps in a different order than that described herein. It should also be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted.

[0072] This disclosure has been described in an illustrative manner, and it should be understood that the terminology used is intended to be descriptive in nature, not restrictive. The use of terms such as “in response to,” “after determining,” etc., indicates a causal relationship, not merely a temporal one. The adjectives “first” and “second” are used throughout this document as identifiers and are not intended to indicate importance, order, or quantity. In light of the foregoing teachings, numerous modifications and variations of this disclosure are possible, and this disclosure may be practiced in ways other than those specifically described.

[0073] According to the present invention, a computer is provided having a processor and a memory, the memory storing instructions executable by the processor to: detect features in sensor data from an imaging sensor; and perform a transformer on the features, the transformer transforming the features into a bird's-eye view embedding space, classifying occlusions from the features, and outputting the spatial location of the occlusions; wherein the transformer is trained to classify occlusions and not to classify objects.

[0074] According to an embodiment, the instructions further include instructions for actuating a component based on the spatial position of the occlusion.

[0075] According to an embodiment, the vehicle includes the imaging sensor and the component.

[0076] According to an embodiment, the instructions further include instructions for actuating the component based on the classification of the occlusion.

[0077] According to an embodiment, the imaging sensor is a camera, and the sensor data is image data.

[0078] According to an embodiment, the transformer is programmed to fuse the features from the image data with distance data from at least one distance sensor in the bird's-eye view embedding space.

[0079] According to an embodiment, the instructions further include instructions for performing the following operations: projecting the feature onto the initial spatial position of the occlusion in the bird's-eye view embedding space, and the transformer being programmed to correct the initial spatial position of the occlusion to the spatial position using the distance data.

[0080] According to an embodiment, the instructions further include instructions for performing the following operation: rasterizing the distance data before the transformer fuses the feature and the distance data.

[0081] According to an embodiment, the at least one distance sensor includes at least one of lidar or radar.

[0082] According to an embodiment, the at least one distance sensor includes a lidar and a radar.

[0083] According to an embodiment, the spatial location is specified in at least two horizontal dimensions.

[0084] According to an embodiment, the instructions for classifying the occlusion include instructions for selecting a type from a predetermined set of potential types.

[0085] According to an embodiment, the predetermined set of potential types includes at least two of rain, fog, dirt on the imaging sensor, or insects on the imaging sensor.

[0086] According to the present invention, a method includes: detecting features in sensor data from an imaging sensor; performing a transformer on the features, the transformer transforming the features into a bird's-eye view embedding space, classifying occlusions from the features, and outputting the spatial location of the occlusions; wherein the transformer is trained to classify occlusions and not trained to classify objects.

[0087] In one aspect of the invention, the method includes actuating a component based on the spatial position of the occlusion, wherein the vehicle includes the imaging sensor and the component.

[0088] In one aspect of the invention, the method includes actuating components based on the classification of the occlusion, wherein the vehicle includes the imaging sensor and the components.

[0089] In one aspect of the invention, the transformer is programmed to fuse the feature with distance data from at least one distance sensor in the bird's-eye view embedding space.

[0090] In one aspect of the invention, the transformer is programmed to correct the spatial position of the occlusion using the distance data.

[0091] In one aspect of the invention, the method includes rasterizing the distance data before the transformer fuses the features and the distance data.

[0092] In one aspect of the invention, classifying the occlusion includes selecting a type from a predetermined set of potential types, and the predetermined set of potential types includes at least two of rain, fog, dirt on the imaging sensor, or insects on the imaging sensor.

Claims

1. A method comprising: Detect features in sensor data from an imaging sensor; as well as A transformer is applied to the features, which transforms the features into a bird's-eye view embedding space, occlusions are classified from the features, and the spatial location of the occlusions is output. The transformer is trained to classify occlusions, but not to classify objects.

2. The method of claim 1, further comprising actuating the component based on the spatial position of the obstruction.

3. The method of claim 2, wherein the vehicle includes the imaging sensor and the component.

4. The method of claim 1, further comprising actuating the component based on the classification of the occlusion.

5. The method of claim 1, wherein the imaging sensor is a camera and the sensor data is image data.

6. The method of claim 5, wherein the transformer is programmed to fuse the features from the image data with distance data from at least one distance sensor in the bird's-eye view embedding space.

7. The method of claim 6, further comprising projecting the feature onto the initial spatial location of the occlusion in the bird's-eye view embedding space, wherein the transformer is programmed to correct the initial spatial location of the occlusion to the spatial location using the distance data.

8. The method of claim 6, further comprising rasterizing the distance data before fusing the feature and the distance data in the transformer.

9. The method of claim 6, wherein the at least one distance sensor comprises at least one of lidar or radar.

10. The method of claim 6, wherein the at least one distance sensor comprises a lidar and a radar.

11. The method of claim 1, wherein the spatial location is specified in at least two horizontal dimensions.

12. The method of claim 1, wherein classifying the occlusion includes selecting a type from a predetermined set of potential types.

13. The method of claim 12, wherein the predetermined set of potential types includes at least two of rain, fog, dirt on the imaging sensor, or insects on the imaging sensor.

14. A computer comprising a processor and a memory storing instructions executable by the processor to perform the method as claimed in any one of claims 1-13.

15. A vehicle comprising the computer as claimed in claim 14 and the imaging sensor.

Images