Vehicle reflective recognition

Abstract

The invention provides vehicle reflection identification. A computer includes a processor and a memory, and the memory stores instructions executable by the processor to: determine a time-varying value derived from a region of sensor data from an optical sensor of an autonomous vehicle; determining that the value over time matches a stored temporal pattern; upon determining that the value that varies over time matches the stored temporal pattern, classifying an object at the region as a surface that reflects a signal from a luminaire of the source vehicle; and actuating a component of the host vehicle based on the classification of the object at the region.

Description

Technical Field

[0001] This disclosure describes techniques for identifying surfaces that reflect light from luminaires on a source vehicle. Background Art

[0002] Object recognition can be used to identify the type or classification of objects that appear in image data. Computers can use conventional image recognition techniques such as convolutional neural networks to identify objects, which are programmed to accept images as input and output recognition. Convolutional neural networks include a series of layers, where each layer uses the previous layer as input. Each layer contains multiple neurons that receive data generated by a subset of neurons in the previous layer as input and generate outputs that are sent to neurons in the next layer. Types of layers include: convolutional layers, which calculate weights and dot products of input data for small areas; pooling layers, which perform downsampling operations along spatial dimensions; and fully connected layers, which are generated based on the outputs of all neurons in the previous layer. The last layer of the convolutional neural network generates a score for each potential type of the object, and the final output is the type with the highest score. The computer can store a set of potential types of objects, such as {vehicle, bicycle, pedestrian, deer, pole, ...}. Summary of the invention

[0003] The illuminator may emit light according to a stored temporal pattern, such as a visible light communication (VLC) signal. The term "source vehicle" is used to indicate the vehicle from which the light originates. The source vehicle may be a host vehicle, or may be a target vehicle, i.e., another vehicle other than the host vehicle. The host vehicle's computer is programmed to determine a time-varying value derived from a region of sensor data from an optical sensor of the host vehicle, determine that the time-varying value matches the stored temporal pattern, and when so determined, classify the object at the region as a surface reflecting a signal from the illuminator of the source vehicle. For example, the computer may have identified a potential target vehicle, and the region may be a light source on the potential target vehicle. If a value such as the intensity of the region matches the stored temporal pattern over time, the potential target vehicle is not actually a vehicle, but a reflection of a source vehicle (i.e., the host vehicle or a known target vehicle). The computer actuates a component of the host vehicle based on the classification. Thus, for the purpose of actuating the host vehicle, the computer may treat the reflection as a non-vehicle rather than a vehicle.

[0004] A computer includes a processor and a memory, and the memory stores instructions executable by the processor to: determine a time-varying value derived from a region of sensor data from an optical sensor of a host vehicle; determine that the time-varying value matches a stored temporal pattern; upon determining that the time-varying value matches the stored temporal pattern, classify an object at the region as a surface that reflects a signal from an illuminator of a source vehicle; and actuate a component of the host vehicle based on the classification of the object at the region.

[0005] In one example, the source vehicle may be a host vehicle, and the instructions may further include instructions for instructing the luminaires to illuminate according to a stored temporal pattern.

[0006] In one example, the stored temporal pattern may be a visible light communication (VLC) signal. In another example, the VLC signal may be a pulse-based transmission.

[0007] In one example, the instructions may further include instructions for identifying a potential vehicle in the sensor data, the region being part of the potential vehicle. In another example, the instructions may further include instructions for performing the following operations: upon determining that the time-varying value matches the stored time pattern, classifying the potential vehicle as a non-vehicle.

[0008] In yet another example, the instructions may further include instructions for classifying the potential vehicle as a vehicle upon determining that the time-varying value does not match a stored time pattern.

[0009] In yet another example, the instructions may further include instructions for identifying a light source of the potential vehicle, the region corresponding to the light source. In yet another example, the instructions may further include instructions for performing the following operations: upon determining that the value over time derived from a second region of the sensor data matches the stored temporal pattern, avoiding classifying the potential vehicle as a non-vehicle, the second region corresponding to a portion of the potential vehicle that is not the light source.

[0010] In one example, the instructions may further include instructions for determining a phase offset between the time-varying value and the stored time pattern. In another example, the instructions may further include instructions for determining a distance to the surface based on the phase offset.

[0011] In one example, the instructions may further include instructions for performing the following operations: determining that the value that varies over time includes multiple matches to the stored time pattern, and when determining that the value that varies over time includes multiple matches to the stored time pattern, filtering out one match to the stored time pattern from the value that varies over time. In another example, the instructions may further include instructions for performing the following operations: determining a phase offset of the match, and selecting one of the matches to be filtered out based on the phase offset.

[0012] In one example, the time varying value may be an intensity value.

[0013] In one example, the optical sensor may be a camera.

[0014] In one example, the source vehicle may be a host vehicle, and the illuminator may be incorporated into the optical sensor. In another example, the optical sensor may be one of a lidar or a radar.

[0015] In one example, the source vehicle may be the host vehicle, and the illuminator may be at least one of a headlight, a daytime running light, a taillight, or a turn signal light of the host vehicle.

[0016] In one example, the component of the host vehicle may include at least one of a braking system or a steering system.

[0017] A method includes determining a time-varying value derived from a region of sensor data from an optical sensor of a host vehicle; determining that the time-varying value matches a stored temporal pattern; upon determining that the time-varying value matches the stored temporal pattern, classifying an object at the region as a surface that reflects a signal from an illuminator of a source vehicle; and actuating a component of the host vehicle based on the classification of the object at the region. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram of an example host vehicle with sensors and lighting.

[0019] Figure 2A is an illustration of an exemplary scenario in which a sensor detects a reflection of a host vehicle.

[0020] Figure 2B is an illustration of an exemplary scenario in which a target vehicle is traveling beside a host vehicle.

[0021] Figure 2C is an illustration of an exemplary scenario in which a sensor of a host vehicle detects a surface that reflects light from a luminaire.

[0022] Figure 3 is a representation of exemplary data from sensors of a host vehicle.

[0023] Figure 4 is a flow chart of an exemplary process for operating a host vehicle based on detecting light emitted by a luminaire. DETAILED DESCRIPTION

[0024] Referring to the accompanying drawings, in which like numbers indicate like parts throughout the several views, a computer 105 includes a processor and a memory, and the memory stores instructions executable by the processor to: determine a time-varying value derived from an area 305 of sensor data 300 from an optical sensor 110 of a host vehicle 100; determine that the time-varying value matches a stored temporal pattern; upon determining that the time-varying value matches the stored temporal pattern, classify an object 205 at the area 305 as a surface 215 that reflects a signal from an illuminator 115 of a source vehicle; and actuate a component of the host vehicle 100 based on the classification of the object 205 at the area 305.

[0025] refer to Figure 1 , the host vehicle 100 may be any passenger vehicle or commercial vehicle, such as a car, truck, sport utility vehicle, crossover, van, minivan, jeep, taxi, bus, etc. The host vehicle 100 may include a computer 105, an optical sensor 110, an illuminator 115, a communication network 120, a propulsion system 125, a braking system 130, a steering system 135, and a user interface 140.

[0026] The computer 105 is a microprocessor-based computing device, such as a general-purpose computing device (which includes a processor and memory, an electronic controller, etc.), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a combination of the foregoing, etc. Typically, hardware description languages ​​such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) are used in electronic design automation to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, and the logic components inside the FPGA can be configured based on, for example, VHDL programming stored in a memory electrically connected to the FPGA circuit. Therefore, the computer 105 may include a processor, a memory, etc. The memory of the computer 105 may include a medium for storing instructions executable by the processor and for electronically storing data and / or a database, and / or the computer 105 may include a structure such as the foregoing structure that provides programming. The computer 105 may be a plurality of computers connected together.

[0027] The computer 105 can transmit and receive data through the communication network 120. The communication network 120 can be, for example, a controller area network (CAN) bus, Ethernet, WiFi, a local interconnect network (LIN), an on-board diagnostic connector (OBD-II), and / or any other wired or wireless communication network. The computer 105 can be communicatively coupled to the optical sensor 110, the illuminator 115, the propulsion system 125, the braking system 130, the steering system 135, the user interface 140, and other components via the communication network 120.

[0028] The optical sensor 110 can detect objects 205 and / or features of the outside world, such as other vehicles, road lane markings, traffic lights and / or signs, pedestrians, etc., of the surrounding environment of the host vehicle 100. The optical sensor 110 detects light, that is, electromagnetic radiation of a certain range of wavelengths. For example, the optical sensor 110 can be a camera capable of detecting visible light, infrared radiation, ultraviolet light, or a certain range of wavelengths including visible light, infrared light, and / or ultraviolet light. For example, the camera may be a complementary metal oxide semiconductor (CMOS), a charge coupled device (CCD), or any other suitable type. For another example, the camera may be a time of flight (TOF) camera that incorporates an illuminator 115 for illuminating the environment 200 and detects both reflected light and ambient light from the modulated illuminator 115 to sense reflectivity amplitude and distance to the scene. For another example, the camera may be an event camera whose pixels in an image frame are asynchronously updated in response to brightness changes. For another example, the optical sensor 110 may be a lidar, such as a scanning lidar device. Lidar detects the distance to an object 205 by emitting laser pulses of a specific wavelength from an illuminator 115 and measuring the flight time for the pulses to travel to the object 205 and back. The Lidar device may be of any suitable type for providing Lidar data that the computer 105 can act on, for example, a spindle-type Lidar, a solid-state Lidar, a flash Lidar, etc. As another example, the optical sensor 110 may be a radar. The radar includes an illuminator 115 that transmits radio waves, and the radar receives reflections of those radio waves to detect objects 205 in the environment 200. The radar may use direct propagation, i.e., measuring the time delay between the transmission and reception of the radio waves, and / or use indirect propagation, i.e., a frequency modulated continuous wave (FMCW) method, i.e., measuring the frequency change between the transmitted and received radio waves.

[0029] The illuminator 115 emits light of a wavelength that can be detected by the optical sensor 110. For example, as described above, the illuminator 115 can be incorporated into the optical sensor 110, such as into a TOF camera, a laser radar, or a radar. For another example, the illuminator 115 can be at least one of the headlights, daytime running lights, taillights, or turn signal lights of the main vehicle 100. The headlights can be fixed relative to the main vehicle 100 and arranged at the front of the main vehicle 100, facing the vehicle's forward direction. The taillights can be fixed relative to the main vehicle 100 and arranged at the rear of the main vehicle 100, facing the vehicle's rearward direction. The turn signal lights can be fixed relative to the main vehicle 100 and arranged at the corners of the main vehicle 100. One lamp or a group of lamps can operate as both taillights and turn signal lights. The illuminator 115 can be any lighting system suitable for illuminating the road ahead of the main vehicle 100 or other vehicles that are easily visible, including tungsten, halogen, high intensity discharge (HID) (such as xenon), light emitting diodes (LEDs), lasers, etc.

[0030] The computer 105 may be programmed to instruct the illuminator 115 to illuminate according to the stored time pattern. The stored time pattern may be a preset modulation sequence of the illuminator 115 over a certain period of time. The stored time pattern may be represented as a function of time, for example, I=f(t), where I is intensity and t is time. The function of time may have a more complex representation to accommodate the physical characteristics of the optical sensor 110, for example, exposure time, multiple exposure method, sub-pixel method, and row reading time. The function may encode a string or symbol (e.g., an identification number or some other string that uniquely identifies the host vehicle 100). For example, the stored time pattern may be a visible light communication (VLC) signal, which is the use of visible light as a transmission medium in telecommunications. The VLC signal may be of any suitable type for transmission by the illuminator 115 and detection by the optical sensor 110, such as a single carrier modulation transmission, such as by amplitude switching keying, phase switching keying, or frequency switching keying; a multi-carrier modulation transmission; or a pulse-based transmission, such as by on-off keying, variable pulse position modulation, or color shift keying. In particular, the VLC signal may be a pulse-based transmission, where data is encoded as a pulse wave, making it particularly suitable for the type of luminaires 115 used in vehicles. For a pulse-based transmission, the stored time pattern I=f(t) may have a binary output.

[0031] The propulsion system 125 of the host vehicle 100 generates energy and converts the energy into motion of the host vehicle 100. The propulsion system 125 can be a conventional vehicle propulsion subsystem, such as a conventional powertrain system, which includes an internal combustion engine coupled to a transmission that transfers rotational motion to the wheels; an electric powertrain system, which includes a battery, an electric motor, and a transmission that transfers rotational motion to the wheels; a hybrid powertrain system, which includes elements of a conventional powertrain system and an electric powertrain system; or any other type of propulsion device. The propulsion system 125 may include an electronic control unit (ECU) that communicates with and receives input from the computer 105 and / or a human operator, etc. The human operator can control the propulsion system 125 via, for example, an accelerator pedal and / or a gear shift lever.

[0032] The braking system 130 is typically a conventional vehicle braking subsystem and resists the motion of the host vehicle 100, thereby slowing and / or stopping the host vehicle 100. The braking system 130 may include friction brakes, such as disc brakes, drum brakes, band brakes, etc.; regenerative brakes; any other suitable type of brake; or a combination thereof. The braking system 130 may include an electronic control unit (ECU) or the like that communicates with and receives input from the computer 105 and / or a human operator. The human operator may control the braking system 130 via, for example, a brake pedal.

[0033] The steering system 135 is typically a conventional vehicle steering subsystem and controls the turning of the wheels. The steering system 135 can be a rack and pinion system with electric power steering, a steer-by-wire system (both of which are known), or any other suitable system. The steering system 135 may include an electronic control unit (ECU) or the like that communicates with and receives input from the computer 105 and / or a human operator. The human operator can control the steering system 135 via, for example, a steering wheel.

[0034] The user interface 140 presents information to and receives information from an operator of the host vehicle 100. The user interface 140 may be located, for example, on a dashboard in a passenger compartment of the host vehicle 100, or anywhere else that the operator can easily see it. The user interface 140 may include dials, digital readouts, screens, speakers, etc., such as known human-machine interface (HMI) elements, for providing information to the operator. The user interface 140 may include buttons, knobs, keypads, microphones, etc., for receiving information from the operator.

[0035] refer to FIG. 2A to FIG. 2C , the techniques herein may help the computer 105 determine the characteristics of the surrounding environment in different scenarios in order to navigate through the environment 200 . Figure 2AThe host vehicle 100 is shown traveling alongside a reflective surface 215 , such as a large window or set of windows, that is oriented to reflect an image of the host vehicle 100 toward the host vehicle 100 . Figure 2B The host vehicle 100 is shown traveling next to a target vehicle 225 that is similar in appearance to the host vehicle 100. The techniques herein can help the computer 105 distinguish between these two scenes by utilizing the stored temporal patterns. Similarly, the techniques herein can help the computer 105 determine whether what appears to be a target vehicle 225 is a different target vehicle 225 or a reflection of an already identified target vehicle 225. Figure 2C A host vehicle 100 is shown traveling beside an object 205, such as a wall that may reflect light from the illuminator 115. The reflected light may be direct, i.e., travel from the illuminator 115 to the object 205 and back to the optical sensor 110, or indirect, i.e., travel from the illuminator 115 to one or more other surfaces before reflecting from the object 205 back to the optical sensor 110. Techniques herein may use the stored temporal patterns to determine, for example, the distance to the object 205.

[0036] refer to Figure 3 , the computer 105 is programmed to receive sensor data 300 from the optical sensor 110. For example, the sensor data 300 may be image data. The image data is a series of image frames of the field of view of the optical sensor 110. Each image frame is a two-dimensional matrix of pixels. The brightness or color of each pixel is represented as one or more numerical values, for example, a scalar unitless value of photometric light intensity between 0 (black) and 1 (white), or a value of each of red, green, and blue, for example, each in an 8-bit scale (0 to 255) or a 12-bit or 16-bit scale. The pixels may be a mixture of representations, such as a repeating pattern of scalar values ​​of the intensity of three pixels and a fourth pixel having three numerical color values, or some other pattern. The position in the image frame (i.e., the position in the field of view of the sensor when the image frame was recorded) may be specified in pixel size or coordinates, for example, a pair of ordered pixel distances, such as a number of pixels from the top edge of the image frame and a number of pixels from the left edge of the image frame.

[0037] As another example, the sensor data 300 may be distance data. The distance data may be, for example, a point cloud. The points of the point cloud specify corresponding positions in the environment 200 relative to the position of the optical sensor 110. For example, the distance data may be in spherical coordinates, where the optical sensor 110 is located at the origin of the spherical coordinate system. The spherical coordinates may include: radial distance, i.e., the measured depth from the optical sensor 110 to the point measured by the ranging sensor; polar angle, i.e., the angle from the vertical axis passing through the optical sensor 110 to the point measured by the optical sensor 110; and azimuth angle, i.e., the angle in the horizontal plane from the horizontal axis passing through the optical sensor 110 to the point measured by the optical sensor 110. The horizontal axis may be, for example, along the vehicle's forward direction. Alternatively, the optical sensor 110 may return the point as a Cartesian coordinate of the optical sensor 110 at the origin or as a coordinate in any other suitable coordinate system, or the computer 105 may convert the spherical coordinates into Cartesian coordinates or another coordinate system after receiving the distance data.

[0038] The computer 105 may be programmed to divide the sensor data 300 into regions 305. For example, the computer 105 may be programmed to segment each image by executing a semantic segmentation algorithm, that is, to divide each image into regions 305. The semantic segmentation algorithm labels each pixel in the image with a category. The categories may be stored in the computer 105 as a list, such as {road, sidewalk, ground, tree, target vehicle 225, bicycle, pedestrian, animal, building / wall, traffic sign, traffic light, light source 220, sky...}. Each region 305 may be a continuous region of a category. The semantic segmentation algorithm may be a machine learning algorithm, such as a deep neural network, a convolutional neural network, etc. For another example, the computer 105 may execute an instance segmentation algorithm. The instance segmentation algorithm divides the image into different objects shown in the image, each of which may be a region 305. The instance segmentation algorithm may be a machine learning algorithm, such as a deep neural network, a convolutional neural network, etc. For another example, the computer 105 may execute a panoramic segmentation algorithm, which is a combination of semantic segmentation and instance segmentation.

[0039] The computer 105 is programmed to determine a time-varying value derived from the region 305 of the sensor data 300. The value is a quantity that can encode the stored time pattern. For example, the value can be an intensity value that can encode the stored time pattern in a pulse-based transmission by switching between a high value and a low value. For another example, the value can be a measure of one or more colors. As the value changes over time, the computer 105 tracks the value. In other words, the value that changes over time is time series data. As is commonly understood, and for the purposes of this disclosure, time series data is the value of a quantity at discrete, continuous time points.

[0040] The computer 105 may be programmed to store the time-varying values ​​in the memory of the computer 105 as a stored time pattern in response to the data indicating that the area 305 is the light source 220 of the target vehicle 225. The stored time pattern from the light source 220 may be a preset modulation sequence of the light source 220, such as the coded string described above for the illuminator 115. Alternatively, the stored time pattern may be a pattern of flickering characteristics of the light source 220, for example, produced by a pulse width modulation (PWM) used to activate the light source 220. For example, the data indicating that the area 305 is the light source 220 of the target vehicle 225 may be derived from other sensors of the host vehicle 100 besides the optical sensor 110, such as lidar data if the optical sensor 110 is a camera. For another example, the data indicating that the area 305 is the light source 220 of the target vehicle 225 may be a vehicle-to-vehicle (V2V) communication from the target vehicle 225 to the host vehicle 100. As an alternative to storing a time-varying value, the V2V communication from the target vehicle 225 may include a stored temporal pattern of the light source 220 of the target vehicle 225 , and the computer 105 may store the stored temporal pattern from the V2V communication in memory.

[0041] The computer 105 is programmed to determine whether the time-varying values ​​match a stored time pattern for each region 305 for which the computer 105 tracks the values. The stored time pattern may be a stored time pattern for illuminating the illuminators 115 of the host vehicle 100, as further described above, or the stored time pattern may be a stored time pattern of the light source 220 from the known target vehicle 225, as described just above. For example, the stored time pattern may be encoded as a string of characters, and the computer 105 may decode the time-varying values ​​into the string of characters and check whether the sequence is the same. When checking whether the sequence is the same, the computer 105 may also shift the stored time sequence along the decoded characters to detect a stored time pattern that has been phase-shifted due to the travel time from the illuminator 115 back to the optical sensor 110. For another example, the computer 105 may apply a matched filter to the time-varying values. The term "matched filter" is used in its signal processing sense, i.e., associating a known delayed signal or template with an unknown signal to detect the presence of the template in the unknown signal. The matched filter may be mathematically equivalent to convolving the time-varying values ​​from region 305 with a conjugate time-reversed version of the stored temporal pattern.

[0042] The computer 105 is programmed to classify the object 205 at the area 305 as the surface 215 that reflects the signal from the illuminator 115 upon determining that the time-varying value of the area 305 matches the stored time pattern. As described in more detail below, the computer 105 may classify the object 205 as a non-vehicle if the object 205 is a light source 220 of a potential vehicle 210. Also as described below, the computer 105 may determine a distance to the surface 215 in response to classifying the object 205 as the surface 215 that reflects the signal from the illuminator 115.

[0043] Computer 105 may be programmed to identify potential vehicles 210 in sensor data 300. Computer 105 may identify potential vehicles 210 using conventional image recognition techniques (e.g., convolutional neural networks) that are programmed to accept images as input and output the identified object. A convolutional neural network includes a series of layers, each of which uses the previous layer as input. Each layer contains a plurality of neurons that receive data generated by a subset of neurons in the previous layer as input and generate outputs that are sent to neurons in the next layer. Types of layers include: convolutional layers that calculate the dot product of weights and input data for a small area; pooling layers that perform downsampling operations along spatial dimensions; and fully connected layers that are generated based on the outputs of all neurons in the previous layer. The last layer of the convolutional neural network generates a score for each potential object type, and the final output is the type with the highest score. When the type "vehicle" or "potential vehicle" has the highest score, object 205 is a potential vehicle 210. Classification as a potential vehicle 210 may be classification as "vehicle" before the following steps occur.

[0044] The computer 105 may be programmed to identify the light source 220 of the potential vehicle 210. For example, the computer 105 may identify an area 305 of the sensor data 300 within the potential vehicle 210 where the brightness or contrast with the surrounding area 305 is above a threshold. The threshold may be selected to be lower than the brightness or contrast of a typical luminaire of the same type as the luminaire 115 and higher than the background brightness or typical contrast between the areas 305. The portion of the sensor data 300 corresponding to the light source 220 of the potential vehicle 210 may be the area 305 for which values ​​are tracked and compared to the stored temporal patterns, as described above.

[0045] The computer 105 may be programmed to classify the potential vehicle 210 as a surface 215 that reflects the signal from the illuminator 115 of the source vehicle, i.e., as a non-vehicle, upon determining that the time-varying value of the area 305 corresponding to the light source 220 matches the stored temporal pattern, e.g., as in Figure 2AFor example, if the signal emitted from the light source 220 of the potential vehicle 210 matches the stored temporal pattern emitted by the illuminator 115 of the host vehicle 100, the computer 105 determines that the potential vehicle 210 is actually a reflection of the host vehicle 100, such as Figure 2A as shown in Figure 2B For another example, if the signal emitted from the light source 220 of the potential vehicle 210 matches the stored time pattern emitted by the illuminator 115 of the already detected target vehicle 225, the computer 105 determines that the potential vehicle 210 is actually a reflection of the target vehicle 225, rather than a second target vehicle 225.

[0046] Computer 105 may be programmed to classify potential vehicle 210 as a vehicle upon determining that the time-varying values ​​of region 305 corresponding to light source 220 do not match a stored temporal pattern, e.g., as in Figure 2B Because the signal emitted from the light source 220 of the potential vehicle 210 does not match the stored temporal pattern emitted by the illuminator 115 of the source vehicle, the computer 105 determines that the potential vehicle 210 is not a reflection of the source vehicle, but a different target vehicle 225. The computer 105 can classify the potential vehicle 210 as a vehicle by changing the classification from "potential vehicle" to "vehicle" or by maintaining the classification "vehicle" after comparing the light source 220 to the stored temporal pattern.

[0047] The computer 105 can be programmed to avoid classifying the potential vehicle 210 as a non-vehicle when determining that the time-varying values ​​derived from the region 305 of the sensor data 300 that is different from the light source 220 match the stored time pattern. The region of interest 305 used for this determination can correspond to a portion of the potential vehicle 210 that is not the light source 220. Therefore, the classification of the potential vehicle 210 as a non-vehicle or a vehicle depends on whether the light source 220 matches the stored time pattern, rather than other parts of the potential vehicle 210 matching the stored time pattern. If the light source 220 of the potential vehicle 210 is not visible to the optical sensor 110, the computer 105 can classify the potential vehicle 210 as a vehicle. This step prevents erroneous classification as a non-vehicle based on the light from the illuminator 115 being reflected by the target vehicle 225.

[0048] The computer 105 can be programmed to determine, for each region 305 that matches a stored temporal pattern, a phase offset between the time-varying value for that region 305 and the stored temporal pattern. The term "phase offset" (also referred to as "phase shift") means that the time-varying value matches the stored temporal pattern if the stored temporal pattern is shifted in time by a certain fraction of the duration of the stored temporal pattern. The computer 105 can determine the difference between the time at which the illuminator 115 emits a reference point in the stored temporal pattern and the time at which the reference point arrives at light detected by the optical sensor 110.

[0049] Computer 105 may be programmed to determine that the time-varying value includes multiple matches to the stored temporal pattern. The time-varying value of region 305 may include multiple copies of the stored temporal pattern superimposed on each other, resulting in multiple matches. This may occur when light emitted by illuminator 115 follows multiple different paths with overlapping final branches terminating at optical sensor 110, for example, one path from illuminator 115 to object 205 and reflected directly from object 205 back to optical sensor 110, and another path from illuminator 115 to a surface then reflected to object 205 and ultimately reflected from object 205 back to optical sensor 110. For example, in Figure 2C In the scene of FIG. 1 , light emitted from illuminator 115 may be reflected from object 205 (a wall in this example) back to optical sensor 110, and light may also be reflected from the road to object 205 and then back to optical sensor 110. For example, computer 105 may apply a matched filter as described above, and the matched filter may return multiple matches.

[0050] The computer 105 may be programmed to filter one match to the stored temporal pattern from the time-varying values ​​upon determining that the stored temporal pattern matches a time-varying value from the region 305 that includes multiple matches to the stored temporal pattern. The computer 105 may determine the phase offset of the match in the manner described above, and the computer 105 may select one of the matches to be filtered out based on the phase offset. The computer 105 may select a copy of the stored temporal pattern to be filtered out that has a larger phase offset. For example, the computer 105 may add a signal to the time-varying values ​​that includes a negative of the stored temporal pattern with a phase offset equal to the phase offset of the match selected to be filtered out.

[0051] Computer 105 may be programmed to determine the distance to surface 215 based on the phase shift when classifying object 205 as surface 215 that reflects the signal from illuminator 115. For example, the distance may be based on the phase shift and the frequency of the stored temporal pattern, i.e., the number of times the stored temporal pattern is transmitted per unit time, for example, as shown in the following equation:

[0052]

[0053] Where L is the distance to the surface 215, c is the speed of light, is the phase offset, and f is the frequency of the stored temporal pattern. Computer 105 may use the phase offset of the copies of the stored temporal pattern that remain in the time varying value after filtering out any multiple copies.

[0054] The computer 105 may be programmed to actuate components of the host vehicle 100 based on the classification of the object 205 at the area 305 and / or based on the distance to the object 205 at the area 305. The components may include, for example, the propulsion system 125, the braking system 130, the steering system 135, the user interface 140, and / or other vehicle systems. In other words, various vehicle systems (including the propulsion system 125, the braking system 130, the steering system 135, and / or the user interface 140) may accept control inputs determined based on the object classification, and therefore may accept inputs based on the classification of the object 205 as described herein. For example, the computer 105 may actuate components when executing an advanced driver assistance system (ADAS). ADAS is an electronic technology that assists the driver in achieving driving functions and parking functions. Examples of ADAS include forward collision detection, lane departure detection, blind spot detection, automatic braking, adaptive cruise control, and lane keeping assistance. For example, the computer 105 may actuate the braking system 130 or the steering system 135 in response to the vehicle turning toward the path of the host vehicle 100, for example, according to an automatic braking algorithm, or the computer 105 may actuate the user interface 140 in response to the vehicle turning toward the path of the host vehicle 100, for example, according to a collision detection algorithm. If the potential vehicle 210 is classified as a non-vehicle as described above, the computer 105 will not determine the path of the vehicle turning toward the host vehicle 100. The computer 105 may autonomously operate the host vehicle 100 (i.e., actuate the propulsion system 125, the braking system 130, and the steering system 135 based on data from the optical sensor 110) to, for example, navigate around an object 205 detected by the optical sensor 110, for example, based on a distance determined from a phase offset. The computer 105 may actuate components based on the distance from the phase offset by combining the distance with other data from other sensors (e.g., using sensor fusion) and actuating components based on the combined (fused) data.

[0055] Figure 44 is a flow chart illustrating an exemplary process 400 for operating a host vehicle 100 based on detecting light emitted by an illuminator 115. The memory of the computer 105 stores executable instructions for performing the steps of the process 400, and / or can be programmed in a structure such as described above. As a general overview of the process 400, the computer 105 instructs the illuminator 115 to illuminate according to a stored temporal pattern, receives sensor data 300 from the optical sensor 110, identifies potential vehicles 210 from the sensor data 300, and identifies the light sources 220 of the potential vehicles 210. For each potential vehicle 210, the computer 105 classifies the potential vehicle 210 as a non-vehicle in response to the time-varying values ​​of the light source 220 matching the stored temporal pattern, otherwise classifies the potential vehicle 210 as a vehicle. For each region 305 in the sensor data 300 having a time-varying value matching the stored temporal pattern, the computer 105 classifies the object 205 in the region 305 as a surface 215, filters multiple copies of the stored temporal pattern from the time-varying values, and determines the distance to the surface 215. For each area 305 with a mismatched time-varying value, the computer 105 maintains the current classification. Finally, the computer 105 actuates components of the host vehicle 100 based on the classification and / or distance. The process 400 continues as long as the host vehicle 100 remains turned on.

[0056] Process 400 begins in block 405, where computer 105 instructs luminaires 115 to illuminate according to a stored temporal pattern, as described above.

[0057] Next, in block 410 , the computer 105 receives the sensor data 300 from the optical sensor 110 , as described above.

[0058] Next, in box 415, the computer 105 identifies potential vehicles 210 in the sensor data 300, as described above.

[0059] Next, in block 420 , the computer 105 identifies the light source 220 of the potential vehicle 210 , as described above.

[0060] Next, in decision block 425, the computer 105 determines whether the light sources 220 from all potential vehicles 210 have been compared to the stored temporal patterns. The computer 105 may store the potential vehicles 210 in a list and proceed sequentially through the list. If the computer 105 has not yet reached the last potential vehicle 210 on the list, the computer 105 selects the next potential vehicle 210 on the list, starting with the first potential vehicle 210 on the list, and proceeds to decision block 430. In response to the most recently selected potential vehicle 210 being the last potential vehicle 210 on the list, the process 400 proceeds to decision block 445.

[0061] In decision block 430, the computer 105 determines whether the time-varying values ​​of any of the light sources 220 of the selected potential vehicle 210 match a stored temporal pattern, as described above. When it is determined that the time-varying values ​​match the temporal pattern, the process 400 proceeds to block 435. When it is determined that the time-varying values ​​do not match the stored temporal pattern, the process 400 proceeds to block 440.

[0062] In block 435 , the computer 105 classifies the selected potential vehicle 210 as a non-vehicle, as described above. After block 435 , the process 400 returns to decision block 425 to proceed to the next potential vehicle 210 , if any.

[0063] In block 440 , the computer 105 classifies the selected potential vehicle 210 as a vehicle, as described above. After block 440 , the process 400 returns to decision block 425 to proceed to the next potential vehicle 210 , if any.

[0064] In decision block 445, the computer 105 determines whether all regions 305 in the sensor data 300 have been compared to the stored temporal patterns. The computer 105 may store the regions 305 in a list and proceed sequentially through the list. If the computer 105 has not yet reached the last region 305 on the list, the computer 105 selects the next region 305 on the list, starting with the first region 305 on the list, and proceeds to decision block 450. In response to the most recently selected region 305 being the last region 305 on the list, the process 400 proceeds to block 475.

[0065] In decision block 450, computer 105 determines whether the time-varying value of selected region 305 matches a stored temporal pattern, as described above. When it is determined that the time-varying value matches the stored temporal pattern, process 400 proceeds to block 455. When it is determined that the time-varying value does not match the stored temporal pattern, process 400 proceeds to block 470.

[0066] In block 455 , the computer 105 classifies the object 205 at the selected area 305 as a surface 215 that reflects signals from the illuminator 115 of the host vehicle 100 , as described above.

[0067] Next, in block 460 , the computer 105 filters out additional matches to the stored temporal patterns from the time-varying values, as described above.

[0068] Next, in block 465, computer 105 determines the distance to surface 215, as described above. After block 465, process 400 returns to decision block 445 to proceed to the next region 305, if any.

[0069] In block 470, the computer 105 avoids classifying the object 205 at the selected area 305 as a surface 215 that reflects signals from the illuminator 115 of the host vehicle 100, as described above. After block 470, the process 400 returns to decision block 445 to proceed to the next area 305, if any.

[0070] In block 475 , the computer 105 actuates components of the host vehicle 100 based on the classification of the object 205 in the sensor data 300 , as described above.

[0071] Next, in decision block 480, the computer 105 determines whether the host vehicle 100 is still on. In response to the host vehicle 100 being on, the process 400 returns to block 405 to continue monitoring the sensor data 300. In response to the host vehicle 100 being off, the process 400 ends.

[0072] Generally speaking, the computing systems and / or devices described may employ any of a variety of computer operating systems, including but not limited to the following versions and / or varieties: Ford Application; AppLink / Smart Device Link middleware; Microsoft Operating system; Microsoft Operating system; Unix operating system (for example, distributed by Oracle Corporation of Redwood Shores, California operating systems); AIX UNIX operating system distributed by International Business Machines Corporation of Armonk, New York; Linux operating system; Mac OSX and iOS operating systems distributed by Apple Inc. of Cupertino, California; BlackBerry operating system distributed by BlackBerry Ltd. of Waterloo, Canada; and Android operating system developed by Google Inc. and the Open Handset Alliance; or provided by QNX Software Systems CAR Infotainment Platform. Examples of computing devices include, but are not limited to, an in-vehicle computer, a computer workstation, a server, a desktop, notebook, laptop or handheld computer, or some other computing system and / or device.

[0073] Computing devices typically include computer-executable instructions, wherein the instructions are executable by one or more computing devices such as those listed above. Computer-executable instructions may be compiled or interpreted from a computer program created using a variety of programming languages ​​and / or technologies, including, alone or in combination, but not limited to Java. TM , C, C++, Visual HTML, etc. Some of these applications can be used in TM Virtual Machine, Dalvik Virtual Machine TM Typically, a processor (e.g., a microprocessor) receives instructions from, for example, a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using various computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0074] Computer-readable media (also referred to as processor-readable media) include any non-transitory (e.g., tangible) media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, including internals that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0075] The database, data repository or other data storage described herein may include various mechanisms for storing, accessing / accessing and retrieving various data, including hierarchical databases, file sets in file systems, application databases in special formats, relational database management systems (RDBMS), non-relational databases (NoSQL), graphic databases (GDB), etc. Each such data storage is typically included in a computing device using a computer operating system such as one of the above-mentioned, and is accessed via a network in any one or more of a variety of ways. The file system is accessible from the computer operating system and may include files stored in various formats. In addition to the language (such as the above-mentioned PL / SQL language) for creating, storing, editing and executing stored programs, RDBMS typically also uses structured query language (SQL).

[0076] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) stored on computer-readable media associated therewith (e.g., disks, memories, etc.). A computer program product may include such instructions stored on a computer-readable medium for performing the functions described herein.

[0077] In the accompanying drawings, like reference numerals indicate like elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes may be practiced by performing the steps in an order different from that described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0078] The present disclosure has been described in an illustrative manner, and it should be understood that the terms that have been used are intended to be descriptive in nature, rather than restrictive. The adjectives "first," "second," and the like are used throughout this document as identifiers, and are not intended to indicate importance, order, or quantity. The use of "in response to" and "after determining..." indicates a causal relationship, not just a temporal relationship. In view of the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure may be practiced in other ways than those specifically described.

[0079] According to the present invention, a computer is provided, the computer having a processor and a memory, the memory storing instructions executable by the processor to: determine a time-varying value derived from an area of ​​sensor data from an optical sensor of a host vehicle; determine that the time-varying value matches a stored temporal pattern; upon determining that the time-varying value matches the stored temporal pattern, classify an object at the area as a surface reflecting a signal from an illuminator of a source vehicle; and actuate a component of the host vehicle based on the classification of the object at the area.

[0080] According to an embodiment, the source vehicle is a host vehicle and the instructions further comprise instructions for instructing the luminaires to illuminate according to a stored time pattern.

[0081] According to an embodiment, the stored temporal pattern is a Visible Light Communication (VLC) signal.

[0082] According to an embodiment, the VLC signal is a pulse-based transmission.

[0083] According to an embodiment, the instructions further comprise instructions for identifying a potential vehicle in the sensor data, the area being part of the potential vehicle.

[0084] According to an embodiment, the instructions further include instructions for classifying the potential vehicle as a non-vehicle upon determining that the time-varying value matches a stored time pattern.

[0085] According to an embodiment, the instructions further include instructions for classifying the potential vehicle as a vehicle upon determining that the time-varying value does not match a stored time pattern.

[0086] According to an embodiment, the instructions further comprise instructions for identifying a light source of the potential vehicle, the area corresponding to the light source.

[0087] According to an embodiment, the instructions also include instructions for performing the following operations: avoiding classifying the potential vehicle as a non-vehicle when determining that the value varying over time derived from a second area of ​​the sensor data matches the stored time pattern, and the second area corresponds to the potential vehicle that is not a part of the light source.

[0088] According to an embodiment, the instructions further comprise instructions for determining a phase offset between the time varying value and the stored time pattern.

[0089] According to an embodiment, the instructions further comprise instructions for determining a distance to the surface based on the phase offset.

[0090] According to an embodiment, the instructions also include instructions for performing the following operations: determining that the value that changes over time includes multiple matches to the stored time pattern, and when determining that the value that changes over time includes multiple matches to the stored time pattern, filtering out one match to the stored time pattern from the value that changes over time.

[0091] According to an embodiment, the instructions further include instructions for determining a phase offset of the matches, and selecting one of the matches to filter out based on the phase offset.

[0092] According to an embodiment, the value varying over time is an intensity value.

[0093] According to an embodiment, the optical sensor is a camera.

[0094] According to an embodiment, the source vehicle is the host vehicle and the illuminator is incorporated into the optical sensor.

[0095] According to an embodiment, the optical sensor is one of a lidar or a radar.

[0096] According to an embodiment, the source vehicle is the host vehicle, and the illuminator is at least one of a headlamp, a daytime running light, a tail light, or a turn signal light of the host vehicle.

[0097] According to an embodiment, the component of the host vehicle comprises at least one of a braking system or a steering system.

[0098] In one aspect of the invention, the method includes determining a time-varying value derived from an area of ​​sensor data from an optical sensor of a host vehicle; determining that the time-varying value matches a stored temporal pattern; upon determining that the time-varying value matches the stored temporal pattern, classifying an object at the area as a surface that reflects a signal from an illuminator of a source vehicle; and actuating a component of the host vehicle based on the classification of the object at the area.

Claims

1. A method comprising: determining a time-varying value derived from a region of sensor data from an optical sensor of a host vehicle; determining that the value over time matches a stored time pattern; upon determining that the value varying over time matches the stored temporal pattern, classifying the object at the area as a surface reflecting a signal from a luminaire of a source vehicle; as well as A component of the host vehicle is actuated based on the classification of the object at the area.

2. The method of claim 1, wherein the source vehicle is the host vehicle, the method further comprising instructing the luminaire to illuminate according to the stored time pattern.

3. The method of claim 1, wherein the stored temporal pattern is a visible light communication (VLC) signal. 4 . The method of claim 1 , further comprising identifying a potential vehicle in the sensor data, the region being a portion of the potential vehicle. 5 . The method of claim 4 , further comprising classifying the potential vehicle as a non-vehicle upon determining that the value varying over time matches the stored temporal pattern.

6. The method of claim 4 further comprising classifying the potential vehicle as a vehicle upon determining that the value over time does not match the stored temporal pattern.

7. The method of claim 4, further comprising identifying a light source of the potential vehicle, the region corresponding to the light source.

8. The method of claim 7 further comprising avoiding classifying the potential vehicle as a non-vehicle upon determining that the time-varying value derived from a second region of the sensor data matches the stored temporal pattern, the second region corresponding to a portion of the potential vehicle that is not the light source.

9. The method of claim 1 further comprising determining a phase offset between the value varying over time and the stored temporal pattern, and determining a distance to the surface based on the phase offset.

10. The method of claim 1 further comprises determining that the value that varies over time includes multiple matches to the stored time pattern, and when determining that the value that varies over time includes multiple matches to the stored time pattern, filtering out one match to the stored time pattern from the value that varies over time. The method of claim 1 , wherein the value that varies over time is an intensity value.

12. The method of claim 1, wherein the source vehicle is the host vehicle and the illuminator is incorporated into the optical sensor.

13. The method of claim 1, wherein the source vehicle is the host vehicle and the illuminator is at least one of a headlight, a daytime running light, a taillight, or a turn signal light of the host vehicle.

14. The method of claim 1, wherein the component of the host vehicle includes at least one of a braking system or a steering system.

15. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method of one of claims 1 to 14.

Images