A method and system for removing specular overflow from vehicle sensors

Abstract

This disclosure provides "highlight clipping removal from vehicle sensors". A system includes: a processor; and a memory including instructions executed by the processor to perform the following actions: acquiring a first image of a scene; acquiring a second image of the scene while illuminating the scene; identifying pixel highlight clipping in a difference image determined by subtracting the first image from the second image; repairing the pixel highlight clipping based on empirically determined parameters; and operating a vehicle based on the difference image.

Description

Technical Field

[0001] This invention relates to a method and system for removing highlight overflow from vehicle sensors. Background Technology

[0002] Vehicles can be equipped to operate in both autonomous and occupant-guided modes. Vehicles can be equipped with computing devices, networks, sensors, and controllers to acquire information about the vehicle environment and operate the vehicle based on that information. Safe and comfortable vehicle operation can depend on acquiring accurate and timely information about the vehicle environment. Vehicle sensors can provide data about routes and objects to avoid in the vehicle environment. Safe and efficient vehicle operation can depend on acquiring accurate and timely information about routes and objects in the vehicle environment when the vehicle is operating on a road. For example, vehicle operation can depend on acquiring images from the vehicle's optical sensors (e.g., cameras) and acting on those images. However, accurately interpreting image data under varying lighting conditions is a problem. For example, digital images acquired under varying lighting conditions may include so-called "bloom." Removing or repairing bloom to make digital image data usable for vehicle operation is a problem. Summary of the Invention

[0003] One method includes: acquiring a first image of a scene; acquiring a second image of the scene while illuminating the scene; identifying pixel highlight overflow in a difference image determined by subtracting the first image from the second image; repairing the pixel highlight overflow based on empirically predetermined parameters; and operating a vehicle based on the difference image.

[0004] The method may also include acquiring a first image using an IR video sensor.

[0005] The method may further include: acquiring a second image using an IR video sensor while illuminating the scene with IR light.

[0006] Parameters that can be predetermined based on experience may include thresholds and bounding box overlap rates.

[0007] Repairing pixel highlight clipping can include removing and / or reducing pixel highlight clipping based on a threshold and bounding box overlap rate.

[0008] The method may further include: determining a set of highlight overflow pixels based on processing the difference image to identify the connection regions of saturated pixels and the connection regions of background pixels.

[0009] Removing pixel highlight clipping can include setting the pixel values ​​of a set of saturated pixels to their original values.

[0010] Reducing pixel highlight clipping can include reducing the pixel intensity value of the group of highlight clipped pixels based on distance measurements.

[0011] Distance measurement can include determining the L2 norm function based on the centroid of saturated pixels and determining the aspect ratio based on the connected regions of saturated pixels.

[0012] A system includes a processor and a memory. The memory includes instructions executed by the processor to perform the following actions: acquiring a first image of a scene; acquiring a second image of the scene while illuminating the scene; identifying pixel highlight overflow in a difference image determined by subtracting the first image from the second image; repairing the pixel highlight overflow based on empirically predetermined parameters; and operating a vehicle based on the difference image.

[0013] The system may also include acquiring a first image using an IR video sensor.

[0014] The system may further include: acquiring a second image using an IR video sensor while illuminating the scene with IR light.

[0015] Parameters determined empirically can include thresholds and bounding box overlap rates.

[0016] Repairing pixel highlight clipping can include removing or reducing pixel highlight clipping based on a threshold and bounding box overlap rate.

[0017] The system may further include: determining a set of highlight overflow pixels based on processing the difference image to identify the connection regions of saturated pixels and the connection regions of background pixels.

[0018] Removing pixel highlight clipping can include setting the pixel values ​​of a set of saturated pixels to their original values.

[0019] Reducing pixel highlight clipping can include reducing the pixel intensity value of the group of highlight clipped pixels based on distance measurements.

[0020] Distance measurement can include determining the L2 norm function based on the centroid of saturated pixels and determining the aspect ratio based on the connected regions of saturated pixels.

[0021] A system includes means for acquiring images; means for controlling vehicle steering, braking, and powertrain systems. The system includes a computer device for acquiring a first image of a scene; acquiring a second image of the scene while illuminating it; identifying pixel highlight clipping in a difference image determined by subtracting the first image from the second image; repairing the pixel highlight clipping based on empirically determined parameters; and operating the vehicle based on the difference image via the means for controlling the vehicle steering, braking, and powertrain systems.

[0022] Parameters determined empirically can include thresholds and bounding box overlap rates. Attached Figure Description

[0023] Figure 1 This is a block diagram of an exemplary vehicle.

[0024] Figure 2 This is an illustration of an exemplary image of a scene illuminated by an active lighting camera.

[0025] Figure 3 This is an illustration of an exemplary image that includes bounding boxes around both non-speculiarly overflowing pixels and specularly overflowing pixels.

[0026] Figure 4 This is an illustration of an exemplary image that has been processed to remove only the pixels with overflowing highlights.

[0027] Figure 5 This is an illustration of an exemplary image that has been processed to determine the connection area.

[0028] Figure 6 This is an illustration of an exemplary image that has been processed to determine non-highlight overflow pixels.

[0029] Figure 7 These are illustrations of two exemplary images processed to determine non-highlight overflow pixels.

[0030] Figure 8 This is an illustration of an exemplary image that has been processed to fix highlight clipping.

[0031] Figure 9 This is an illustration of an exemplary L2 norm for adjusting the aspect ratio of a highlight-blown object in an image.

[0032] Figure 10 Is using Figure 9 Illustrations of two exemplary images before and after specular clipping repair using the L2 norm.

[0033] Figure 11 This is a flowchart of an exemplary process for operating a vehicle based on repairing pixel highlight overflow in an image.

[0034] Figure 12 This is a flowchart of an exemplary process for repairing pixel highlight overflow in an image. Detailed Implementation

[0035] The vehicle can be equipped to operate in both autonomous and occupant-guided modes. Semi-autonomous or fully autonomous modes refer to operating modes in which the vehicle can be guided by a computing device as part of a vehicle information system with sensors and controllers. The vehicle may be occupied or unoccupied, but in either case, the vehicle can be guided without occupant assistance. For the purposes of this disclosure, autonomous mode is defined as follows: each of vehicle propulsion (e.g., via a powertrain including an internal combustion engine and / or an electric motor), braking, and steering is controlled by one or more vehicle computers; in semi-autonomous mode, the vehicle computer controls one or more of vehicle propulsion, braking, and steering. In non-autonomous vehicles, none of these are controlled by a computer.

[0036] For example, a computing device in a vehicle can be programmed to acquire data about the vehicle's external environment and use that data to determine a trajectory to maneuver the vehicle from its current location to a destination location. The data may include images acquired from sensors included in the vehicle while illuminating a scene visible to the sensors using electromagnetic radiation. These active lighting sensors include RADAR, LIDAR, and video sensors (including those for visible and infrared (IR) light). Acquiring data from active lighting sensors while illuminating a scene can provide data that allows the computing device to operate the vehicle under varying environmental conditions, including, for example, at night or other low-light conditions.

[0037] This paper discloses a method comprising: acquiring a first image of a scene without active illumination; acquiring a second image of the scene while illuminating the scene; identifying pixel specular clipping in a difference image determined by subtracting the first image from the second image; repairing the pixel specular clipping based on empirically predetermined parameters; and operating a vehicle based on the difference image. The first image can be acquired using an IR video sensor, and the second image can be acquired using the IR video sensor while illuminating the scene with IR light. Empirically predetermined parameters may include a threshold and a bounding box overlap rate. Repairing pixel specular clipping may include substantially completely removing or reducing pixel specular clipping based on the threshold and the bounding box overlap rate. Determining a set of clipped pixel specular ...

[0038] Removing pixel highlight clipping may include setting the pixel values ​​of a set of saturated pixels to their original values, and reducing pixel highlight clipping may include reducing the pixel intensity values ​​of the set of clipped pixels based on distance measurements. Distance measurements may include determining an L2 norm function based on the centroid of the saturated pixels. The L2 norm function may be determined based on the logarithm of the row and column distances from the centroid of the saturated pixels. The L2 norm function may be normalized by squaring the result of dividing the distance by the maximum distance. The L2 norm function may be modified to take into account the aspect ratio of the clipped object, such that highlight clipping reduction is specifically adjusted for the shape of the object. After reducing pixel highlight clipping, a second bounding box ratio may be determined, and whether to remove pixel highlight clipping may be determined based on the second bounding box ratio. Processing the difference image to identify connected regions of saturated pixels and connected regions of background pixels may include: determining an extended highlight clipping mask image based on a threshold applied to the difference image. Processing the difference image to identify connected regions of saturated pixels and connected regions of background pixels includes: determining a filtered extended highlight clipping mask image based on the extended highlight clipping mask image and the background mask image.

[0039] A computer-readable medium is also disclosed, storing program instructions for performing some or all of the above-described method steps. A computer is also disclosed, programmed to perform some or all of the above-described method steps, the computer including a computer device programmed to acquire a first image of a scene; acquire a second image of the scene while illuminating the scene; identify pixel highlight overflow in a difference image determined by subtracting the first image from the second image; repair the pixel highlight overflow based on empirically predetermined parameters; and operate a vehicle based on the difference image. The first image can be acquired using an IR video sensor, and the second image can be acquired using an IR video sensor while illuminating the scene with IR light. Empirically predetermined parameters may include a threshold and a bounding box overlap rate. Repairing pixel highlight overflow may include removing or reducing pixel highlight overflow based on the threshold and the bounding box overlap rate. Determining a set of highlight overflow pixels may be based on processing the difference image to identify connected regions of saturated pixels and connected regions of background pixels.

[0040] Removing pixel highlight clipping may include setting the pixel values ​​of a set of saturated pixels to their original values, and reducing pixel highlight clipping may include reducing the pixel intensity values ​​of the set of clipped pixels based on distance measurements. Distance measurements may include determining an L2 norm function based on the centroid of the saturated pixels. The L2 norm function may be determined based on the logarithm of the row and column distances from the centroid of the saturated pixels. The L2 norm function may be normalized by squaring the result of dividing the distance by the maximum distance. The L2 norm function may be modified to take into account the aspect ratio of the clipped object, such that highlight clipping reduction is specifically adjusted for the shape of the object. After reducing pixel highlight clipping, a second bounding box ratio may be determined, and whether to remove pixel highlight clipping may be determined based on the second bounding box ratio. Processing the difference image to identify connected regions of saturated pixels and connected regions of background pixels may include: determining an extended highlight clipping mask image based on a threshold applied to the difference image. Processing the difference image to identify connected regions of saturated pixels and connected regions of background pixels includes: determining a filtered extended highlight clipping mask image based on the extended highlight clipping mask image and the background mask image.

[0041] Figure 1 This is an illustration of a vehicle information system 100, which includes a vehicle 110 capable of operating in autonomous (“autonomous” hereinforcingly means “fully autonomous”) and occupant-guided (also referred to as non-autonomous) modes. The vehicle 110 also includes one or more computing devices 115 for performing calculations to guide the vehicle 110 during autonomous operation. The computing devices 115 can receive information about the operation of the vehicle from sensors 116. The computing devices 115 can operate the vehicle 110 in autonomous, semi-autonomous, or non-autonomous modes. For the purposes of this disclosure, autonomous mode is defined as a mode in which the computing devices control each of the propulsion, braking, and steering of the vehicle 110; in semi-autonomous mode, the computing devices 115 control one or both of the propulsion, braking, and steering of the vehicle 110; and in non-autonomous mode, a human operator controls the propulsion, braking, and steering of the vehicle.

[0042] The computing device 115 includes, for example, a processor and memory as known. Additionally, the memory includes one or more forms of computer-readable medium and stores instructions that can be executed by the processor to perform various operations as disclosed herein. For example, the computing device 115 may include programming to operate one or more of the following: vehicle braking, propulsion (e.g., controlling the acceleration of vehicle 110 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior and / or exterior lights, etc., and to determine whether and when the computing device 115, rather than a human operator, controls such operations.

[0043] The computing device 115 may include one or more computing devices (e.g., controllers in the vehicle 110 for monitoring and / or controlling various vehicle components, such as powertrain controller 112, brake controller 113, steering controller 114, etc.) or may be coupled to the one or more computing devices, for example, via a vehicle communication bus further described below. The computing device 115 is typically arranged to communicate via a vehicle communication network (e.g., including buses in the vehicle 110, such as a controller area network (CAN), etc.); the vehicle 110 network may additionally or alternatively include known wired or wireless communication mechanisms, such as Ethernet or other communication protocols.

[0044] Via the vehicle network, computing device 115 can transmit messages to and / or receive messages from various devices in the vehicle (e.g., controllers, actuators, sensors (including sensor 116)). Alternatively or additionally, where computing device 115 actually comprises multiple devices, the vehicle communication network can be used for communication between the devices represented as computing device 115 in this disclosure. Furthermore, as mentioned above, various controllers or sensing elements (such as sensor 116) can provide data to computing device 115 via the vehicle communication network.

[0045] Additionally, computing device 115 can be configured to communicate with remote server computer 120 (e.g., a cloud server) via network 130 through vehicle-to-infrastructure (V-to-I) interface 111, as described below. Interface 111 includes hardware, firmware, and software that allows computing device 115 to communicate with remote server computer 120 via network 130 such as Wi-Fi or cellular networks. Therefore, V-to-I interface 111 may include processors, memory, transceivers, etc., configured to utilize various wired and / or wireless networking technologies, such as cellular, Bluetooth®, and wired and / or wireless packet networks. Computing device 115 can be configured to communicate with other vehicles 110 via V-to-I interface 111 using vehicle-to-vehicle (V-to-V) networks (e.g., according to Dedicated Short Range Communication (DSRC) and / or similar communications) formed between nearby vehicles 110 on a mobile ad hoc network basis or via infrastructure-based networks. Computing device 115 also includes known non-volatile memory. The computing device 115 can record information for later retrieval by storing it in non-volatile memory and transmit it to the server computer 120 or the user mobile device 160 via the vehicle communication network and the vehicle-to-infrastructure (V-to-I) interface 111.

[0046] As mentioned above, instructions typically included in memory and executable by the processor of computing device 115 are programmed to operate (e.g., brake, steer, propulsion, etc.) one or more vehicle 110 components without human intervention. Using data received in computing device 115 (e.g., sensor data from sensor 116, server computer 120, etc.), computing device 115 can make various determinations and / or control various vehicle 110 components and / or operations without driver operation of vehicle 110. For example, computing device 115 may include programming to regulate vehicle 110 operating behavior (i.e., the physical manifestation of vehicle 110 operation), such as speed, acceleration, deceleration, steering, etc., and strategic behavior (i.e., operational behavior control generally performed in a manner intended to achieve safe and efficient route travel), such as distance and / or time between vehicles, lane changes, minimum clearance between vehicles, minimum left-turn crossing distance, arrival time at a specific location, and minimum arrival time at an intersection (without indicator lights) to pass through an intersection.

[0047] A controller (as used herein) includes a computing device typically programmed to control a particular vehicle subsystem. Examples include a powertrain controller 112, a brake controller 113, and a steering controller 114. A controller may be an electronic control unit (ECU), such as those known, which may include additional programming as described herein. A controller may be communicatively connected to a computing device 115 and receive instructions from the computing device 115 to actuate subsystems according to those instructions. For example, the brake controller 113 may receive instructions from the computing device 115 to operate the brakes of the vehicle 110.

[0048] One or more controllers 112, 113, 114 of vehicle 110 may include known electronic control units (ECUs) or similar units. As a non-limiting example, the known electronic control units (ECUs) or similar units include one or more powertrain controllers 112, one or more brake controllers 113, and one or more steering controllers 114. Each of controllers 112, 113, 114 may include a corresponding processor and memory, as well as one or more actuators. Controllers 112, 113, 114 may be programmed and connected to a vehicle 110 communication bus (such as a Controller Area Network (CAN) bus or a Local Interconnect Network (LIN) bus) to receive instructions from computer 115 and control the actuators based on those instructions.

[0049] Sensor 116 may include various known devices to provide data via a vehicle communication bus. For example, a radar fixed to the front bumper (not shown) of vehicle 110 may provide the distance from vehicle 110 to the next vehicle in front of vehicle 110, and a Global Positioning System (GPS) sensor located in vehicle 110 may provide the geographic coordinates of vehicle 110. The distances provided by radar and / or other sensors 116 and / or the geographic coordinates provided by GPS sensors may be used by computing device 115 to operate vehicle 110 autonomously or semi-autonomously.

[0050] Vehicle 110 is typically a ground-based autonomous vehicle 110 with three or more wheels (e.g., a bus, light truck, etc.). Vehicle 110 includes one or more sensors 116, a V-to-I interface 111, a computing device 115, and one or more controllers 112, 113, 114. Sensor 116 can collect data related to vehicle 110 and its operating environment. By way of example, but not limitation, sensor 116 may include, for example, a height gauge, a camera, a lidar (LIDAR), radar, an ultrasonic sensor, an infrared sensor, a pressure sensor, an accelerometer, a gyroscope, a temperature sensor, a pressure sensor, a Hall sensor, an optical sensor, a voltage sensor, a current sensor, a mechanical sensor (such as a switch), etc. Sensor 116 can be used to sense the operating environment of vehicle 110; for example, sensor 116 can detect phenomena such as weather conditions (rain, outside temperature, etc.), road slope, road position (e.g., using road edges, lane markings, etc.) or the position of target objects (such as nearby vehicles 110). Sensor 116 can also be used to collect data, including dynamic vehicle 110 data related to the operation of vehicle 110 (such as speed, yaw rate, steering angle, engine speed, brake pressure, oil pressure, power levels of controllers 112, 113, 114 applied to vehicle 110, connectivity between components, and the accuracy and timeliness of components of vehicle 110).

[0051] Figure 2This is an illustration of an exemplary image 200 of an illuminated scene. Image 200 is a pseudo-color image rendered in black and white to comply with patent office rules. The pseudo-color image 200 is a two-dimensional array of data values ​​or "pixels," where each pixel is represented by a single numerical value, and the pixels are displayed by assigning a color based on the numerical value of each pixel. The relationship between pixel values ​​and colors can be represented by a color scale 202, which depicts an intensity represented in arbitrary units from 1500 to 5000 as the color represented here by halftone rendering in black and white. In this example, the scene is illuminated using IR light, and image 200 is acquired by IR video sensor 116 and transmitted to computing device 115. The scene in image 200 can be illuminated by an IR light source. The IR light source can include an IR light-emitting diode (LED), which emits IR light, for example, in response to an electrical signal from computing device 115. While computing device 115 instructs the IR LED to emit IR light, computing device 115 can instruct IR video sensor 116 to acquire IR video image 200 of the illuminated scene. Because the pixel data included in image 200 is based on active lighting, maximizing the amount of data can include increasing the amount of IR light radiated onto the scene to increase the amount of data acquired. In this example, the maximum projected power on the surface of the scene (in watts / cm²) 2 (Measured in units) can be limited by saturation in the IR video sensor 116, which receives energy reflected from surfaces in the scene.

[0052] Maximizing the projected power of IR illumination can provide image 200 data including background data 206, 208. A problem may arise from saturation reflection 210 from retroreflective surfaces in the scene (such as road signs, construction markings, lane markings, traffic lights, etc.), because at the IR illumination power allowing imaging of background data 206, 208, saturation reflection 210 can saturate the IR video sensor 116 and cause pixel highlight overflow. A retroreflective surface is a surface coated with a film (plastic) or paint containing microscopic glass beads that reflect a high percentage of light falling on it directly back in the direction the light was received; hence, it is called a retroreflector. On standard CCD or CMOS-based sensors, pixel highlight overflow is caused by excess charge generated by saturated sensor pixels overflowing into adjacent pixels on the sensor. Pixel highlight overflow can render part or all of the pseudo-color image 200 unusable.

[0053] Pixel highlight clipping caused by regressive reflective surfaces in image 200 can be corrected by removing or reducing pixel highlight clipping based on the separation between saturation return 210 and background returns 206, 208. Processing image 200 in this way removes parasitic noise and makes image 200 more readily processable by computing device 115 for tasks related to the operation of vehicle 110, such as machine learning networks, including, for example, object detection and classification algorithms. When the pixels associated with saturation return 210 and the pixels associated with background returns 206, 208 are sufficiently separated, pixels affected by pixel highlight clipping can be removed substantially completely from image 200. The following section discusses… Figures 4 to 8 This section discusses the removal of pixel specular clipping. When the saturation return 210 is not sufficiently separated from pixels associated with background returns 206 and 208, removing pixels affected by specular clipping may result in the removal of pixels associated with background returns 206 and 208, which is undesirable. In this case, the impact of pixel specular clipping can be reduced by applying a computed distance-based intensity reduction matrix to decrease pixel specular clipping. The following section discusses... Figure 9 and Figure 10 The intensity reduction matrix is ​​discussed. After the step of reducing pixel highlight clipping, image 200 can be examined again to see if the new saturation return 210 no longer overlaps with the background returns 206, 208, and can now be removed almost completely. In both cases, the original intensity information from the regressive reflector that caused the saturation return 210 is included in the processed image 200.

[0054] Processing image 200 to repair pixel highlight overflow begins by acquiring image 200 without illuminating the scene. This provides a reference image 200, which can be subtracted from a subsequent image 200 acquired while illuminating the scene with IR light. This removes background illumination and pixels with intensity values ​​below the sensor noise level. After background subtraction, bounding boxes 212 can be determined based on saturation return 210. Computational device 115 can threshold image 200 to determine a binary image comprising eight-way connected regions, each corresponding to saturation return 210. Thresholding image 200 is an image transformation in which an output pixel is assigned a value "1" if the value of an input pixel is greater than a predetermined threshold, and an output pixel is assigned a value "0" if the value of an input pixel is less than the predetermined threshold. Eight-way connection means that if a second pixel is adjacent to the first pixel within a 3-pixel by 3-pixel window centered on the first pixel, then the first "1" pixel is determined to be connected to the second "1" pixel. Connected regions of "1" pixels can be formed by grouping the eight-way connected "1" pixels. Since saturation return 210 can have a return intensity many orders of magnitude larger than background returns 206, 208, it is simple to determine the threshold empirically and it can be used accurately to separate saturation return 210 from reflection regression objects from background returns 206, 208 from background objects.

[0055] After saturating the background return 210 to form a binary image, the computing device 115 can use a machine vision software program to determine the bounding box 212 corresponding to each connected region in the binary image of the saturated return 210. For example, MATLAB. TM The "regionprops" function, included in the image processing library (Mathworks, Natick, MA 01760, Rev. R2017a), can be used to determine the bounding box 212 in the binary image associated with the segmented saturation return 210. The bounding box 212 specifies the starting point. x, y Pixel coordinates and the height and width of the bounding box 212 (in pixels). Figure 1 In the context of saturation return 210, pixel highlight overflow is indicated by high-intensity pixels within bounding box 212.

[0056] Figure 3Further processing of an exemplary binary mask image 300, including a bounding box, is illustrated. The binary mask image 300 can be formed from the pseudo-color image 200 by thresholding. A color scale 302 indicates that the pixel values ​​represented in the binary image 300 include “1” represented by black pixels and “0” represented by white pixels. In this case, a threshold for forming the binary mask image 300 from the pseudo-color image 200 can be empirically determined to form connected regions for forming the background returns of binary masks 306 and 308 and the saturation returns of binary mask 310. The binary mask image 300 can be processed by a computing device 115 using a machine vision software program including morphological operators to create one or more binary mask images 300 including binary masks 306, 308, and 310 based on image 200. The binary mask image 300 can be a tight specular overflow mask created by performing “closing” and “spur” operations on the binary image 200. For example, “closing” and “spur” operations are included in MATLAB. TM In the "bwmorph" function of the image processing library, these morphological operations create clean and compact binary masks based on connected regions, which include binary masks 306, 308, and 310 in binary mask image 300 that are larger than an empirically determined minimum size (e.g., 10 pixels). For binary masks 306, 308, and 310 in binary image 300, the above can be applied... Figure 2 Define the bounding boxes 312, 314, and 316 as discussed.

[0057] The overlap ratio between the bounding box 316 associated with binary mask 310 based on saturation return 210 and the bounding box 314 associated with binary mask 308 based on background return 208 can be used to determine whether to remove or reduce specular clipping pixels. The overlap ratio can be determined using the intersection-union technique. In this technique, the pixel area ratio of the intersection (binary OR) of the two bounding boxes 314, 316 is divided by the union (binary AND) of the two bounding boxes 314, 316. The resulting overlap ratio can then be compared to an empirically determined ratio. If the determined overlap ratio is equal to or greater than a predetermined ratio, specular clipping pixels associated with saturation return 210 can be successfully removed from image 200 without affecting pixels associated with background return 208.

[0058] If the overlap rate is less than a predetermined ratio, removing the highlight overflow pixels associated with saturation return 210 will also remove the pixels associated with background return 208, which is an undesirable result. In this case, the highlight overflow pixels associated with saturation return 210 can be reduced rather than removed to retain the pixels associated with background return 208. In image 300, the overlap rate between the bounding box 314 associated with binary mask 308 and the bounding box 316 associated with binary mask 310 is less than a predetermined ratio (e.g., 5%), where the predetermined ratio can preferably be between 1% and 10%. The overlap rate can also be determined based on the overlap between the bounding box 212 from image 200 and the bounding box 314 in image 300, and compared with an empirically determined value, as described in the previous paragraph, to obtain a second overlap measurement.

[0059] Figure 4 This is a black-and-white rendering of the pseudo-color extended specular overflow mask 400. The pseudo-color extended specular overflow mask 400 encodes the binary pixels of the extended specular overflow mask 400 as 1 or 2, instead of 0 or 1, indicated by color scale 402. The extended specular overflow mask 400 is formed by subtracting the pixels associated with the binary mask 310 from the intersection of the bounding box 212 from image 200 and the bounding box 314 associated with the binary mask 308 from image 300, based on the saturation return 210.

[0060] Figure 5 This is a black-and-white rendering of the pseudo-color binary extended specular overflow mask image 500. The pseudo-color binary extended specular overflow mask image 500 encodes the binary pixels of the extended specular overflow mask image 500 as 0 or 1, as indicated by color scale 502. The extended specular overflow mask is formed by first thresholding the pseudo-color image 200 using an empirically determined threshold lower than the threshold used to form the binary image 300. This threshold can be empirically determined using a calibrated retroreflective material, where, for example, the intensity of the returned signal can be predetermined. Note that the extended specular overflow mask 400 includes more pixels of specular overflow than the binary image 300.

[0061] Figure 6This is a black-and-white rendering of the pseudo-color extended specular overflow and background mask image 600. The pseudo-color extended specular overflow and background mask image 600 encodes pixels as 0, 1, or 2, as shown by color swatch 602. The pixels in the extended specular overflow and background mask image 600 are formed by multiplying the pixels of the extended specular overflow mask 400 and the background mask image 300. This is achieved by removing the value "1" (black) pixels from the extended specular overflow and background mask image 600 and retaining "2" (half-tone) pixels for the bounding box 610 that overlaps with the bounding box 212 associated with saturation return 210, the overlap indicating that the bounding box 610 is associated with pixel specular overflow. Note that both "1" and "2" pixels can be retained because the bounding box 608 does not overlap with the bounding box 212 associated with saturation return 210, since the bounding box 608 is not associated with pixel specular overflow.

[0062] Figure 7 This is a black-and-white rendering of two pseudo-color filtered, extended specular and background mask images 700 and 704. The filtered, extended specular and background mask images 700 and 704 encode pixels as 0, 1, or 2, as shown by color stops 702 and 706. The pixels of the filtered, extended specular and background mask images 700 and 704 are formed by multiplying the pixels of the extended specular mask 400 and the background mask image 300, as described above regarding... Figure 6 The process is described, and then a morphological filter is used to filter and expand the retained background pixels while still excluding highlight clipping pixels. An exemplary morphological filter is included in MATLAB. TM The "imdilate" function in the Image Processing Toolbox. Filtered, expanded specular clipping and background mask images 700 and 704 represent filtering using 3×3 and 7×7 windows, respectively. As can be seen in the filtered, expanded specular clipping and background mask images 700 and 704, the larger filter window includes more background pixels, as well as more specular clipping pixels. The window size used to determine the ratio between background pixels and retained specular clipping pixels can be empirically determined based on test data using a calibrated regressive reflector, as mentioned above. Figure 5 The discussion.

[0063] Figure 8This is a black-and-white rendering of the pseudo-color output image 800. The pseudo-color output image encodes pixels according to color stops 802, where intensity is represented in arbitrary units from 1500 to 5000. The output image 800 includes background returns 804, 806, which are formed by multiplying image 200 by a binary mask formed by filtered, expanded specular overflow and background mask images 700, 704. In addition to the mask pixels from image 200, the output image 800 also includes pixels 808 that set the original intensity of the retroreflector, indicated by bounding box 810. This process removes the maximum number of specular overflow pixels while preserving the original intensity of the retroreflector.

[0064] Figure 9 This is a black-and-white rendering of a pseudo-color image 900 with an L2 norm function. The pixel values ​​of the L2 norm function image 900 are indicated by color stops 902. The L2 norm function is a distance-based intensity reduction matrix used to reduce pixel highlight clipping in pixels surrounding the saturation return by reducing the pixel's intensity value based on its distance from the saturation return. For example, the L2 norm function can be convolved with the pseudo-color image 900 based on the saturation return to reduce the pixel's intensity value based on distance.

[0065] Figure 10 The image shows a black-and-white rendering of a pseudo-color image 1000 and a pseudo-color output image 1004. The intensity values ​​in image 1000 and output image 1004 are encoded in arbitrary units, as indicated by color stops 1002 and 1006. Image 1000 includes a background return 1006 and saturation returns 1010 and 1012 from a retroreflector, where saturation returns 1010 and 1012 are as described above regarding... Figure 2 The identified pixels are surrounded by bounding boxes 1014 and 1016. In this example, the highlight overflow pixels caused by saturation returns 1010 and 1012 cause bounding boxes 1014 and 1016 to overlap with the bounding box surrounding background return 1006, with an overlap rate greater than a predetermined ratio. This means that, as mentioned above... Figures 3 to 8 The aforementioned highlight clipping removal cannot be performed without removing a significant portion of the background pixels. In this example, performing highlight clipping removal results in the elimination of all background pixels in the output image 1004.

[0066] In this example, for instance, the intensity-weighted centroids for saturation returns of 1010 and 1012 can be calculated by determining the centroids for each saturation return of 1010 and 1012. This can be done, for example, using MATLAB. TMThe "regionprops" function in the image processing library calculates a weighted centroid for each saturation return value of 1010, 1012, where a distance-based intensity reduction is performed based on the pixel distance to the weighted centroid of the saturation return value of 1010, 1012. The intensity reduction can be calculated using Equations 1 and 2 as the absolute value of the logarithm of the L2 norm of the distance to the weighted centroid:

[0067]

[0068] in row and col Indicates the rows and columns of image 1000. centroid x and centroid y Indicator of saturation return to the intensity-weighted centroids of 1010 and 1012 x,y coordinates, and width and height These are the width and height of the bounding box, which are 1014 and 1016 pixels respectively. dists The intensity is calculated based on the distance-based intensity reduction matrix of the L2 norm function 900, through... dists Divide by the largest dists The result is squared and then normalized.

[0069] The exemplary distance-based intensity reduction matrix shown in the L2 norm function image 900 is calculated based on saturation returns 1010 and 1012. Note the asymmetry in the L2 norm function image 900 due to the asymmetry in the dimensions of the two saturation returns 1010 and 1012. After reducing specular clipping based on the L2 norm function image 900, the original pixel values ​​of saturation returns 1020 and 1022 can be recovered in the output image 1004 within bounding boxes 1024 and 1026, respectively. In this way, pixel specular clipping can be reduced while preserving the maximum number of background pixels.

[0070] Figure 11 It is about Figures 1 to 10 The flowchart illustrates a process 1100 for operating a vehicle based on repairing pixel highlight overflow in a difference image. For example, process 1100 may be implemented by a processor of computing device 115, receiving input information from sensor 116, executing commands, and sending control signals via controllers 112, 113, and 114. Process 1100 includes multiple steps performed in the disclosed order. Process 1100 also includes embodiments with fewer steps or may include steps performed in a different order.

[0071] Process 1100 begins at step 1102, which includes a computing device 115 in vehicle 110 acquiring a first image. The first image may be acquired by vehicle 110 sensors 116, including those described above. Figure 2 The aforementioned IR video sensor.

[0072] In step 1104, the light source that uses emitted light energy to illuminate the scene (e.g., the one mentioned above) is used. Figure 2 While the IR LED illuminates the scene in the field of view of the sensor 116, the computing device 115 uses the IR video sensor 116 to acquire a second image.

[0073] At step 1106, the computing device 115 subtracts the first image from the second image to eliminate background noise and create a difference image 200, and identifies pixel highlight clipping in the difference image 200 by determining the saturation return 210 associated with the regressive reflector in the scene, as described above. Figure 2 As stated above.

[0074] At step 1108, the computing device 115 repairs pixel highlight clipping in the difference image 200 based on empirically determined parameters. Repairing pixel highlight clipping includes removing or reducing saturation return 210 based on a comparison of a determined overlap rate with a predetermined ratio, wherein the overlap rate is based on... Figures 2 to 8 The bounding boxes 212, 314, and 316 are mentioned above. Additionally, after reducing pixel highlight clipping, for example, the output image 1004 can be processed to determine whether any reduced saturation return 210 is now suitable for use with respect to... Figures 2 to 8 The described technique is used for removal.

[0075] At step 1110, the computing device 115 operates the vehicle 110 based on the output image 800 with pixel highlight overflow removed or the output image 1004 with reduced pixel highlight overflow. Operating the vehicle 110 based on the output image 800 or the output image 1004 means inputting the output image 800 or the output image 1004 into a software program executed on the computing device 115 to determine the control signals to be transmitted to the controllers 112, 113, and 114 to control the vehicle's steering, braking, and powertrain components, thereby guiding the vehicle 110 to its destination, as described above. Figure 1 As described above. After step 1110, process 1100 ends.

[0076] Figure 12 It is about Figures 1 to 10 A flowchart illustrating a process 1200 for repairing pixel highlight clipping in a difference image. Process 1200 corresponds to... Figure 11Step 1108 in process 1100. For example, process 1200 may be implemented by a processor of computing device 115, which receives input information from sensor 116, executes commands, and sends control signals via controllers 112, 113, and 114. Process 1200 includes multiple steps performed in the disclosed order. Process 1200 also includes embodiments with fewer steps or may include steps performed in a different order.

[0077] Process 1200 begins at step 1202, which includes inputting a difference image 200 into a computing device 115 in vehicle 110. The difference image 200 has been determined to include highlight overflow pixels, indicated by saturation returns 210 associated with one or more regressive reflectors in the difference image 200. The saturation returns 210 are identified by thresholding the difference image based on an empirically predetermined threshold, as described above regarding... Figure 2 The discussion.

[0078] At step 1204, computing device 115 determines the bounding boxes 212, 312, 314, 316 associated with saturation return 210 and binary masks 306, 308, 310, and thereby determines the overlap ratio of the overlapping bounding boxes 212, 314, 316, as described above regarding Figure 3 The discussion.

[0079] At step 1206, the overlap ratio determined in step 1204 is compared with an empirically predetermined ratio. In an example where the overlap parameter is greater than the predetermined ratio, process 1200 proceeds to step 1212 to reduce pixel highlight clipping. In an example where the overlap parameter is greater than or equal to the predetermined ratio, process 1200 proceeds to step 1208 to remove pixel highlight clipping.

[0080] At step 1208, the computing device 115 removes pixel highlight clipping by processing image 200 using mask images 700 and 704 according to the overlap ratio to create output image 800, as described above. Figures 2 to 8 The discussion continues. After step 1208, process 1200 proceeds to step 1210 to output images 800 and 1004.

[0081] At step 1212, the computing device 115 reduces pixel specular clipping by convolving the image 700 with the L2 norm function 900 to reduce pixel specular clipping through distance-based intensity reduction, as described above regarding... Figure 9 and Figure 10 The discussion focuses on producing output image 1004.

[0082] At step 1214, the computing device 115 may determine the bounding boxes 1014 and 1016 in the output image 1004 to determine whether the overlap ratio has changed due to processing the output image 1004 to reduce pixel highlight clipping. If the overlap ratio has changed to be less than an empirically predetermined value, the computing device, process 1200, proceeds to step 1208 to remove pixel highlight clipping in the output image 1004. If the determined overlap ratio is still greater than or equal to an empirically predetermined ratio, process 1200 proceeds to step 1210 to output the output images 800 and 1004.

[0083] At step 1210, the computing device 115 outputs repaired output images 800 and 1004, with specular overlay pixels removed or reduced, for storage in a non-volatile memory device associated with the computing device 115. The output images 800 and 1004 can be retrieved by the computing device 115 from the non-volatile memory for operation of the vehicle 110, as described above. Figure 1 As described above. After step 1210, process 1200 ends.

[0084] Computing devices such as those discussed herein typically include commands that can be executed by one or more computing devices such as those described above and are used to perform blocks or steps of the processes described above. For example, the process blocks discussed above can be embodied as computer-executable commands.

[0085] Computer-executable commands can be compiled or interpreted by computer programs created using various programming languages ​​and / or technologies, including but not limited to single or combined forms of Java™, C, C++, Visual Basic, JavaScript, Perl, HTML, etc. Typically, a processor (e.g., a microprocessor) receives commands, for example, from memory, computer-readable media, etc., and executes these commands to perform one or more processes, including one or more of the processes described herein. Various computer-readable media can be used to store and transfer such commands and other data in files. Files in a computing device are typically collections of data stored on computer-readable media, such as storage media, random access memory, etc.

[0086] Computer-readable media includes any medium that participates in providing data (e.g., commands) that can be read by a computer. Such media can take many forms, including but not limited to non-volatile media, volatile media, etc. Non-volatile media include, for example, optical discs or magnetic disks, and other persistent storage. Non-volatile media includes dynamic random access memory (DRAM), which typically constitutes main memory. Common forms of computer-readable media include, for example, floppy disks, floppy disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROM, EPROM, flash EEPROM, any other memory chip or cassette tape, or any other medium from which a computer can read.

[0087] Unless otherwise expressly indicated herein, all terms used in the claims are intended to be given their ordinary and common meaning as understood by those skilled in the art. In particular, unless the claims explicitly limit the opposite, the use of singular articles (such as “a”, “the”, “the”, etc.) should be understood to refer to one or more of the indicated elements.

[0088] The term “exemplary” is used in this document to mean an example; for example, a reference to “exemplary widget” should be understood as referring only to an example of a widget.

[0089] The adverb "approximately" when modifying a value or result refers to the fact that the shape, structure, measurement, value, determination, calculation result, etc., may deviate from the exact description of the geometric structure, distance, measurement, value, determination, calculation result, etc. due to defects in materials, machining, manufacturing, sensor measurement, calculation, processing time, communication time, etc.

[0090] In the accompanying drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements may be changed. Regarding the media, processes, systems, methods, etc., described herein, it should be understood that although the steps of such processes, etc., are described as occurring in a specific sequence, such processes can be practiced by performing the described steps in an order other than that described herein. It should also be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted. In other words, the description of processes herein is provided for illustrative purposes and should in no way be construed as limiting the claimed invention.

[0091] According to the present invention, a method includes: acquiring a first image of a scene; acquiring a second image of the scene while illuminating the scene; identifying pixel highlight overflow in a difference image determined by subtracting the first image from the second image; repairing the pixel highlight overflow based on empirically predetermined parameters; and operating a vehicle based on the difference image.

[0092] According to an embodiment, the invention is further characterized by acquiring a first image using an IR video sensor.

[0093] According to an embodiment, the present invention is further characterized in that, while illuminating the scene with IR light, a second image is acquired using an IR video sensor.

[0094] According to an embodiment, parameters predetermined by experience include a threshold and a bounding box overlap rate.

[0095] According to an embodiment, fixing pixel highlight overflow includes removing and / or reducing pixel highlight overflow based on a threshold and bounding box overlap rate.

[0096] According to an embodiment, the present invention is further characterized in that a set of highlight overflow pixels is determined based on processing the difference image to identify the connection regions of saturated pixels and the connection regions of background pixels.

[0097] According to an embodiment, removing pixel highlight overflow includes setting the pixel values ​​of a set of saturated pixels to their original values.

[0098] According to an embodiment, reducing pixel highlight overflow includes reducing the pixel intensity value of the group of highlight overflow pixels based on distance measurements.

[0099] According to an embodiment, distance measurement may include determining the L2 norm function based on the centroid of the saturated pixel and determining the aspect ratio based on the connected regions of the saturated pixel.

[0100] According to the present invention, a system is provided comprising: a processor; and a memory including instructions executed by the processor to perform the following actions: acquiring a first image of a scene; acquiring a second image of the scene while illuminating the scene; identifying pixel highlight overflow in a difference image determined by subtracting the first image from the second image; repairing the pixel highlight overflow based on empirically predetermined parameters; and operating a vehicle based on the difference image.

[0101] According to an embodiment, the invention is further characterized by acquiring a first image using an IR video sensor.

[0102] According to an embodiment, the present invention is further characterized in that, while illuminating the scene with IR light, a second image is acquired using an IR video sensor.

[0103] According to an embodiment, parameters determined empirically include a threshold and a bounding box overlap ratio.

[0104] According to an embodiment, fixing pixel highlight overflow includes removing or reducing pixel highlight overflow based on a threshold and bounding box overlap rate.

[0105] According to an embodiment, the present invention is further characterized in that a set of highlight overflow pixels is determined based on processing the difference image to identify the connection regions of saturated pixels and the connection regions of background pixels.

[0106] According to an embodiment, removing pixel highlight overflow includes setting the pixel values ​​of a set of saturated pixels to their original values.

[0107] According to an embodiment, reducing pixel highlight overflow includes reducing the pixel intensity value of the group of highlight overflow pixels based on distance measurements.

[0108] According to an embodiment, distance measurement may include determining the L2 norm function based on the centroid of the saturated pixel and determining the aspect ratio based on the connected regions of the saturated pixel.

[0109] According to the present invention, a system is provided comprising: means for acquiring an image; means for controlling a vehicle's steering, braking, and powertrain; and a computer device for: acquiring a first image of a scene; acquiring a second image of the scene while illuminating the scene; identifying pixel highlight overflow in a difference image determined by subtracting the first image from the second image; repairing the pixel highlight overflow based on empirically determined parameters; and operating the vehicle based on the difference image using the means for controlling the vehicle's steering, braking, and powertrain.

[0110] According to an embodiment, parameters determined empirically include a threshold and a bounding box overlap ratio.

Claims

1. A method for removing specular overflow from vehicle sensors, characterized in that, include: Get the first image of the scene; Acquire a second image of the scene while illuminating it; Identify pixel highlight overflow in the difference image determined by subtracting the first image from the second image; The pixel specular overexposure is repaired based on empirically predetermined parameters, including a threshold and a bounding box overlap ratio. The method for fixing the pixel highlight overflow includes: removing or reducing the pixel highlight overflow based on the threshold and the bounding box overlap rate; It also includes: determining a set of highlight overflow pixels based on processing the difference image to identify the connection regions of saturated pixels and the connection regions of background pixels; as well as The vehicle is operated based on the difference image; Reducing the pixel highlight overflow includes reducing the pixel intensity value of the group of highlight overflow pixels based on distance measurement, wherein the distance measurement includes determining the L2 norm function based on the centroid of the saturated pixel.

2. The method as described in claim 1, characterized in that, It also includes acquiring the first image using an IR video sensor.

3. The method as described in claim 1, characterized in that, Also includes: While illuminating the scene with IR light, the second image is acquired using an IR video sensor.

4. The method as described in claim 1, characterized in that, Removing the pixel highlight overflow involves setting the pixel values ​​of a set of saturated pixels back to their original values.

5. The method as described in claim 1, characterized in that, The L2 norm function is determined based on the logarithm of the row and column distances from the centroid of the saturated pixel.

6. The method as described in claim 5, characterized in that, After reducing the pixel specular overflow, a second bounding box ratio is determined, and based on the second bounding box ratio, it is determined whether to remove the pixel specular overflow.

7. The method as described in claim 6, characterized in that, Processing the difference image to identify the connection regions of saturated pixels and background pixels includes determining an extended specular overflow mask image based on a threshold applied to the difference image.

8. The method as described in claim 7, characterized in that, Processing the difference image to identify the connection regions of saturated pixels and background pixels includes: determining a filtered extended specular overflow mask image based on the extended specular overflow mask image and the background mask image.

9. A system for removing specular overflow from vehicle sensors, characterized in that, This includes a computer programmed to perform the method as described in any one of claims 1 to 8.

Images