Method and apparatus for determining vehicle speed by processing road images

Abstract

Method and apparatus for determining the speed of a vehicle by processing road images. An apparatus for determining the speed of a vehicle along a road by processing first and second images of the road captured by a camera on the vehicle, the first and second images including respective road marking images of road markings, the apparatus being arranged to: determine the position of the road markings in the first image; predict the position of the road markings in the second image based on the determined position, the estimate of the vehicle's speed, and a time period between the capturing of the images; detect the road markings in a portion of the second image where the predicted position is located; estimate a distance traveled by the vehicle during the time period based on the determined position and the position of the detected road markings in the portion of the second image; and calculate the speed based on the estimated distance and the time period.

Description

Technical Field

[0001] Example aspects herein relate generally to the field of image processing, and more particularly, to processing digital images acquired by a vehicle-mounted camera. Background Art

[0002] The ability to detect areas of the road that are not occupied by vehicles or other objects is useful in many active safety (AS) technologies. Detection of road occupancy is central to many active safety features, such as lateral control (LC) and lane change assist (LCA). Furthermore, lane detection via road markings such as Bott's Dot (BD) is used in many active safety features, such as lane departure warning (LDW), traffic jam assist (TJA), and adaptive cruise control (ACC). Equipping vehicles with LDW functionality is currently a requirement, for example, during European New Car Assessment Program (NCAP) testing. Over the past few years, there has been a strong desire to develop these technologies. Summary of the Invention

[0003] A method for determining the speed of a vehicle traveling along a road by processing images of the road captured by a camera mounted on the vehicle has been devised. The method can allow the vehicle's speed to be independently determined in a computationally efficient manner and has been found to be able to reliably determine the vehicle's speed based on images acquired at different times with widely varying lighting and contrast conditions.

[0004] More specifically, according to the first aspect of the present invention, a method is designed for determining the speed of a vehicle traveling along a road by processing an image pair of a road, the image pair being captured by a camera mounted on the vehicle, each of the image pairs having a common image coordinate system and including a first image of the road and a second image of the road, the first image being captured at a first moment and the second image being captured at a second moment, the second moment being a predetermined time period later than the first moment, wherein the first image and the second image include corresponding road marker images of road markers on the road, the method comprising processing each of the image pairs by the following steps: determining the position of the road marker image in the first image in the image coordinate system; predicting the position of the road marker image in the second image in the image coordinate system based on the determined position of the road marker image, the estimate of the speed of the vehicle and the predetermined time period; detecting the road marker image in the portion of the second image where the predicted position is located; estimating the distance traveled by the vehicle along the road during the predetermined time period based on the determined position of the road marker image in the first image and the position of the detected road marker image in the portion of the second image; and calculating the speed of the vehicle based on the estimated distance and the predetermined time period.

[0005] In the above method, a calculation of the cross-correlation between the portion containing the determined position of the first image and the portion containing the predicted position of the second image can be performed, and the result of the calculation can be used to detect and determine the position of the road sign image in the portion of the second image, and to estimate the distance traveled by the vehicle along the road within the predetermined time period.

[0006] The estimate of the vehicle's speed used to predict the position of the road marking image in the second image during processing of one of the image pairs may be the speed of the vehicle measured by the vehicle's speedometer, or alternatively may be the speed of the vehicle calculated during processing of one of the image pairs previously captured by the camera.

[0007] The predicted position of the road marker image in the second image can be predicted using a mapping between a first variable and a second variable, wherein the first variable represents the position of a portion of the image in the image coordinate system, and the second variable represents the distance of the portion of the road represented by the portion of the image from the vehicle.

[0008] Alternatively, a mapping between a first variable and a second variable can be used to estimate the distance traveled by the vehicle along the road during the predetermined time period, the first variable representing the position of a portion of an image in an image coordinate system, and the second variable representing the distance of the portion of the road represented by the portion of the image from the vehicle. The distance traveled by the vehicle along the road during the predetermined time period can be estimated by: determining a first value using the determined positions of road marker images in the first image and the mapping, the first value representing the distance of the portion of the road represented by the determined positions of the road marker images in the first image from the vehicle; determining a second value using the determined positions of road marker images in the second image and the mapping, the second value representing the distance of the portion of the road represented by the determined positions of the road marker images in the second image from the vehicle; and estimating the distance traveled by the vehicle along the road during the predetermined time period using the first value and the second value. The mapping can also be used to predict the predicted position of the road marker images in the second image.

[0009] The camera may be arranged to capture an image of the road on one side of the vehicle as the vehicle travels along the road as the image, in which case the mapping is a linear mapping. Alternatively, the camera may be arranged to capture an image of the road behind or in front of the vehicle as the vehicle travels along the road as the image, in which case the mapping is a non-linear mapping.

[0010] The mapping may be a polynomial relating the first variable to the second variable, a lookup table relating the first variable to the second variable, or a polyline relating the first variable to the second variable.

[0011] The position of the road sign image in the first image can be determined by the following operations: generating an M-th level LL subband image of the (M+1)-level discrete wavelet transform DWT of the first image by iteratively low-pass filtering and downsampling the first image M times, wherein M is an integer equal to or greater than 1; generating an (M+1)-th level subband image of the (M+1)-level DWT decomposition of the first image by high-pass filtering the M-th level LL subband image and downsampling the result of the high-pass filtering; generating boundary data representing the boundary of the road sign image in the first image by determining the boundary of a pixel area of the (M+1)-th level subband image, wherein the pixel area is surrounded by pixels having pixel values substantially different from pixel values of pixels in the pixel area; and generating boundary data representing the boundary of the road sign image in the first image by amplifying the boundary data of the pixel area of the (M+1)-th level subband image by 2 M+1 times to determine the position of the road marker image in the first image.

[0012] A first low-pass filter having a symmetrical first sequence of filter coefficients may be used in at least one iteration of the iterative process.

[0013] The filter coefficients of the first sequence of filter coefficients may be set to values in a row of Pascal's triangle having the same number of values as the order of the first low-pass filter.

[0014] The high-pass filtering for generating the (M+1)th level sub-band image may include applying a high-pass filter having a symmetrical second sequence of filter coefficients. In addition, the filter coefficients of the second sequence of filter coefficients may be set to the values of the corresponding positions in the rows of the Pascal triangle having the same number of values as the order of the high-pass filter, and each of the remaining filter coefficients of the second sequence of filter coefficients may be set to a value obtained by multiplying the value of the corresponding position in the row of the Pascal triangle by -1.

[0015] The sub-band image of the (M+1)th level of the (M+1)-level DWT decomposition may be one of an LH sub-band image, an HL sub-band image, and an HH sub-band image.

[0016] In one embodiment, generating the (M+1)th level subband image of the (M+1)th level DWT decomposition of the first image may include generating the (M+1)th level LH subband image of the (M+1)th level DWT decomposition of the first image by the following operations: generating a low-pass filtered LL subband image by applying a row kernel on the rows of the LL subband image of the Mth level, the row kernel corresponding to the low-pass filter; downsampling the columns of the low-pass filtered LL subband image by a factor of 2 to generate a downsampled subband image; generating a high-pass filtered LL subband image by applying a column kernel on the columns of the downsampled subband image, the column kernel corresponding to the high-pass filter; and downsampling the rows of the high-pass filtered LL subband image by a factor of 2 to generate the (M+1)th level LH subband image.

[0017] In a first variant of this embodiment, generating the (M+1)th level subband image of the (M+1)th level DWT decomposition of the image may include generating LH subband images of the (M+1)th level DWT decomposition of the image by the following operations: generating a high-pass filtered LL subband image by applying a column kernel on the columns of the Mth level LL subband image, the column kernel corresponding to the high-pass filter; downsampling the rows of the high-pass filtered LL subband image by a factor of 2 to generate a downsampled subband image; generating a low-pass filtered subband image by applying a row kernel on the rows of the Mth level downsampled subband image, the row kernel corresponding to the low-pass filter; and downsampling the columns of the low-pass filtered subband image by a factor of 2 to generate the (M+1)th level LH subband image.

[0018] In a second variant of this embodiment, generating a (M+1)th level subband image may include generating a (M+1)th level LH subband image of the (M+1)th level DWT decomposition of the image by the following operations: generating a filtered subband image by applying a two-dimensional kernel to the Mth level LL subband image, the two-dimensional kernel being divisible into the product of a row kernel and a column kernel, the row kernel defining a low-pass filter, and the column kernel defining a high-pass filter; and downsampling the rows and columns of the filtered subband image by a factor of 2.

[0019] The road sign may be a Bottom point, and the (M+1)th level subband image of the (M+1)th level of the (M+1)-level DWT decomposition may be the (M+1)th level LH subband image of the (M+1)-level DWT decomposition.

[0020] Boundary data can be generated by: determining the pixel positions of pixels whose pixel values exceed a predetermined threshold in the sub-band image of the (M+1)th level; and using the determined pixels to perform a contour tracing algorithm to identify the boundary of the pixel area, wherein the boundary separates pixels of the pixel area that are adjacent to the boundary and have pixel values higher than the predetermined threshold from pixels outside the pixel area and adjacent to the boundary and have pixel values lower than the predetermined threshold.

[0021] Furthermore, according to a second aspect herein, a computer program is provided comprising computer-readable instructions which, when executed by a processor, cause the processor to carry out the method as described above.

[0022] In addition, according to the third aspect herein, a device has been designed to determine the speed of a vehicle traveling along a road by processing a pair of road images captured by a camera mounted on the vehicle, each of these image pairs having a common image coordinate system and comprising a first image of the road and a second image of the road, the first image being captured at a first moment and the second image being captured at a second moment, the second moment being a predetermined time period later than the first moment, wherein the first image and the second image include corresponding road marker images of road markers on the road. The device includes: a position determination module, which is arranged to determine the position of the road marker image in the first image in the image coordinate system; a position prediction module, which is arranged to predict the position of the road marker image in the second image in the image coordinate system based on the determined position of the road marker image, the estimate of the speed of the vehicle and the predetermined time period; a road marker image detection module, which is arranged to detect the road marker image in the part of the second image where the predicted position is located; a distance estimation module, which is arranged to estimate the distance traveled by the vehicle along the road during the predetermined time period based on the determined position of the road marker image in the first image and the position of the detected road marker image in the part of the second image; and a speed calculation module, which is arranged to calculate the speed of the vehicle based on the estimated distance and the predetermined time period. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments will now be explained in detail, by way of non-limiting example only, with reference to the accompanying drawings described below. The same reference numerals appearing in different figures may indicate identical or functionally similar elements, unless otherwise specified.

[0024] Figure 1 is a schematic diagram of an apparatus for processing an image captured by a vehicle-mounted camera according to an embodiment.

[0025] Figure 2 is a block diagram illustrating an example implementation of a device implemented in programmable signal processing hardware.

[0026] Figure 3 is a schematic diagram of the sequence of filtering and downsampling operations performed in a single-level DWT decomposition of a two-dimensional image to generate four subband images.

[0027] Figure 4 It shows Figure 1 A flowchart of a method in which an unoccupied road area determination module of a device processes an image to generate boundary data.

[0028] Figure 5Examples of low-pass filters and high-pass filters having different filter orders used in the present embodiment are shown, with filter coefficients set according to values in a row of Pascal's triangle.

[0029] Figure 6 Shown Figure 5 Frequency responses of the low-pass filter and complementary high-pass filter for different filter orders.

[0030] Figure 7A A method is shown that may be performed by a boundary data generator module of an unoccupied road area determination module to identify a boundary of an area in a sub-band image that represents a boundary of an unoccupied road area in an image captured by an onboard camera.

[0031] Figure 7B An alternative method that may be performed by a boundary data generator module to identify boundaries of regions in a sub-band image that represent boundaries of unoccupied road areas in an image acquired by an onboard camera is shown.

[0032] Figure 8A Shown in Figure 7B In an example implementation of the method, a boundary of a region in a sub-band image is generated, the boundary representing a boundary of an unoccupied road region in an image acquired by a vehicle-mounted camera.

[0033] Figure 8B Shown in Figure 7B In another example implementation of the method, a boundary of a region in a sub-band image is generated, where the sub-band image represents a boundary of an unoccupied road region in an image acquired by a vehicle-mounted camera.

[0034] Figure 9A An example of an image of a scene including a road captured by a vehicle-mounted camera is shown.

[0035] Figure 9B Shown from Figure 9A An example of LL subband images generated from the images in .

[0036] Figure 9C Shown from Figure 9A Example of LH subband image generated from the image in .

[0037] Figure 10A 、 Figure 10B 、 Figure 10C shows different images acquired by the vehicle-mounted camera, Figure 10D 、 Figure 10E 、 Figure 10F Show use Figure 4 The image processing methods are respectively Figures 10A to 10C The LH subband image is derived from the image.

[0038] Figure 11 A method performed by a refined boundary data generator module of an embodiment is shown.

[0039] Figure 12 A method of refining a boundary determined by a boundary data generator module is shown, and the method is performed by the refined boundary data generator module of this embodiment.

[0040] Figure 13 A method for determining a boundary in a sub-band image, which represents a boundary of a road marker area in an image acquired by a vehicle-mounted camera, is shown, and is performed by the road marker determination module of this embodiment.

[0041] Figure 14A An image of a road with road signs processed by the road sign identification module of this embodiment is shown.

[0042] Figure 14B Shown according to Figure 14A The sub-band image is generated by an image of the road marking area, which shows the road marking area as a bright spot on the dark area corresponding to the unoccupied road area.

[0043] Figure 15 Shown Figure 1 Components of the vehicle speed detection module are shown.

[0044] Figure 16 It is shown by Figure 15 Flowchart of a method for determining vehicle speed performed by a vehicle speed detection module.

[0045] Figure 17A An image captured by a forward-facing camera mounted on a vehicle is shown.

[0046] Figure 17B It is through Figure 17A The image is transformed and evenly spaced horizontal lines are added to obtain a bird's-eye view of the road.

[0047] Figure 17C Shown in the Figure 17B The bird's-eye view image shown is an image of a road after inverse perspective transformation has been performed.

[0048] Figure 18 The mapping between the value of the Y-axis coordinate of a pixel of the image and the distance from the vehicle to the road portion represented by the pixel is shown.

[0049] Figure 19A A first image is shown which includes an image of a road marking which has been captured by a vehicle-mounted camera at a certain moment.

[0050] Figure 19BShown include Figure 19B A second image of an image of a road sign in the image, the second image having been captured at a second moment, the second moment being a predetermined time period after the first moment. DETAILED DESCRIPTION

[0051] The image processing method described here is based on the discrete wavelet transform (DWT), by which an image can be decomposed into its composite frequency components for analysis. In a typical application of the DWT for image processing, after computing the DWT of an input image, a reconstruction stage is used to reconstruct the original image. In order to reconstruct the original signal (or a denoised version of the original signal, if image denoising is performed on subband images before the reconstruction stage), an orthogonal filter bank comprising orthogonal filters corresponding to orthogonal wavelets is required. However, the use of orthogonal wavelets such as those derived from the Daubechies wavelet to implement an orthogonal filter bank makes it computationally expensive to implement the DWT decomposition process (e.g., due to the presence of complex filter coefficients required to implement these filters). This is problematic in real-time image recognition systems, where processing power is typically limited and high-speed feature detection is required.

[0052] The image processing method herein involves some modifications to the DWT method described above which are made possible by the characteristics of the images being processed and which allow the simplifications and attendant advantages described above to be achieved.

[0053] More specifically, it has been discovered that, in the DWT subband images generated by the action of at least one high-pass filter, low spatial frequency components in the input image corresponding to unoccupied areas of the road in the image can be effectively removed, leaving in their place substantially dark regions of the subband image whose boundaries can be easily delineated using the boundary-finding algorithm described herein. The removal of these low-frequency components enhances contrast regardless of the lighting conditions under which the camera captured the image. The combination of image processing operations described herein makes the overall process robust to varying noise and lighting conditions.

[0054] Because reconstruction of the original signal is not required in the image processing method described herein, there is no need to ensure the orthogonality of the analysis filter bank. This allows for greater freedom in choosing the structure of the filter and, in particular, allows the use of symmetrical integer-valued filters to implement the DWT decomposition, which promotes efficient, low-complexity implementation in hardware. In addition, in some embodiments, the filters used to generate the subband images of the DWT decomposition are designed based on the values in the rows of the Pascal triangle. It has been found that such filters with a simple filter structure and a flat frequency response are well suited for extracting low-frequency components associated with unoccupied areas of the road in the original image.

[0055] Furthermore, not all subbands of the multi-level DWT decomposition process need to be calculated at each level of the DWT decomposition. Instead, to determine the boundaries of the unoccupied areas of the road, only one subband image from each level of the DWT decomposition needs to be calculated, thereby significantly reducing the complexity of the search process.

[0056] Figure 1 FIG2 is a schematic diagram of an apparatus 1 for processing an image acquired by a vehicle-mounted camera according to an embodiment. The apparatus 1 includes a vehicle speed determination module 30 and, as in this embodiment, may further include an unoccupied road area determination module 10 and a road marker determination module 20.

[0057] The (optional) unoccupied road area determination module 10 is configured to process an image of a scene including a road captured by an onboard camera to generate boundary data representing the boundary of an area in the captured image that represents an unoccupied area of the road. In this embodiment, the captured image may be a grayscale image. However, the image processing techniques described below may also be used to process one or more color components of a color image.

[0058] The unoccupied road area determination module 10 includes a discrete wavelet transform (DWT) decomposition module 12, which is arranged to generate an N-th level LL subband image of an (N+1)-level DWT decomposition of a captured image by performing an iterative process of iteratively low-pass filtering and downsampling the image N times, where N is an integer equal to or greater than 1. In other words, the DWT decomposition module 12 is arranged to generate an N-th level LL subband image of an N+1-level DWT decomposition of an image by low-pass filtering and downsampling the image in a first iteration, applying the same low-pass filtering and downsampling to the result of the first iteration to generate a result of a second iteration, applying the same low-pass filtering and downsampling to the result of the second iteration to generate a result of a third iteration, and so on.

[0059] The DWT decomposition module 12 is further configured to generate a sub-band image of the (N+1)th level of the (N+1)-level DWT decomposition of the image by high-pass filtering the LL sub-band image of the Nth level and down-sampling the result of the high-pass filtering, such that the sub-band image of the (N+1)th level has at least one pixel region having substantially equal pixel values, the pixel region representing at least one unoccupied area of the road in the image acquired by the onboard camera, and the remainder of the sub-band image of the (N+1)th level represents the remainder of the scene in the image acquired by the onboard camera. The pixels in the at least one pixel region may be "substantially equal" to each other in the sense that the pixel value spread of these pixels (e.g., as quantified in terms of the range, variance, or standard deviation of the pixel values) is smaller than the (same defined) spread of the pixels forming the remainder of the sub-band image of the (N+1)th level (which represents the remainder of the scene in the image acquired by the onboard camera).

[0060] The unoccupied road area determination module 10 further comprises a boundary data generator module 14 arranged to generate boundary data by determining boundaries of pixel areas having substantially equal pixel values.

[0061] Figure 2 FIG2 is a schematic diagram of a programmable signal processing device 200, which can be configured to implement the functionality of device 1. Signal processing device 200 includes an interface module 210 for receiving image data defining a digital image captured by an onboard camera. Signal processing device 200 also includes a processor (CPU) 220 for controlling device 1, a working memory 230 (e.g., random access memory), and an instruction storage unit 240 storing a computer program comprising computer-readable instructions that, when executed by processor 220, cause processor 220 to perform the processing operations of device 1. Instruction storage unit 240 can include a ROM (e.g., in the form of an electrically erasable programmable read-only memory (EEPROM) or flash memory) preloaded with computer-readable instructions. Alternatively, instruction storage unit 240 can include RAM or a similar type of memory, and the computer-readable instructions can be input thereto from a computer program product (e.g., a computer-readable storage medium 250, such as a CD-ROM, etc.) or a computer-readable signal 260 carrying the computer-readable instructions.

[0062] In this exemplary embodiment, Figure 2 The illustrated combination of hardware components 270 (including processor 220, working memory 230, and instruction storage 240) is configured to implement Figure 1 The device 1 is shown with the functionality of each of its component modules.

[0063] Figure 3shows the sequence of filtering and downsampling operations performed on a single-level DWT decomposition of a two-dimensional image. Figure 3 In, use Figure 3 The filter bank structure shown in the figure decomposes the input image X(n) into four sub-band images, namely, the LL sub-band image X LL (n), LH subband image X LH (n), HL subband image X HL (n) and HH subband image X HH (n). The subscripts LL, LH, HL, and HH denote the type of filter used to generate the corresponding subband image. For example, Figure 3 As shown, the LL subband image X LL (n) is generated by applying a low-pass filter 310 to the rows of the input image X(n), downsampling the columns of the resulting intermediate image by a factor of 2, applying a low-pass filter 310 to the columns of the downsampled image, and downsampling the rows of the resulting image by a factor of 2. The LH subband image is generated in a similar manner and differs from the generation of the LL subband only in that a high-pass filter 320 is applied to the columns of the downsampled image instead of the low-pass filter 310.

[0064] The low-pass filter 310 and the high-pass filter 320 in the DWT decomposition form an orthogonal mirror filter bank pair, so that the magnitude response of the high-pass filter 320 is derived from the magnitude response of the low-pass filter 310 by mirroring the magnitude response of the low-pass filter 310 around the value π / 2 in the frequency domain. For the multi-level DWT decomposition process, iteratively perform Figure 3 The sequence of filtering and downsampling steps shown in FIG. That is, for the first level of the multi-level DWT decomposition process, the input image X(n) is decomposed into LL, LH, HL and HH sub-band images of the first level. For the n-th level of the multi-level DWT decomposition process, the LL sub-band image generated at the (n-1)th level of the DWT decomposition is subjected to Figure 3 The sequence of filtering and downsampling operations is shown in order to obtain the LL, LH, HL and HH subband images of level n.

[0065] Figure 4 It is shown by Figure 1 Flowchart of a method for processing an image of a road scene acquired by a vehicle-mounted camera performed by the unoccupied road area determination module 10. The image is processed to generate boundary data representing the boundary of an area of the image representing an unoccupied area of the road.

[0066] In this embodiment, the vehicle-mounted camera may be a forward-looking camera mounted on a car, van, truck, or other road vehicle, and the image captured may include an unoccupied area of the road in front of the vehicle. It should be noted that the image may alternatively be a rear-view image of the road captured by a rear-view camera mounted on the vehicle, or a side-view image captured by a side-view camera mounted on the left or right side of the vehicle.

[0067] In this embodiment, the unoccupied road area determination module 10 can process the image of the scene acquired by the vehicle-mounted camera, that is, no pre-processing of the image acquired by the vehicle-mounted camera is performed before the image is input into the unoccupied road area determination module 10 for processing.

[0068] exist Figure 4 In step S10, the DWT decomposition module 12 generates an N-th level LL subband image of the (N+1)-level DWT decomposition of the image by iteratively performing low-pass filtering and downsampling N times on the acquired image, where N is an integer equal to or greater than 1. That is, the first level LL subband image of the (N+1)-level DWT decomposition is generated by low-pass filtering and downsampling the image. In addition, for an N value equal to or greater than 2, the L-th level LL subband image of the DWT decomposition is generated by low-pass filtering the (L-1)-th level LL subband image, where L is an integer greater than 1 and less than or equal to N.

[0069] In the present embodiment, the DWT decomposition module 12 can use a low-pass filter 310 having a first sequence of filter coefficients that are symmetrical in at least one iteration of the iterative low-pass filtering and downsampling iterative process performed in step S10. In addition, in the present embodiment, the filter coefficients in the filter coefficient sequence of the low-pass filter 310 can have integer values. In addition, in the present example, the filter coefficients in the sequence can be set to values in a row of the Pascal triangle having the same number of values as the order of the low-pass filter 310. However, the filter coefficient sequence of the low-pass filter 310 does not need to be selected based on the rows of the Pascal triangle and can alternatively take the form of other symmetrical integer values.

[0070] In this embodiment, each iteration of the iterative low-pass filtering and downsampling may use Figure 33. The sequence of low-pass filtering and downsampling steps for generating the LL subband is shown. That is, each iteration of the iterative low-pass filtering and downsampling process performed in step S10 may include: applying a 1×L1 row kernel corresponding to the low-pass filter 310 to the rows of the LL subband image of the previous level of DWT decomposition (also referred to herein as the 'DWT level') (or across the rows of the acquired image if the iteration corresponds to the first level of DWT decomposition), downsampling the columns of the low-pass filtered image by a factor of 2. Further applying an L2×1 column kernel corresponding to the low-pass filter 310 to the columns of the downsampled image, and then downsampling the rows of the further filtered image.

[0071] In this embodiment, the same low-pass filter 310 is used to filter both the rows and columns of the input image at each of the first N levels of the (N+1)-level DWT decomposition, so as to generate an LL subband image for each of the first N levels. However, it should be noted that the generation of the LL subband image of the Nth level of the DWT decomposition is not limited to this. In particular, at each level of the DWT decomposition in which the LL subband image is calculated, L1 does not need to be equal to L2, and different low-pass filters can be used to filter the rows and columns. In addition, different low-pass filters with different filter coefficients and / or different filter orders can alternatively be used for different levels of the DWT decomposition.

[0072] In an alternative embodiment, each iteration of iterative low-pass filtering and downsampling in step S10 may alternatively include applying an L2×1 column kernel corresponding to the low-pass filter 310 to the columns of the LL subband image of the previous level of DWT decomposition (or to the columns of the input image if the iteration corresponds to the first level of DWT decomposition), downsampling the low-pass filtered image by a factor of 2 for the rows of the low-pass filtered image, applying a 1×L1 row kernel corresponding to the low-pass filter 310 to the rows of the downsampled image, and downsampling the columns of the further filtered image by a factor of 2.

[0073] Furthermore, in some embodiments, Figure 4 Each iteration of iterative low-pass filtering and downsampling performed in step S10 comprises generating a low-pass filtered sub-band image by applying a two-dimensional kernel to the LL sub-band image of the previous level of DWT decomposition (or to the acquired (input) image if the iteration corresponds to the first level of DWT decomposition). The two-dimensional kernel can be divided into the product of two one-dimensional kernels, namely the row kernel and the column kernel. Both the row kernel and the column kernel are defined by the low-pass filter 320. The low-pass filtered image can then be downsampled by a factor of 2 along its rows and columns.

[0074] exist Figure 4In step S20, the DWT decomposition module 12 generates an (N+1)th subband image of the (N+1)th level of DWT decomposition of the input image by performing high-pass filtering on the Nth level LL subband image and then down-sampling the result of the high-pass filtering, so that the (N+1)th level subband image has a pixel area with substantially equal pixel values, which represents an unoccupied area of the road in the image acquired by the vehicle-mounted camera.

[0075] Pixels adjacent to a region in the (N+1)th level subband image may have pixel values substantially different from the pixel values of pixels within the region. In this embodiment, the region may be a substantially dark region defined by pixels having low pixel values because high-pass filtering of the Nth level LL subband image removes low spatial frequency components of the image representing the substantially uniform surface of the road. Therefore, in this embodiment, the region in the (N+1)th level subband image may have pixels having pixel values substantially lower than the pixel values of pixels outside the region adjacent to (i.e., adjacent to) the region. Furthermore, the average pixel value of the region may be substantially lower than the average pixel value of pixels outside the region adjacent to the region.

[0076] In this embodiment, Figure 4 The high-pass filtering for generating the (N+1)th level subband image in step S20 may include applying a high-pass filter 320 having a symmetrical filter coefficient sequence. Furthermore, in this embodiment, the filter coefficients in the sequence may also be integer values. Furthermore, as in this embodiment, the alternate filter coefficients in the filter coefficient sequence of the high-pass filter 320 may be set to the values of the corresponding positions in the rows of the Pascal triangle having the same number of values as the order of the high-pass filter 320. Furthermore, in this embodiment, each of the remaining filter coefficients in the filter coefficient sequence of the high-pass filter 320 is set to a value obtained by multiplying the value of the corresponding position in the rows of the Pascal triangle by -1. However, it should be noted that the filter coefficients of the low-pass filter 310 and the high-pass filter 320 used to generate the LH subband are not limited to the filter coefficients of the symmetrical integer-valued filters described above.

[0077] In this example, N is taken as 4, so that Figure 4 In step S10, the LL subband of the 4th level of the 5-level DWT decomposition of the image is generated by the DWT decomposition module. However, the value of N is not limited thereto. In some embodiments, the DWT decomposition module 12 is configured to set the value of N to a value based on the image resolution of the image captured by the on-board camera, for example by using a predetermined mapping between the value of the image resolution and the value of N. In other embodiments, particularly where the resolution of the image captured by the on-board camera is known and is not expected to change, the value of N may be predetermined and, for example, may be set to a value of 4 or greater. Figure 4 The iterative low-pass filtering and downsampling across the (N+1) levels of DWT decomposition performed in step S10 achieves the effect of smoothing the image and extracting low spatial frequency components with increasingly narrower bandwidths with each iteration. As the number of iterations of iterative low-pass filtering increases (corresponding to an increase in the level of DWT decomposition), signal components corresponding to unoccupied areas of the road are extracted, as these signal components typically form the lowest frequency components in the image of the scene acquired by the camera, due to the largely uniform appearance of the road surface.

[0078] Furthermore, in this embodiment, Figure 4 The (N+1)th level subband image of the DWT decomposition generated in step S20 is the LH subband of the 5th level (also referred to herein as '5th level') of the 5-level DWT decomposition. However, it should be noted that the (N+1)th level subband image does not have to be the LH subband image, but may alternatively be the HL subband image or the HH subband image of the (N+1)th level of the DWT decomposition. In this embodiment, generating the 5th level LH subband image may include applying a 1×L1 row kernel corresponding to the low-pass filter 310 to the rows of the 4th level LL subband image, downsampling the columns of the low-pass filtered image by a factor of 2, further applying an L2×1 column kernel corresponding to the low-pass filter 320 to the columns of the downsampled image, and then downsampling the rows of the further filtered image by a factor of 2 to generate the 5th level LH subband image.

[0079] Furthermore, in this embodiment, a low-pass filter 310 and a high-pass filter 320 forming an orthogonal mirror image pair can be used to generate a fifth-order LH subband image. In this embodiment, the low-pass filter 310 and the high-pass filter 320 used to generate the fifth-order LH subband image are filters of the same order, such that L1 = L2. However, it should be noted that the low-pass filter 310 does not need to have the same order as the high-pass filter 320, and the size of the kernel corresponding to each filter can be selected based on the characteristic properties of the image feature being sought.

[0080] In some embodiments, generating the LH subband image of level 5 may alternatively include applying an L2×1 column kernel corresponding to the high-pass filter 310 on the columns of the LL subband image of level 4, downsampling by a factor of 2 on the rows of the low-pass filtered image, applying a 1×L1 row kernel corresponding to the low-pass filter 310 on the rows of the downsampled image, and downsampling by a factor of 2 on the columns of the further filtered image to generate the LH subband image of level 5 of the DWT decomposition.

[0081] Furthermore, in some embodiments, generating the LH subband at level 5 can alternatively be performed by first generating a filtered subband image by applying a two-dimensional kernel to the LL subband image at level 4. The two-dimensional kernel can be divided into the product of two one-dimensional kernels, namely a row kernel and a column kernel. The row kernel corresponds to the low-pass filter 310, and the column kernel corresponds to the high-pass filter 320. The low-pass filtered image can then be downsampled by a factor of 2 along its rows and columns to generate the LH subband image at level 5.

[0082] In the present embodiment, each of the low-pass filter 310 and the high-pass filter 320 may have filter coefficients of symmetrical integer values set according to the values in the rows of the Pascal triangle. More specifically, in the present embodiment, the low-pass filter 310 for generating the LH subband image of the 5th level of the DWT decomposition has filter coefficients of [1, 6, 15, 20, 15, 6, 1], and the complementary high-pass filter forming an orthogonal mirror pair with the low-pass filter has filter coefficients obtained by multiplying every second coefficient of the low-pass filter by -1, so that the high-pass filter has filter coefficients of [1, -6, 15, -20, 15, -6, 1]. In addition, in the present example, each of the high-pass filter 320 and the low-pass filter 310 may be further multiplied by a factor of 1 / 64 (or 1 / 2 in the general case of using an F-order filter). F ) normalization. However, to reduce computational complexity, the filtering operations (i.e., convolution of the filter input with the filter kernel) can be performed using integer-valued coefficients of the low-pass filter 310 and the high-pass filter 320, and the normalization factors associated with each filter can be applied separately after the filtering operations.

[0083] Many significant advantages can be achieved by using symmetrical integer-valued filters as described above. First, filters with integer coefficients facilitate low-complexity implementation of the image processing algorithms described herein in software and / or hardware (FPGA or ASIC) systems. In particular, small integer-valued coefficients facilitate efficient implementation of the filtering process in fixed-point arithmetic, where multiplications can be implemented with very small summations and divisions can be implemented by shifting bits. Furthermore, by using a symmetrical sequence of filter coefficients, a high-pass filter of a complementary orthogonal mirror pair can be designed from the filter coefficient sequence of a low-pass filter by multiplying every second coefficient in the sequence by -1.

[0084] It has been discovered that using a symmetric filter whose coefficients are set according to the values in a row of the Pascal triangle can allow for more accurate determination of unoccupied road areas in acquired images. In particular, a low-pass filter with such filter coefficients has a substantially flat frequency response, where the cutoff frequency decreases toward zero as the filter order increases. Since the signal components of interest in an image are low-frequency components corresponding to unoccupied road areas, a symmetric filter whose filter coefficients are set according to the values in a row of the Pascal triangle can allow for more effective filtering of unwanted high-frequency noise components in the image and more effective extraction of low-frequency components corresponding to unoccupied road areas in the image. Thus, compared to orthogonal filters based on orthogonal wavelets (e.g., Daubechies wavelets), symmetric filters designed based on the values in a row of the Pascal triangle can allow for more effective detection of unoccupied road areas.

[0085] Figure 5 Low-pass filter coefficients 510 for low-pass filters of different orders are shown, the low-pass filter coefficients 510 being based on the values in the rows of Pascal's triangle. Figure 5 Also shown are high-pass filter coefficients 520 for high-pass filters of different orders, which are also based on the values of the rows of Pascal's triangle. Figure 5 Integer-valued high-pass and low-pass filters use a factor of 1 / 2. F Normalization. However, Figure 5 The filters shown may be considered integer filters, as the filtering step may be performed using a filter defined entirely by integer-valued coefficients, whereas a normalization factor may be applied, for example, as a separate step before or after the filtering operation.

[0086] Figure 6 It is shown that for different filter orders, using Figure 5 The frequency responses of the low-pass filter and the complementary high-pass filter designed by the rows of Pascal's triangle are shown. Figure 6 As the filter order increases, the cutoff frequency of the low-pass filter decreases toward zero frequency, while the lower cutoff frequency of the high-pass filter approaches the Nyquist frequency. This provides a desirable characteristic that allows for more efficient extraction of low-frequency components in an image.

[0087] It should be noted that although Figure 3 , a single level of four subbands of the DWT decomposition process is shown in FIG, but the present method of determining the boundary representing the boundary of the unoccupied road area only needs to generate the LL subband of the first N levels of the (N+1)-level DWT decomposition and generate one of the LH subband image, HL subband image, and HH subband image of the (N+1)-th level of the DWT decomposition. There is no need to generate the remaining subband images of each level.

[0088] Refer again Figure 4In step S30, the boundary data generation module 16 generates boundary data by determining the boundary of a pixel region in the (N+1)th level subband image (in this example, the LH subband of level 5), which pixel region represents an unoccupied area of the road in the acquired image. In this embodiment, the region may include pixels having pixel values that are significantly lower than those of neighboring pixels in the region. For example, the region may have an average pixel value that is substantially lower than the average pixel value of neighboring pixels in the region. In addition, the spread of pixel values of pixels in the region (e.g., measured in terms of range, variance, or standard deviation of pixel values) may be substantially lower than the spread of pixel values of pixels adjacent to the region (similarly defined).

[0089] Figure 7A A method is shown which can be Figure 4 The boundary data generator module 14 in step S30 is executed to identify the boundary of the area in the LH sub-band image of level 5, which area represents the unoccupied road area in the acquired image.

[0090] exist Figure 7A In step S32-1, the boundary data generator module 14 first determines the pixel location of a pixel in the level 5 LH sub-band image whose pixel value is below a predetermined pixel threshold. This pixel location can be iteratively determined by examining each pixel in the LH sub-band image until a pixel whose pixel value is below the predetermined pixel threshold is found. The search for the pixel location can, for example, begin at the bottom boundary of the LH sub-band image, as the area of the LH sub-band image representing the unoccupied road area can typically extend beyond the bottom boundary of the LH sub-band (because cameras are typically mounted on vehicles such that the lower portion of their field of view does not include a portion of the vehicle, such as the hood or roof). However, the search is not limited to this.

[0091] exist Figure 7A In step S34-1, once the pixel positions have been determined, the boundary data generator module 14 uses the determined pixel positions to perform a contour tracing algorithm to identify the boundary of the region. The boundary separates the pixels of the region that are adjacent to the boundary and have pixel values below a predetermined threshold from the pixels outside the boundary that are adjacent to the boundary and have pixel values above the predetermined threshold. Any suitable boundary / contour tracing algorithm may be used in step S34-1, such as a Moore-neighbour-tracing algorithm, a radial sweep algorithm, or a square tracing algorithm. However, it should be noted that the boundary need not be determined by using Figure 4 The contour tracing algorithm in step S30 is determined, and may be found in any other way.

[0092] For example, Figure 7B The boundary data generator module 14 is shown in FIG. Figure 4 An alternative method for determining the boundary of the above-mentioned region in the LH sub-band image of level 5 is performed in step S30 .

[0093] exist Figure 7B In step S32-2, for each of the multiple columns of pixels in the LH sub-band image of level 5, the boundary data generator module 14 determines the pixel position of the pixel in the column, at which the difference between the pixel value of the pixel and the pixel value of the adjacent pixel in the column exceeds a predetermined threshold.

[0094] exist Figure 7B In step S34-2, the boundary data generator module 14 also uses the determined pixel positions to define line segments in the LH sub-band image of level 5. The line segments define the boundaries of the aforementioned regions in the LH sub-band image. A pixel position (at which the difference between the pixel value of the pixel and the pixel value of an adjacent pixel in the column exceeds a predetermined threshold) can be determined for each of the plurality of columns of pixels in the LH sub-band image, for example, by evaluating the difference of consecutive pixels in the column, starting from the pixel at the bottom of the column. Furthermore, each line segment can be defined as connecting a pixel at a determined pixel position in a corresponding column of the plurality of columns of pixels to a pixel at a determined pixel position in an adjacent column of the plurality of columns of pixels.

[0095] Figure 8A Shown Figure 7B An example implementation of the method. Figure 8A In FIG. 8 , a plurality of pixel columns 805 extend across the image, which define the boundaries around dark regions 810 in the image. Although not shown in FIG. 8 for clarity, Figure 8A , but assuming that the areas of the image that are within the boundary 810 are significantly darker than the areas of the image that are outside the boundary 810. Figure 8A , for each column 805 selected for analysis, the change in pixel value between each pixel in the column 805 and the adjacent pixels in the same column 805 is evaluated. For each column 805, an open point 806 is designated (at the pixel location where the change in pixel value between adjacent pixels exceeds a negative threshold value, i.e., a transition point from a "bright" pixel to a "dark" pixel) Figure 8A In determining the on-point 806 in each column 805, the comparison of the pixel values of the adjacent pixels in the column is continued from the on-point 806 along the column, and the off-point is designated at the next pixel position where the change in pixel value between the adjacent pixels exceeds the positive threshold value (i.e., the transition point from a 'dark' pixel to a 'bright' pixel). Figure 8A(shown as black squares in the figure). When determining the opening point 806 and closing point 807 of each column 805, the opening points 806 in adjacent columns are connected using a broken line 820, and the closing points 807 in adjacent columns are connected using a broken line 830. In addition, for the columns 805 closest to the left and right boundaries of the LH sub-band image, the opening points 806 and closing points 807 can also be connected by line segments. In this way, Figure 8A As shown, a closed contour defined by polyline segments is generated. The closed contour defines the boundary of the unoccupied road area in the original image representing the scene of the LH sub-band image of level 5.

[0096] In some embodiments, the Figure 8B The order shown is selected Figure 7B The pixel column 805 in step S32-2. Figure 8B In Figure 7B The method is applied to the LH sub-band image, wherein a boundary is determined around a dark region 830 in the LH sub-band image, the dark region 830 representing an unoccupied road area in the scene image. Figure 8B , a plurality of columns 840 of the LH sub-band image are sequentially selected and analyzed. The order in which the columns 840 are analyzed is based on the number appearing above each column. Figure 8B As shown, the first pixel column selected for analysis can be a pixel column in the center region of the LH subband image, such that this pixel column divides the LH subband image into two halves. Then, two additional pixel columns are selected at horizontal coordinates in the LH subband image, such that each of the two halves is divided into two new, substantially equal regions. This process is then repeated. For example, for an image of size 255×255 pixels, pixel columns at the following x-coordinate locations can be selected at each step of the analysis process:

[0097] Step 1: 128;

[0098] Step 2: 64, 192;

[0099] Step 3: 32, 96, 160, 224;

[0100] Step 4: 16, 48, 80, 112, 144, 176, 208, 240.

[0101] exist Figure 8B, the order in which pixel columns 840 are analyzed is illustrated by the numbers above the illustrated pixel columns 840. It should be noted that the order in which pixel columns 840 are selected is by no means limited to this example. Furthermore, in some embodiments, when a significant difference is found between the y-values (i.e., y-axis coordinate values) of the open points (or closed points) in adjacent columns 840 selected for analysis, additional columns 840 may be selected for analysis between the two adjacent columns 840 to allow for determination of a more detailed boundary.

[0102] In some embodiments, the image of the scene can be a 360-degree field of view image of the road surrounding the vehicle. Such an image can be constructed by combining images of the road acquired from forward-facing, side-facing, and rear-facing cameras mounted on the vehicle. In such embodiments, the boundaries of regions representing unoccupied areas of the road can be determined from sub-band images of the 360-degree image of the road by analyzing variations in pixel values between adjacent pixels in each of a plurality of selected pixel lines extending from the center of the sub-band image to the edge of the sub-band image. For example, if the pixels of the sub-band image are identified by coordinates in a polar coordinate system, each of the selected pixel lines can be extended from the center coordinates of the image, and each pixel line can correspond to a different polar angle.

[0103] In some embodiments, when the image processed by the unoccupied road area determination module 10 also includes a portion of the sky, the unoccupied road area determination module 10 may exclude the portion representing the sky from processing the image in order to simplify the search for unoccupied areas of the road. Specifically, the unoccupied road area determination module 10 may perform the search described herein only for the portion below the horizon of the image. Figure 4

[0046] In the process described, the horizon can be defined as a horizontal line across the image, the placement of which in the image will depend on the tilt of the camera and can be set by examining the images acquired by the camera.

[0104] Furthermore, in some embodiments, the boundary data generator module 14 may be arranged to apply a binary thresholding operation to the LH subband image of level 5 of the DWT decomposition (or the HL or HH subband image of level 5, as the case may be) before determining the boundary of the area representing the unoccupied road area in the LH subband image (or the HL or HH subband image of level 5, as the case may be). Applying a binary thresholding operation to the LH subband image may allow for more reliable determination of dark areas corresponding to unoccupied road areas from the LH subband image.

[0105] In some embodiments, prior to determining the boundary of the region representing the unoccupied road area, nonlinear preprocessing may be performed to normalize the level 5 LH subband image. More specifically, the boundary data generator module 14 may be configured to apply a scaling factor to the pixel value of each pixel of the LH subband image and saturate scaled pixel values above a pixel saturation threshold before generating boundary data (representing the boundary of the region representing the unoccupied road area of the image). For example, the scaling factor may be multiplied by the pixel value of each pixel of the level 5 LH subband image, and the scaled pixel values may be converted to 8-bit grayscale values, saturating scaled pixel values above 255. This scaling followed by saturation process has the effect of normalizing the LH subband image and improving its contrast, thereby making the subsequent boundary determination process more robust to varying lighting conditions (i.e., varying contrast, brightness, and noise). Furthermore, normalization can be effectively implemented via a simple multiplication operation, which is computationally more efficient than performing normalization using a nonlinear filter.

[0106] Once the boundary data generator module 14 determines the boundaries of the areas in the LH sub-band image corresponding to the unoccupied road areas, the determined boundaries can be enlarged and mapped to the input image or the sub-band image of the previous level (i.e., the lower value level) of the DWT decomposition to facilitate the extraction of other features or objects in the input image or the sub-band image, as will be described in more detail below.

[0107] Figure 9A is an example image of a road, Figure 9B and Figure 9C It is through Figure 4 The sub-band image is generated by processing the image in steps S10 and S20. More specifically, Figure 9A is a grayscale image 910 of a scene including a vehicle in front of the camera and an unoccupied area of the road between the vehicle carrying the camera and other vehicles. Figure 9B yes Figure 9A The LL subband image 920 of the fourth level of the DWT decomposition of the input image 910 is obtained by Figure 4 The processing of step S10 in [ ] is performed and a low-pass filter is generated using symmetric filter coefficients [1, 6, 15, 20, 15, 6, 1] based on the values of the rows of the Pascal triangle. Figure 9B As can be seen in , the iterative low-pass filtering and downsampling process generates sub-band images with reduced resolution, in which the high spatial frequency components corresponding to small-sized features in the input image are essentially removed, leaving only the low-frequency components. Figure 9C yes Figure 9A The LH subband image 930 of the fifth level of DWT decomposition of the input image 910 is shown. The LH subband image 930 is obtained by Figure 4 Generated in processing S20. Figure 9C The LH subband image 930 shown in FIG is generated using Figure 9B The 4th level LL subband image 920 in the image shown in is generated by using the same low pass filter and by using a high pass filter with filter coefficients [1, -6, 15, -20, 15, -6, 1]. Figure 9C The image shown also shows the boundary 931 of the dark area 932 of the LH subband image 930 of level 5, which has been Figure 4 The boundary determination process performed in step S30 discovered this boundary 931. Dark area 932 represents the road area occupied in the image acquired by the vehicle camera. In addition, in this example, both the low-pass filter and the high-pass filter are normalized by a factor of 1 / 64. In this example, normalization can be implemented as a separate operation from the filtering operation, as described in the previous example.

[0108] 10A to 10C are images with different contrast, brightness, and uneven brightness of the road, and 10D to 10F The above references are shown respectively Figure 4 The described process is applied to images 10A to Figure 10C results. Figure 10A is an image 1010 of a scene with high contrast and low brightness, and Figure 10D The LH subband image 1040 of level 5 of the DWT decomposition is shown, which is obtained by performing Figure 4 Step S20 and use in combination with the above Figure 9A 、 Figure 9B and Figure 9C The same filter described from Figure 10A The image 1010 in is generated. Figure 10B is an image 1020 of a scene with high contrast and low brightness, and Figure 10E Shown by executing Figure 4 The processing of step S20 and the use of the above Figure 9A 、 Figure 9B and Figure 9C The same filter described from Figure 10B The LH subband image 1050 of the 5th level of DWT decomposition is generated from the image 1020. Figure 10C is an image 1030 of a scene in which a road includes road lanes having different brightness. Figure 10F The LH subband image 1060 of level 5 of the DWT decomposition is shown, which is obtained by performing Figure 4 Step S20 and using the above Figure 9A 、 Figure 9B and Figure 9C The same filter from Figure 10CThe image 1030 shown is generated. Although the brightness level on the road varies, due to the high pass filtering performed at the 5th level of the DWT decomposition, Figure 10F The LH subband image 1060 represents Figure 10C The unoccupied road area in the image appears as a uniform dark area.

[0109] Although Figures 9A to 9C and 10A to 10F In the example of , the dark area corresponding to the unoccupied area of the road is determined from the LH sub-band, but alternatively, the area can be determined from the HL or HH sub-band with similar effect. Figures 9A to 9C and 10A to 10F As shown, when the image being processed is of the road ahead of a vehicle and has been captured by a forward-facing camera mounted at a relatively low height, the LH subband is suitable for identifying unoccupied road areas. In this case, the left and right boundaries of the road (e.g., the boundaries separating the road from adjacent road surfaces or road guardrails) are substantially horizontal in the acquired image. By applying a high-pass filter along the columns of the LL subband image at level 4 of the DWT decomposition, these substantially horizontal boundaries are highlighted as white areas in the LH subband at level 5 of the DWT decomposition. As a result, dark regions representing unoccupied road areas can be easily detected in the LH subband image. Because the characteristics of the image being processed can vary (e.g., depending on the curvature of the road, the position of the vehicle relative to the road boundary, or the height of the onboard camera), the orientation of the road boundary containing the unoccupied road area can also vary. Therefore, in some cases, it may be more appropriate to use the HL subband (which emphasizes vertical features in the image) or the HH subband (which emphasizes diagonal features in the image) to locate dark regions corresponding to unoccupied road areas.

[0110] More generally, applying a high-pass filter to the LL subband image at a given level of the DWT decomposition of an image (obtained by iteratively low-pass filtering and downsampling the image as described above) can effectively identify unoccupied areas of road. This is because the low-pass filtering operation tends to smooth that area of the image, resulting in pixels having a narrower range of pixel values than the rest of the image, thereby allowing that area to be easily distinguished from the rest of the image by the high-pass operation, which results in that area having more uniform and lower pixel values than the rest of the image. Furthermore, the downsampling performed during the iterative process for generating the LL subband image significantly reduces the amount of data to be processed when determining region boundaries, thereby effectively identifying unoccupied areas of road.

[0111] Note that although Figures 9A to 9C and 10A to 10FThe examples in are based on images captured by a forward-facing camera mounted on a vehicle, but the image processing methods described above can alternatively be applied to road images captured by side-facing or rear-facing cameras mounted on the vehicle. Rear-view and side-view images are often captured to construct a vehicle's road occupancy map, which is particularly useful, for example, to inform the driver of the presence of objects in the vehicle's blind spot. However, the rear-view and side-view cameras used on vehicles are typically wide-angle cameras that introduce high levels of distortion into the captured images. Therefore, it is often necessary to correct the acquired images for distortion before the resulting distortion-corrected images can be seamlessly merged to generate the vehicle's road occupancy map.

[0112] However, conventional methods for generating such road occupancy maps are often processor-intensive due to the high complexity of the distortion correction and seamless merging operations, which require performing an affine transformation on all pixels in the image.

[0113] The unoccupied road area determination module 10 overcomes the above-mentioned shortcomings of conventional methods for generating road occupancy maps and allows the generation of road occupancy maps in a computationally inexpensive manner. More specifically, when an image with a high level of distortion is to be processed to generate a road occupancy map, the unoccupied road area determination module 10 may first determine the road occupancy map according to the above reference. Figure 4 The image is processed using the method of the described embodiment (or one of the described variants) to generate boundary data representing the boundaries of the unoccupied road area represented by the (distorted) image. An affine transformation for correcting the distortion can then be applied only to the boundary data (either the original generated or a scaled version of the original generated) rather than to every pixel of a sub-band or input image, thereby allowing the unoccupied road area in the image to be accurately determined with significantly reduced computational complexity. The distortion-corrected boundary data can then be used to construct a road occupancy map for the vehicle. For example, the unoccupied road area determination module 10 can combine the boundary data obtained for each of the multiple images to generate a road occupancy map.

[0114] See again Figure 1 In this embodiment, the unoccupied road area determination module 10 may include a refined boundary data generator module 16, which is arranged to generate refined boundary data representing the boundary of the area representing the unoccupied area of the road in the input image. Specifically, in some cases, Figure 4The boundary of the area representing the unoccupied road area determined in step S30 may provide an overly conservative estimate of the boundary of the unoccupied road area, such that the determined boundary lies within the actual boundary of the unoccupied road area in the input image (as determined by visually inspecting the input image). Therefore, it may be desirable to refine the image in the input image by using one or more sub-band images obtained at a lower value level of the DWT decomposition (i.e., a level of the DWT decomposition that provides a higher resolution sub-band image containing more detailed / higher spatial frequency components). Figure 4 The boundary data determined from the (N+1)th level sub-band image in step S30.

[0115] Figure 11 The method performed by the refined boundary data generator module 16 is shown. Figure 11 In step S110, the refined boundary data generator module 16 generates a refined boundary by high-pass filtering the LL subband image of the Pth level (where P is an integer less than N) of the (N+1)th level DWT decomposition of the image and down-sampling the high-pass filtering result of the Pth level LL subband image to generate a second subband image, which is the subband image of the (P+1)th level of the (N+1)th level DWT decomposition of the image. Figure 4 The LL subband image of level P can be generated using the same type of filters and the same filtering sequence as described in step S10 of generating the LL subband of level N. Similarly, the subband image of level (P+1) can be generated using the same type of filters and the same filtering sequence as described previously in step S10 of generating the LL subband of level N. Figure 4 The same type of filter and the same filtering sequence used to generate the sub-band of the (N+1)th level described in step S20 are used to generate the sub-band.

[0116] exist Figure 11 In step S120, the refinement boundary data generator module 16 uses a scaling factor of 2 T (where T=NP) The determined boundary of the region of the (N+1)th level subband image is enlarged to generate an enlarged boundary, and the enlarged boundary is mapped to the second subband image to generate a second boundary, which is in the second subband image.

[0117] exist Figure 11In step S130, the refined boundary data generator module 16 processes the second sub-band image using the determined second boundary to generate data representing a refined boundary of a second region in the second sub-band image as refined boundary data, the second region containing (including) the second boundary and representing an unoccupied area of a road in the image. As in the present embodiment, the data representing the boundary of the second region can be generated by extending (or modifying) the determined second boundary to further define at least one region in the second sub-band image defined by adjacent pixels having substantially equal pixel values. The pixels in the at least one pixel region can be "substantially equal" to each other in the sense that the pixel value spread of the pixel values of these pixels (e.g., as quantified by their range, variance, or standard deviation) is smaller than the (same defined) pixel value spread of the pixels forming the remainder of the second sub-band image.

[0118] In this example, the values of N and P are chosen to be 4 and 2 respectively, and Figure 11 The filtered subband image of the (P+1)th level generated in step S110 is the LH subband image of the 3rd level of the DWT decomposition (also referred to herein as the '3rd level LH subband image' or the '3rd DWT level LH subband image'). However, the value of P may be selected to be any integer value less than the value of N. In addition, Figure 11 The second sub-band image in step S110 is not limited to being an LH sub-band image, but may be any high-pass filtered sub-band image (including an HL sub-band image or an HH sub-band image) calculated at a level of DWT decomposition lower than the level of DWT decomposition at which the initial boundary corresponding to the unoccupied road area is determined (at Figure 4 in step S30).

[0119] exist Figure 11 In step S120, the refined boundary data generator module 16 enlarges the boundary of the area (representing the unoccupied road area) in the LH sub-band image of the 5th level of DWT decomposition by 4 times to generate an enlarged boundary, and then Figure 11 In step S120, the image is mapped to the LH subband image of level 3 of the DWT decomposition as the second boundary. In this embodiment, the boundary of the region in the LH subband of level 5 of the DWT decomposition can be scaled up by multiplying the x-axis coordinates and y-axis coordinates of the pixels defining the boundary by a factor of 4. It should be noted that since downsampling is performed at each level of the two-dimensional multi-level DWT decomposition, a scaling factor of 4 is used in this example because it represents the scaling difference between the resolution of the LH subband of level 3 of the DWT decomposition and the resolution of the LH subband of level 5 of the DWT decomposition.

[0120] exist Figure 11In step S130, in this example, the refined boundary data generator module 16 may use the enlarged boundary in the level 3 LH subband image and the level 3 LH subband image itself to extend the enlarged boundary to further define an area in the level 3 LH subband image that spans or touches the enlarged boundary and is defined by pixels having pixel values below a predetermined threshold. In other words, as part of step S130, the enlarged boundary in the level 3 LH subband image of the DWT decomposition is extended to include additional dark regions adjacent to the enlarged boundary to form the refined boundary of step S130 that includes the enlarged boundary. In this way, the enlarged boundary obtained by scaling the boundary discovered using the level 5 LH subband image of the DWT decomposition is refined or extended using the more detailed information available in the level 3 LH subband image of the DWT decomposition. The refined boundary determined in the level 3 LH subband image thus more accurately reflects the boundary of the area representing the unoccupied road area in the original (input) image of the road.

[0121] Figure 12 It is shown that it can be used as Figure 11 Flowchart of a sequence of steps performed as part of step S130 of FIG. 1 to generate a thinned (or expanded) boundary in the level 3 LH sub-band image of the aforementioned example.

[0122] exist Figure 12 In step S210, the refined boundary data generator module 16 determines a cluster of adjacent pixels in the level 3 LH sub-band image that touch or cross the enlarged boundary, wherein the pixels in the cluster have substantially equal pixel values. In this example, without performing an addition process in addition to the DWT decomposition step as described above, the area of the level 3 LH sub-band image representing the unoccupied road area will be substantially dark (having low pixel values). Thus, the pixels in the cluster will have pixel values that are much lower than those of the pixels adjacent to the cluster.

[0123] exist Figure 12 In step S220, the refined boundary data generator module 16 generates image data of an enlarged LH sub-band image of level 5 by enlarging a portion of the LH sub-band image of level 5 with the same scaling factor as used in step S120, the enlarged LH sub-band image having the same image size as the LH sub-band image of level 3, which is 4 in this example. In this way, each pixel in a cluster in the LH sub-band image of level 3 has a corresponding pixel in the enlarged portion of the LH sub-band image of level 5. The enlargement may be performed using any suitable image scaling algorithm (e.g., nearest neighbor interpolation or bilinear interpolation).

[0124] Furthermore, the magnified portion of the (N+1)th level sub-band image is determined based on the positions of the pixels of the cluster in the second sub-band image and the mapping between the position of each pixel of the second sub-band image and the position of the pixels of the (N+1)th level sub-band image. More specifically, in this example, for each pixel in the cluster in the level 3 LH sub-band image, the corresponding pixel of the level 5 LH sub-band image is determined using the mapping and magnified to generate image data for the magnified level 5 LH sub-band image. Due to the resolution difference between the level 3 LH sub-band image and the level 5 LH sub-band image, the mapping between the position of each pixel in the level 3 LH sub-band image and the position of the corresponding pixel in the level 5 LH sub-band image can be a many-to-one mapping, as in this example. Thus, when multiple pixels in the cluster in the level 3 LH sub-band image are mapped to the same pixel in the level 5 LH sub-band image, that pixel in the level 5 LH sub-band image only needs to be magnified once.

[0125] exist Figure 12 In step S230 , the refined boundary data generator module 16 evaluates statistics of pixels in the magnified portion of the LH sub-band image of level 5 and compares the evaluated statistics with a predetermined value.

[0126] Then, in Figure 12 In step S240 , the refined boundary data generator module 16 determines whether to extend the enlarged boundary mapped to the level 3 LH sub-band image to further define clusters of adjacent pixels in the level 3 LH sub-band image based on a comparison between the evaluated statistic and a predetermined value.

[0127] As in this embodiment, Figure 12 The evaluation of the statistics of the pixels in the magnified portion of the level 5 LH sub-band image, performed in step S230, may include evaluating a proportion of pixels in the magnified portion of the level 5 LH sub-band image having pixel values below a predetermined pixel value. Furthermore, as part of step S240, in this embodiment, the refined boundary data generator module may compare the evaluated proportion with a predetermined threshold. In this embodiment, upon determining that the evaluated proportion exceeds the predetermined threshold, the refined data generator module may extend the magnified boundary in the level 3 LH sub-band image to include the dark pixel cluster. However, if the evaluated proportion is below the predetermined threshold, the magnified boundary in the level 3 LH sub-band is not extended to include the adjacent cluster.

[0128] It should be noted that in Figure 12 The statistic evaluated in step S230 is not limited to the proportion of pixels having pixel values below a predetermined pixel value, and alternatively, another statistical measure may be used to determine whether the enlarged boundary should be extended to include the cluster. For example, Figure 12The evaluation in step S230 may alternatively or additionally include evaluating a mean value of pixel values in the enlarged portion of the level 5 LH sub-band image, and step S240 may further include extending the enlarged boundary in the level 3 LH sub-band image to include the dark pixel cluster only if the evaluated mean value is below a predetermined threshold value for the mean value. Furthermore, in some embodiments, the evaluation in step S240 may alternatively or additionally include evaluating a variance of pixel values in the enlarged portion of the level 5 LH sub-band image, and step S240 may further include extending the enlarged boundary in the level 3 LH sub-band image to include the dark pixel cluster only if the evaluated variance is below a predetermined threshold value for the variance.

[0129] In this embodiment, when the LH sub-band image of the third level is used to determine the refinement boundary, the refinement boundary can be enlarged to the sub-band image of the second level of DWT decomposition, and the enlarged boundary in the sub-band image of the second level is used in combination with the sub-band image of the second level to repeat the refinement. Figure 12 The enlarged boundary in the level 2 sub-band image is further refined or extended using a method similar to that described for the enlarged boundary in the level 3 LH sub-band image. That is, the refinement of the enlarged boundary in the level 2 sub-band image can be performed by determining a cluster of dark pixels that touches the enlarged boundary in the level 2 sub-band image, evaluating statistics of corresponding pixels in the enlarged portion of the level 3 LH sub-band image, and determining, based on the evaluation, whether to extend the enlarged boundary in the level 2 sub-band image to include the cluster of dark pixels. In this manner, the boundary refinement process can be iteratively repeated using the sub-band images of the low-value levels of the DWT.

[0130] In some embodiments, when the onboard camera is arranged to capture an image of a scene including a road and a horizon as the image, the determination of whether to expand the second boundary to further define the cluster of adjacent pixels in the second sub-band image may be further based on the vertical distance of the cluster from the horizon in the image (i.e., the distance along the y-axis in the image). In particular, a cluster of dark pixels close to the horizon is more likely to contain erroneous data and therefore may not be used to expand the boundary.

[0131] In some embodiments, when the image acquired by the vehicle-mounted camera is a road segment containing one or more road markers, Figure 4 The initial boundary determined at step S30 from the (N+1)th level subband of the DWT decomposition (in this example, the 5th level LH subband) or Figure 11The refined boundary determined in step S130 from the (P+1)th level sub-band (the 3rd level LH sub-band in this example) of the DWT decomposition (or the further refined boundary obtained by processing the 2nd level as described above) can be used to define a search area within the DWT decomposed sub-band image, within which a search for boundary data representing the boundary around the region of the input image containing the road marker can be performed. In particular, using the DWT decomposed sub-band image to search for boundary data representing the boundary of the road marker region in the road image can allow the computational complexity of the search to be reduced because the sub-band image has a lower resolution compared to the original (input) image. In addition, by determining the boundary of the region representing the unoccupied road in the (N+1)th level sub-band image and using the determined boundary (or the boundary according to Figure 11 Using the refined boundaries obtained in the step (i.e., the refinement of the image) to define a search region in the sub-band image within which the search for road marking regions is performed can make the search more efficient. This is because the search for road marking regions is limited to regions of the image representing unoccupied road areas, allowing available processing resources to be focused on searching only for portions of the image where road marking regions are likely to be located.

[0132] Therefore, in this embodiment, Figure 1 The apparatus in may comprise a road marking determination module 20 arranged to process an input image to determine boundary data representing boundaries of road marking regions in the image of a road.

[0133] Figure 13 It is shown by Figure 1 Flowchart of a method executed by the road sign determination module 20. In step S310, the road sign determination module 20 performs high-pass filtering on the Mth-level LL subband image of the (N+1)th-level DWT decomposition of the image, and downsamples the high-pass filtered result of the Mth-level LL subband image to generate the (M+1)th-level subband image of the DWT decomposition, where M is an integer less than the value of N.

[0134] exist Figure 13 In step S320, the road marker determination module 20 enlarges the boundary determined from the (N+1)th level sub-band image by 2 D The search area in the (M+1)th level sub-band image is determined by multiplying D by NM, where D=NM, and the enlarged boundary is mapped to the (M+1)th level sub-band image. The search area is located within the enlarged boundary mapped to the (M+1)th level sub-band image.

[0135] exist Figure 13In step S330, the road marker identification module 20 determines boundary data for a boundary of a pixel region within a search region in the (M+1)th level sub-band image, wherein the region is surrounded by pixels having pixel values substantially different from the pixel values of pixels in the region, and wherein the boundary of the region represents a boundary of a road marker region in the road image. More specifically, in this embodiment, the pixel values of the pixels in the region in the (M+1)th level sub-band image may be significantly higher than the pixel values of adjacent pixels adjacent to the region. Furthermore, the average pixel value of the region may be substantially higher than the average pixel value of adjacent pixels in the region.

[0136] exist Figure 13 In step S310, the same method as previously described can be used. Figure 4 The LL subband image for the Mth level can be generated using the same type of filter and the same filter sequence as described in step S10 for generating the LL subband image for the Nth level. More specifically, the LL subband image for the Mth level can be generated using one or more low-pass filters having a symmetrical sequence of filter coefficients, and the coefficients can also include integer-valued coefficients. Furthermore, the coefficients of the low-pass filters can be set to values in a row of the Pascal triangle in the same manner as described above.

[0137] In addition, for step S310, the same Figure 4 The (M+1)th level subband is generated using the same type of filter and the same filter sequence as described in step S20 for generating the (N+1)th level subband. Specifically, the (M+1)th level subband can be generated using one or more filters having a symmetrical filter coefficient sequence, and the coefficients can also include integer-valued coefficients. In addition, the filter can be designed using the values of the rows of the Pascal triangle in the same manner as in the aforementioned embodiment.

[0138] Therefore, about Figure 4 All previous examples and embodiments described for the generation of the LL subband of the Nth level in step S10 are also applicable to the Figure 13 The generation of the LL subband of the Mth level processed in step S310 is similarly Figure 4 All previous examples and embodiments described for the generation of the (high-pass filtered) sub-band of the (N+1)th level in step S20 also apply to Figure 13 Generation of the (M+1)th level sub-band in step S310.

[0139] Figure 14A An image of a section of a road is shown having a road marker 1410 in the example form of a bodega. However, the road marker 1410 may be of a different type, such as a cat's eye reflective road stud. In this example, Figure 14A The images in the above reference Figure 13 The steps described by Figure 1 The road sign determination module is processed.

[0140] Figure 14B Demonstrates that by executing Figure 13 The step S310 generates Figure 14A The LH subband image of the 3rd level DWT decomposition of the image is generated using a low-pass filter with filter coefficients [1, 6, 15, 20, 15, 6, 1] and a high-pass filter with filter coefficients [1, -6, 15, -20, 15, -6, 1]. However, the (M+1)th level subband image generated in step S310 is not limited to the LH subband image and may alternatively be selected as a HL subband image or an HH subband image. The selection of a specific subband may be based on the expected shape or orientation of the road sign in the image captured by the camera. In this example, the road sign 1410 in the image is a Bott point, such as Figure 14A As shown in FIG, the bt point appears as a substantially elliptical object elongated in the horizontal (x) direction in the image. Therefore, a high-pass filter is applied along the columns of the LL sub-band image of level 4 to generate the LH sub-band image of level 3 for highlighting the horizontal features in the input image and simultaneously removing the low-frequency components corresponding to the unoccupied road area, so that the road marker 1410 is Figure 14B The LH sub-band image appears as a bright object 1420. Thus, the LH sub-band image is particularly suitable for determining the location of the Bott point.

[0141] In this example, Figure 13 In step S320, the road marker determination module 20 enlarges the boundary of the area representing the unoccupied road area found using the LH subband of level 5 of the DWT by 4 times (in Figure 4 In step S30 of ), in order to generate an enlarged boundary, the enlarged boundary is then mapped to the LH sub-band image of the third level (which is in Figure 13 (Generated at step S310 of FIG5 ). As in this example, the boundary of the area in the LH sub-band image of level 5 (representing the unoccupied road area) can be enlarged by multiplying the x and y coordinates of the pixels defining the boundary by a factor of 4. The enlarged boundary mapped onto the LH sub-band image of level 3 defines a search area within which a search is performed for the boundary of the area representing the road marking boundary.

[0142] Figure 14B An enlarged boundary 1430 is shown, which is obtained by enlarging the coordinates of the boundary found on the LH subband of level 5 and mapping the enlarged boundary onto the LH subband image of level 3 of the DWT. Figure 14BIn FIG, the search area 1440 is defined as the area of the LH subband image that is within the enlarged boundary 1430. It should be noted that although the search area 1440 in this example is defined by directly enlarging the boundary found using the LH subband of level 5, in some embodiments, the enlarged boundary 1430 can be defined according to the above reference. Figure 11 The described steps are further refined or extended by the refined boundary data generator module 16. The refined, enlarged boundary can then be used to define a search area in the level 3 LH sub-band image.

[0143] exist Figure 13 In step S330, in this embodiment, the road sign determination module 20 can use Figure 4 The boundary of the region in the level 3 LH sub-band image representing the boundary of the road sign is determined in a similar manner to the method for determining the boundary of the region representing the unoccupied road region described in step S30. More specifically, in Figure 4 In step S30, a boundary is determined around the substantially dark areas of the level 5 LH sub-band image. In contrast, each road marking 1410 in the input image is represented by a substantially brighter area / point 1420 in the level 3 LH sub-band image. Therefore, reference is made here to Figure 7A The described algorithm can be easily adapted to detect boundaries representing road marking boundaries in the level 3 LH sub-band images.

[0144] More specifically, in this example, determining the boundary representing the road marker boundary in step S330 is performed by the road marker determination module 20 first determining the pixel locations of pixels of the level 3 LH sub-band image within the region of the level 3 LH sub-band image located within the magnified boundary whose pixel values exceed a predetermined pixel threshold. This pixel location can be determined iteratively by examining each pixel of the LH sub-band image until a pixel whose pixel value falls below the predetermined pixel threshold is determined. After determining the pixel locations, the road marker determination module 20 further uses the determined pixel locations to perform a contour tracing algorithm to identify the boundary of the region representing the road marker boundary. Specifically, the boundary of the region separates pixels of the region adjacent to the boundary and having pixel values above the predetermined pixel threshold from pixels outside the boundary adjacent to the boundary and having pixel values above the predetermined threshold. Any suitable boundary / contour tracing algorithm can be used in this regard, such as moiré neighborhood tracing, radial scanning, and square tracing algorithms.

[0145] In some embodiments, when determining the boundary representing the road marker boundary from the Level 3 LH sub-band image, the road marker determination module 20 may further define the boundary of the road marker region in the input image captured by the camera. Specifically, the road marker determination module 20 may magnify the boundary representing the road marker region in the Level 3 LH sub-band image by 8 times (corresponding to the different ratios between the resolution of the original road image and the resolution of the Level 3 LH sub-band image) to generate an enlarged road marker boundary, which is then mapped onto the image as the defined boundary of the road marker region.

[0146] Road Marker Determination Module 20 may also be configured to determine whether the portion of the image that is within the defined boundaries of the road marker region represents a road marker on the road. For example, Road Marker Determination Module 20 may associate the portion of the image that is within the defined boundaries of the road marker region with one or more stored images of road markers. Alternatively, Road Marker Determination Module 20 may input the portion of the image that is within the defined boundaries of the road marker region into a trained statistical classifier (e.g., a convolutional neural network (CNN)).

[0147] In some embodiments, nonlinear preprocessing may be performed to normalize the level-3 LH subband image prior to determining a boundary representing a road marking boundary. More specifically, the road marking determination module 20 may be arranged to apply a scaling factor to the pixel value of each pixel of the LH subband image and saturate scaled pixel values above a pixel saturation threshold before generating a boundary representing a road marking region in the level-3 LH subband image. For example, the scaling factor may be multiplied by the pixel value of each pixel of the level-3 LH subband image, and the scaled pixel values may be converted to 8-bit grayscale values, saturating scaled pixel values above 255. The scaling and subsequent saturation process has the effect of normalizing the LH subband image and improving its contrast, thereby making the subsequent boundary determination process more robust to varying lighting conditions (i.e., varying contrast, brightness, and noise). Furthermore, the normalization can be effectively implemented via a simple multiplication operation, which is computationally more efficient than performing the normalization using a nonlinear filter.

[0148] Furthermore, in some embodiments, the road marking determination module 20 may be arranged to apply a binary thresholding operation to the Level 3 LH sub-band image before determining the boundaries of the region of the LH sub-band image representing the boundaries of the road marking region. Applying a binary thresholding operation to the LH sub-band image may allow bright regions corresponding to road marking regions to be more accurately determined from the LH sub-band image.

[0149] It should be noted that although in this example the boundary found in the LH subband of level 3 is scaled up to the input image, the boundary could alternatively be scaled up to a subband of level 1 or any DWT level as long as the level is higher than the level at which the boundary representing the road sign boundary was initially found.

[0150] In having Figure 1 In the embodiment shown of both the refined boundary data generator module 16 and the road marker determination module 20, and wherein Figure 11 The value of P in step S110 is Figure 13 The value of M in step S310 is the same as that in step S310. If Figure 11 If the step S110 is executed, there is no need to execute the step S310 again. Instead, the output of the step S110 can be reused as the output of the step S310.

[0151] It should be noted that although the road sign determination module 20 in this embodiment uses the boundary found from the level 5 LH sub-band image to define the search area in the level 3 LH sub-band image, in some embodiments, the road sign determination module 20 can directly search for the boundary representing the boundary of the road sign area in the level 3 LH sub-band image, without initially using the boundary found in the level 5 LH sub-band image (the boundary representing the unoccupied road area) to define the search area.

[0152] That is, in some embodiments, the road marker determination module 20 may act as an independent image data processor, which may be independent of Figure 1 When operating independently, the road marker determination module 20 may be arranged to process an image of a scene including a road with road markers, the image being acquired by a camera mounted on a vehicle, to generate boundary data representing a boundary of a road marker area representing road markers in the image.

[0153] Specifically, when used as a standalone image data processor, the road marking determination module 20 includes a discrete wavelet transform (DWT) decomposition module that is arranged to generate an M-th level LL subband image of an (M+1)-level discrete wavelet transform (DWT) decomposition of the image by performing an iterative process of iteratively low-pass filtering and downsampling the image M times, where M is an integer equal to or greater than 1. The DWT decomposition module is further arranged to generate an (M+1)-th level subband image of an (M+1)-level DWT decomposition of the image by high-pass filtering the M-th level LL subband image and downsampling the result of the high-pass filtering.

[0154] Furthermore, the road marker determination module 20 further includes a boundary data generator module configured to generate boundary data by determining a boundary of a pixel region of the (M+1)th level sub-band image. The pixel region is surrounded by pixels having pixel values substantially different from the pixel values of pixels in the region. The boundary of the region represents a boundary of a road marker region in the road image.

[0155] The DWT decomposition module of the road sign determination module 20 (when used as an independent image data processor) can be arranged to generate the LL sub-band image of the Mth level in the same manner as the DWT decomposition module 12 of the unoccupied road area determination module 10 is arranged to generate the LL sub-band image of its Nth level. In addition, the DWT decomposition module of the road sign determination module 20 (when used as an independent image data processor) can be configured to generate the sub-band image of the (N+1)th level in the same manner as the DWT decomposition module 12 of the unoccupied road area determination module 10 is arranged to generate the sub-band image of its (N+1)th level. Figure 4 All embodiments, examples and variations described in steps S10 and S20 of FIG. 1 are also applicable to the DWT decomposition module of the road sign determination module 20 (when acting as an independent image data processor). Figure 13 In step S330 , the boundary data generator module of the road marking determination module 20 (when used as a standalone image data processor) may be arranged to generate boundary data representing the boundary of the road marking area using the same method as described above.

[0156] In addition, the previous Figure 1 All embodiments and variants described in the road sign determination module 20 are embodiments and variants of the road sign determination module 20 acting as an independent image data processor, with the only difference being that the boundary representing the boundary of the road sign area is determined in the (M+1)th level sub-band image, instead of using the boundary found from the lower level sub-band image to define the search area in the (M+1)th level sub-band image.

[0157] Device 1 can determine the vehicle's speed using information derived from images captured by an onboard camera as the vehicle travels along a road. Device 1's determination of vehicle speed can be performed in parallel with the vehicle's conventional vehicle speed measurement (based on the angular velocity of the vehicle's wheels). These independent measurements of vehicle speed can improve safety and are particularly important in autonomous driving.

[0158] like Figure 1As shown, the device 1 may also include a vehicle speed detection module 30 for determining the vehicle speed of the vehicle as it travels along the road by processing pairs of road images captured by an onboard camera. Each pair of images has a common image coordinate system and includes a first image of the road and a second image of the road. The first image is captured at a first moment, and the second image is captured at a second moment, which is a predetermined period of time after the first moment. The first image and the second image include corresponding road marking images of road markings (or lane markings) on the road.

[0159] Figure 15 Shown Figure 1 Components of the vehicle speed detection module 30 shown. Figure 15 As shown, the vehicle speed detection module 30 includes a position determination module 31, which is arranged to determine the position of the road marker image in the first image in the image coordinate system. The vehicle speed detection module also includes a position prediction module 32, which is arranged to predict the position of the road marker image in the second image in the image coordinate system using the determined position of the road marker image, the estimate of the vehicle speed, and the predetermined time period. The vehicle speed detection module 30 also includes a road marker image detection module 33, which is arranged to detect the image of the road marker at the predicted position in a portion of the second image. In addition, the vehicle speed detection module 30 includes a distance estimation module 34, which is arranged to estimate the distance traveled by the vehicle along the road during the predetermined time period using the determined position of the road marker in the first image and the position of the detected road marker image in the portion of the second image. The vehicle speed detection module 30 also includes a speed calculation module 35, which is arranged to calculate the speed of the vehicle based on the estimated distance and the predetermined time period.

[0160] Figure 16 Shown by Figure 15 The vehicle speed detection module 30 performs a method for determining the vehicle speed.

[0161] exist Figure 16 In step S510, the position determination module 31 determines the position of the road sign image in the first image captured by the camera in the image coordinate system. Figure 1 The method executed by the road marker determination module 20 determines the position of the road marker image in the first image.

[0162] More specifically, the position determination module 31 can, for example, generate an M-th level LL subband image of the (M+1)-th level discrete wavelet transform DWT decomposition of the first image by iteratively low-pass filtering and downsampling the first image M times, thereby determining the position of the road sign image in the first image, where M is an integer equal to or greater than 1. The position determination module 31 of this embodiment also generates an (M+1)-th level subband image of the (M+1)-th level DWT decomposition of the first image by high-pass filtering the N-th level LL subband image and downsampling the result of the high-pass filtering. In addition, the position determination module 31 can generate boundary data representing the boundary of the road sign area of the first image by determining the boundary of the pixel area of the (M+1)-th level subband image, the pixel area having substantially equal pixel values and surrounded by pixels having pixel values substantially different from the pixel values of the pixels in the area. The position determination module 31 of this embodiment also generates boundary data representing the boundary of the road sign area of the first image by determining the boundary of the pixel area of the (M+1)-th level subband image by 2 M+1 times to determine the position of the road sign image in the first image.

[0163] The location determination module 31 can be used in the same manner as previously described for Figure 1 The DWT decomposition is performed in the same manner as described for the road marker determination module 20 in and boundary data representing the boundary of the road marker area is generated. Specifically, the position determination module 31 can generate the LL subband of the Mth level in the same manner as the road marker determination module 20. In addition, the position determination module 31 can generate the subband of the (M+1)th level in the same manner as the road marker determination module 20. In addition, the position determination module 31 can generate boundary data in the same manner as the road marker determination module 20. Therefore, all embodiments and examples of the road marker determination module 20 described above are examples of the position determination module 31. However, it should be understood that it is not necessary to use the above method or one of its variants to determine the position of the road marker, and another method can be adopted alternatively.

[0164] exist Figure 16In step S520, the position prediction module 32 predicts the position of the image of the road marker in the second image in the image coordinate system. This prediction is based on the determined position of the image of the road marker in the first image, the estimate of the vehicle's speed, and a predetermined time period between the first time the first image was captured and the second time the second image was captured. For example, in an embodiment where the camera is a side-view camera that is arranged to capture images of the road to the side of the vehicle as the vehicle travels along the road as the first image and the second image, the position prediction module 32 may use a mapping, such as a linear mapping (e.g., in the form of a linear function or a lookup table that linearly relates input variables to lookup entries) to map the estimate of the distance traveled by the vehicle along the road (obtained by multiplying the estimate of the vehicle's speed by the predetermined time period) to a corresponding displacement in the image coordinate system. The mapping may be obtained in any suitable manner, such as by measuring distances between objects at different intervals in the camera's field of view and corresponding distances in the image coordinate system of the objects in the acquired camera images, and correlating the measured distances to obtain the mapping. The position of the road marking image in the second image (in the image coordinate system) may be predicted by combining the determined displacement with the determined position of the road marking image in the first image.

[0165] Therefore, the predicted position of the road marker image in the second image can be predicted using a mapping between a first variable and a second variable, where the first variable represents the position of a portion of the image in the image coordinate system and the second variable represents the distance of a portion of the road represented by the portion of the image from the vehicle (along the direction of travel of the vehicle).

[0166] In this embodiment, the camera is arranged to capture images of the road in front of the vehicle as a first image and a second image when the vehicle is traveling along the road, and in this case the mapping (between the first variable and the second variable) is a nonlinear mapping, which can be obtained in a manner similar to the above-mentioned linear mapping.

[0167] exist Figure 16 In step S530, the road marker image detection module 33 detects the image of the road marker in the portion of the second image where the predicted position is located. As in the present embodiment, the portion of the second image may be an area including the predicted position or having a predetermined spatial relationship with the predicted position in the second image. Furthermore, in the present embodiment, the portion of the second image is a subregion of the second image. By detecting the image of the road marker only in the portion of the second image where the predicted position is located (i.e., the subregion), rather than searching for the image of the road marker in the entire second image, the processing complexity of the road marker detection in the second image can be significantly reduced.

[0168] exist Figure 16In step S530, the method also used to determine the position of the road marker image in the first image is used, that is, by using the above reference Figure 1 Detection of the road marker image in the portion of the second image where the predicted position is located is performed using the method described in road marker determination module 20 to determine at least a portion of the boundary of the road marker image. However, it should be understood that detection of the road marker image in the portion of the second image may alternatively be performed using any suitable method, and such detection may more generally include applying any suitable method to detect the position of one or more pixels of the road marker image in the second image.

[0169] In an alternative embodiment, instead of determining the position of the road marker image in the second image by determining the road marker's boundary data as described above, the road marker detection module 33 may alternatively calculate a cross-correlation between a portion of the first image at the determined position and a portion of the second image at the predicted position. The result of the cross-correlation calculation can be used to detect and determine the position of the road marker image in the portion of the second image. In this embodiment, the position of the portion of the second image in the image coordinate system (which is correlated with the determined position of the portion of the first image) can be based on the predicted position of the road marker image (having a predetermined relationship with the predicted position of the road marker image). For example, the portion of the second image at the predicted position can be a subregion of the second image that includes the predicted position. By cross-correlating the portion of the first image at the determined position with only a portion (i.e., a subregion) of the second image at the predicted position, the complexity of the process of detecting the road marker in the second image can be significantly reduced because the cross-correlation is performed using pixels in the second image where the road marker is likely to be located, thereby avoiding the more computationally demanding task of cross-correlating the first and second images as a whole.

[0170] exist Figure 16In step S540, the distance estimation module 34 estimates the distance traveled by the vehicle along the road during the predetermined time period based on the determined positions of the road marker images in the first image and the determined positions of the road markers in the portion of the second image. More specifically, in this embodiment, the estimation of the distance traveled by the vehicle may be further based on a mapping between a first variable representing the position of the portion of the image in the image coordinate system and a second variable representing the distance of the portion of the road represented by the portion of the image from the vehicle (along the direction of travel). For example, in some embodiments, the distance estimation module 34 may use the determined positions of the road markers in the first image and the mapping to determine a first value representing the distance of the road marker at the determined position of the road marker image in the first image from the vehicle. The distance estimation module 34 may also use the determined positions of the road marker images in the second image and the mapping to determine a second value representing the distance of the road marker at the determined position of the road marker image in the second image from the vehicle. The distance estimation module 34 may then estimate the distance traveled by the vehicle along the road during the predetermined time period by calculating the difference between the first and second values.

[0171] exist Figure 16 In step S550 , the speed calculation module 35 calculates the speed of the vehicle based on the distance estimated in step S540 and the predetermined time period, specifically by dividing the estimated distance by the predetermined time period.

[0172] Figure 17A An image of a scene including a road captured by a forward-looking camera mounted on a vehicle is shown. Figure 17B is obtained by applying the transformation matrix to Figure 17A The top view image of the road is obtained by using the image in . Figure 17A and Figure 17B In the images, the (0, 0) coordinate is located at the lower left corner of each image. Figure 17B In the image, the distance between the point along the road and the vehicle increases as the Y-axis value representing the position of the point along the road in the image increases. Figure 17B The top view image is marked with a plurality of horizontal lines 172 spaced equally along the longitudinal axis of the top view image. Each horizontal line 172 is located at a corresponding position along the vertical axis, indicating a corresponding distance from the vehicle. Figure 17B In FIG, marks D1 , D2 , D3 and D4 represent four different distances from multiple points on the road to the vehicle, which are represented by four respectively marked horizontal lines.

[0173] Figure 17C Yes Figure 17B The road image is obtained by performing inverse perspective transformation on the marked top view image. Figure 17C in Figure 17BIn the top view image, the Y-axis position of the horizontal line 174 is obtained from the inverse perspective transformation of the Y-coordinate position of the horizontal line 172, so that Figure 17B The horizontal lines 172 at distances D1, D2, D3 and D4 from the vehicle are respectively mapped to Figure 17C Horizontal lines 174 at y-axis positions y1, y2, y3, and y4. Figure 17C As shown, the density of horizontal lines 174 is Figure 17C The portion of the image representing the road portion farther from the vehicle is higher, thus showing Figure 17B A nonlinear mapping between the Y-axis position of a portion of the image and the distance from the vehicle to the portion of the road represented by the image portion.

[0174] Figure 18 Show Figure 17C The Y-axis coordinate of the image in Figure 17C A mapping 1800 between a pixel at a Y-axis coordinate in an image representing a portion of a road and a distance from the vehicle. Figure 18 The mapping 1800 in FIG. 1 is an example of a mapping that may be used to perform step S520 and / or step S540 .

[0175] Figure 19A A first image 1910 including an image 1912 of a road marker is shown, which has been captured by a vehicle-mounted camera at a first moment in time. The vehicle-mounted camera is a front-facing (forward-facing) camera that captures images of the road in front of the vehicle. Figure 19B A second image 1920 is shown that includes the image 1922 of the same road marking, which is captured by the vehicle's onboard camera at a second moment in time when the vehicle is traveling forward.

[0176] exist Figure 19A and Figure 19B In the example of FIG. 1 , the position determination module 31 determines the coordinates (x ) of the central pixel of the road sign image 1912 included in the first image 1910. m ,y m ), as the determined position L1 of the road sign image 1912. In this example, the position of the center pixel can be determined based on the coordinates of the pixels defining the boundary of the road sign image 1912. However, the determined position (x m ,y m ) is not limited to the location of the center pixel of road marking image 1912 and may alternatively be defined by the coordinates of any predetermined pixel (e.g., the lowest, highest, leftmost, or rightmost pixel) of road marking image 1912. The determined location of the road marking image may alternatively be defined by the coordinates of a plurality of pixels that define at least a portion of the boundary of road marking image 1912.

[0177] exist Figure 16 In step S520, Figure 19A and Figure 19B In the example of FIG. 1 , the position prediction module 32 can use the determined position (x m ,y m ), an estimate of the vehicle speed, a predetermined time period between the first moment and the second moment, and Figure 18 The mapping shown in FIG1 is used to predict the position of the (same) road sign 1922 in the second image 1920. More specifically, in this embodiment, the position prediction module 32 may first use Figure 18 The Y-axis coordinate (x m ,y m ) to determine the distance d between the road marker represented by the road marker image 1912 and the vehicle along the road m (i.e., by using Figure 18 The polynomial function shown in the figure is used to determine the Y-axis coordinate value y m The position prediction module 32 can also determine the distance Δd traveled by the vehicle during the predetermined time period between taking the first image and taking the second image by multiplying the estimated vehicle speed by the predetermined time period between taking the first image and taking the second image. The position prediction module 32 can then use Figure 18 The mapping described in the figure converts the distance d m -Δd is mapped to the Y-axis position in the image coordinate system of the first image and the second image to predict the position of the road sign image in the second image 1920 (in the common image coordinate system of the first image 1910 and the second image 1920). Figure 19A As shown in the first image 1910, the predicted position L of the road sign image in the second image P At coordinate position (x m ,y p ). It should be noted that Figure 19A and Figure 19B In the example of , the predicted position of the road sign image 1922 in the second image 1920 only considers the expected offset of the Y-axis position of the road sign image 1912 in the first image 1910 .

[0178] exist Figure 19A and Figure 19B In the example of Figure 16 In step S530, the predicted position (x m ,y p ) in the portion 1930 of the road sign image. In this example, portion 1930 is shown at the predicted position (xm ,y p ) as its center. However, the shape of portion 1930 of second image 1920 is not limited in this respect and may take different forms, such as a rectangle or a circle, which nevertheless has a predetermined spatial relationship with the predicted position of the road marker image. In this example, the detection of the road marker in portion 1930 of second image 1920 is determined using the same method as used to determine the position of road marker image 1912 in first image 1910. Specifically, the boundary of road marker image 1922 in portion 1930 of image 1920 is first determined, and the boundary at (x m2 ,y m2 ) is determined as location L2 of road marker image 1922 in image 1920. However, as previously explained with respect to the detection of road marker image 1912 in first image 1910, any suitable method may be used to detect road marker image 1922 in second image 1920.

[0179] exist Figure 19A and Figure 19B In the example, Figure 16 In step S540, the distance estimation module 34 estimates the distance traveled by the vehicle along the road during the predetermined time period based on the determined positions of the road marker images 1912 in the first image 1910 and the determined positions of the road marker images 1922 in the second image 1920. More specifically, in this example, the distance estimation module 34 may calculate the value d m -d m2 Calculated as the estimated distance, where d m2 The Y-axis value y m2 represents the distance of a portion of the road from the vehicle, and d m The Y-axis value y m Indicates the distance of a portion of the road from the vehicle.

[0180] exist Figure 16 In step S550, the speed calculation module 35 calculates the vehicle's speed based on the estimated distance and the predetermined time period from step S540, specifically by dividing the estimated distance by the predetermined time period. In this example, the first image 1910 and the second image 1920 are consecutive images captured by a camera that captures images at a predetermined frame rate, so the predetermined time period is calculated as the inverse of the frame rate. However, the first image and the second image do not need to be consecutive images captured by the camera, and can be any pair of images captured by an onboard camera at two relatively close moments, such that common objects (such as road signs) appear in both images.

[0181] exist Figure 16In step S520, in this example, the estimate of the vehicle speed used to predict the position of the road marking image in the second image 1920 can be the vehicle speed calculated during the above-mentioned processing of a pair of images previously taken by the camera. Figure 16 The estimate of vehicle speed computed for a pair of images at any given iteration of the method can be used as Figure 16 However, in other embodiments, the vehicle speed estimate used in step S520 may be obtained in other ways, such as by receiving a measured speed from a speedometer of the vehicle.

[0182] It should be noted that although this example employs a mapping (between a first variable and a second variable) to predict the position of the road marking image in the second image 1920, where the first variable represents the position of a portion of the image in the image coordinate system and the second variable represents the distance of the portion of the road represented by the portion of the image from the vehicle, in some embodiments, the mapping is not used in step S520, and an alternative method may be used to predict the position of the road marking image in the second image 1920. That is, in some embodiments, the mapping (between a first variable representing the position of the portion of the image in the image coordinate system and a second variable representing the distance of the portion of the road represented by the portion of the image from the vehicle) may be used only in step S540 to estimate the distance traveled by the vehicle along the road during the predetermined time period.

[0183] The mapping described with respect to steps S520 and S540 can be implemented in a variety of ways. In this embodiment, the mapping is provided by a polynomial function that associates a first variable (representing the position of a portion of the image in the image coordinate system) with a second variable (representing the distance from the vehicle to the portion of the road represented by the portion of the image). This mapping method is independent of the vertical resolution of the image, as any value of the first variable can be transformed into a corresponding value of the second variable. In some embodiments, the mapping can alternatively be implemented as a lookup table that associates the first variable with the second variable. Using a lookup table allows for rapid execution of the mapping and has a lower processor burden, as processor-intensive calculations are not required. In some embodiments, the mapping can be defined by polylines that associate the first variable with the second variable, with each polyline corresponding to a linear function. More specifically, the mapping can be implemented using a lookup table that maps multiple non-overlapping ranges of values of the first variable to corresponding linear functions (corresponding to the polylines). The linear functions of the non-overlapping ranges can be used to map values in the non-overlapping ranges to corresponding values of the second variable.

[0184] It should be noted that although the above reference Figure 19A and Figure 19BThe described image processing operations are applied to a pair of images of the road in front of the vehicle captured by a forward-facing camera mounted on the vehicle as the vehicle travels forward along the road, but in an alternative embodiment, steps S510 to S550 and the previously described mapping can be similarly used to calculate the speed of the vehicle by processing a first rear-view image and a second rear-view image, which are captured at two different moments by a rear-facing camera mounted on the vehicle as the vehicle travels in the forward direction. Specifically, in an alternative embodiment, the steps performed are almost the same as steps S520 and S540. For example, assuming that the position of the road marker image in the first rear-view image is (x′ m ,y′ m ), in step S520, use Figure 18 In the case of predicting the position of the road sign in the second rearview image by using the mapping in the alternative embodiment, the position prediction module 32 will use the distance d′ m +Δd′ is mapped to the Y-axis position in the image coordinate system (of the rearview image) to predict the position of the road sign image in the second rearview image, where the distance d′ m is the distance from the vehicle to the road marker represented by the road marker image in the first rear view image, and Δd′ is the distance traveled by the vehicle during a predetermined time period between the capturing of the first rear view image and the capturing of the second rear view image. In addition, in step S540, the position of the determined road marker image in the second rear view image is expressed as (x′ m2 ,y′ m2 ), the distance estimation module 34 of the alternative embodiment calculates the value d′ m2 -d′ m As the estimated distance (the distance traveled by the vehicle during the predetermined time period), where d′ m2 The Y-axis value y′ m2 represents the distance of a portion of the road from the vehicle, and d′ m The Y-axis value y′ m Indicates the distance between the portion of the road and the vehicle.

[0185] Additionally, in some embodiments, the onboard camera may be a side-view camera arranged to capture images of the road to the side of the vehicle as the vehicle travels along the road as the pair of images. The side-view camera is preferably a high-speed camera with a sufficiently high frame rate (e.g., greater than 30fps) to ensure that the same road marker can be captured in both images. In this case, steps S510 to S550 may be similarly used to calculate the speed of the vehicle based on the first side-view image and the second side-view image captured at two different moments in time. Where side-view images are used, the mapping that may be used in steps S520 and / or S540 would be a linear mapping that maps displacements along the X-axis of the image coordinate system (of the first and second side-view images) to a value representing the distance traveled by the vehicle along the road. For example, assuming the position of the road marker image in the first side-view image is (x″ m ,y″ m ), in the case where the mapping is used to predict the position of the road sign in the second side view image at step S520, the position prediction module 32 will alternatively calculate the value x″ m -Δx″ to predict the X-axis position of the road marker image in the second side view image. Here, Δx″ is an estimated displacement along the X-axis of the image coordinate system, which can be determined using the mapping and the distance traveled by the vehicle during the predetermined time period between the capture of the first side view image and the capture of the second side view image. In addition, in step S540, when the estimated distance (the distance traveled by the vehicle during the predetermined time period) is calculated using the mapping, the determined position of the road marker image in the second side view is expressed as (x″ m2 ,y″ m2 ), the distance estimation module 34 will instead determine x″ m -x″ m2 The value of x″ is determined by mapping m -x″ m2 The estimated distance is calculated by mapping it to a distance representing the distance the vehicle traveled along the road.

[0186] The above implementation is summarized in the following numbered clauses E1 to E23:

[0187] E1. A method for determining a speed of a vehicle traveling along a road by processing an image pair (1910, 1920) of the road, the image pair (1910, 1920) of the road being captured by a camera mounted on the vehicle, each of the image pairs (1910, 1920) having a common image coordinate system and comprising a first image (1910) of the road and a second image (1920) of the road, the first image being captured at a first moment and the second image being captured at a second moment, the second moment being a predetermined period of time later than the first moment, wherein the first image (1910) and the second image (1920) comprise respective road marker images (1912, 1922) of road markers on the road, the method comprising processing each of the image pairs (1910, 1920) by the following steps:

[0188] Determining (S510) a position (L1) of a road marker image (1912) in the first image (1910) in an image coordinate system;

[0189] The position (L1) of the road marking image in the second image (1920) in the image coordinate system is predicted (S520) based on the determined position (L1) of the road marking image (1912), the estimate of the speed of the vehicle and the predetermined time period. p );

[0190] At the predicted position (L p ) in the portion (1930) where the road sign image (1922) is located (S530);

[0191] estimating (S540) a distance traveled by the vehicle along the road during the predetermined time period based on the determined positions (L1) of the road markers in the first image (1910) and the positions (L2) of the road marker images (1922) detected in the portion (1930) of the second image (1920); and

[0192] The speed of the vehicle is calculated (S550) based on the estimated distance and the predetermined time period.

[0193] E2. The method according to E1, wherein a portion of the first image (1910) including the determined position (L1) and the predicted position (L2) of the second image (1920) is included in the image. p) is located, and the result of the calculation is used to detect and determine the position (L2) of the road marker image (1922) in the part of the second image (1920), and estimate the distance traveled by the vehicle along the road during the predetermined time period.

[0194] E3. A method according to claim E1 or E2, wherein the estimate of the vehicle's speed used to predict the position of the road marking image in the second image during processing of the image pair is one of the speed of the vehicle measured by the vehicle's speedometer and the speed of the vehicle calculated during processing of one of the image pairs previously taken by the camera.

[0195] E4. The method according to any one of E1 to E3, wherein the predicted position (L) of the road marking image in the second image (1920) p ) is predicted using a mapping (1800) between a first variable and a second variable, the first variable representing a position of a portion of the image in the image coordinate system, and the second variable representing a distance from the vehicle to a portion of the road represented by the portion of the image.

[0196] E5. A method according to any one of E1 to E3, wherein estimating the distance traveled by the vehicle along the road during the predetermined time period is also based on a mapping (1800) between a first variable and a second variable, the first variable representing the position of a portion of the image in the image coordinate system, and the second variable representing the distance between the portion of the road represented by the portion of the image and the vehicle.

[0197] E6. The method according to E5, wherein the step of estimating the distance based on the mapping (1800) comprises:

[0198] determining a first value representing a distance from the vehicle to a portion of the road represented by the determined location (L1) of the road marking image (1912) in the first image (1910) and the mapping (1800);

[0199] determining a second value representing a distance from the vehicle to the portion of the road represented by the determined position (L2) of the road marking image (1922) in the second image (1920) using the position (L2) determined by the road marking image (1922) in the second image (1920) and the mapping (1800); and

[0200] The first value and the second value are used to estimate a distance traveled by the vehicle along the road during the predetermined time period.

[0201] E7. The method according to E5 or E6, wherein the predicted position (L) of the road sign image in the second image (1920) p ) is predicted using the mapping (1800).

[0202] E8. The method according to any one of E4 to E7, wherein the camera is arranged to capture a road image on one side of the vehicle as the image when the vehicle travels along the road, and the mapping is a linear mapping.

[0203] E9. The method according to any one of E4 to E7, wherein the camera is arranged to capture an image of the road behind or in front of the vehicle as the image when the vehicle travels along the road, and the mapping is a nonlinear mapping.

[0204] E10. A method according to E8 or E9, wherein the mapping is one of a polynomial associating the first variable with the second variable, a lookup table associating the first variable with the second variable, and a broken line associating the first variable with the second variable.

[0205] E11. The method according to any one of E1 to E10, wherein the position (L1) of the road marking image (1912) in the first image (1910) is determined by:

[0206] generating an M-th level LL subband image of an (M+1)-level discrete wavelet transform (DWT) of the first image (1910) by iteratively low-pass filtering and downsampling the first image (1910) M times, where M is an integer equal to or greater than 1;

[0207] generating an (M+1)th level subband image of the (M+1)th level DWT decomposition of the first image (1910) by performing high-pass filtering on the Mth level LL subband image and downsampling the result of the high-pass filtering;

[0208] generating boundary data representing a boundary of the road marking image (1912) of the first image (1910) by determining a boundary of a pixel region of the sub-band image of the (M+1)th level, the pixel region being surrounded by pixels having pixel values substantially different from pixel values of pixels in the region; and

[0209] By enlarging the boundary data of the pixel area of the (M+1)th level sub-band image by 2 M+1times to determine the position (L1) of the road marker image (1912) in the first image (1910).

[0210] E12. The method according to claim 11, wherein a first low-pass filter (510) having a symmetrical first sequence of filter coefficients (510) is used in at least one iteration of the iterative process.

[0211] E13. A method according to E12, wherein the filter coefficients of the first sequence of filter coefficients (510) are set to values in a row of Pascal's triangle having the same number of values as the order of the first low-pass filter (310).

[0212] E14. A method according to E12 or E13, wherein the high-pass filtering for generating the subband image of the (M+1)th level comprises applying a high-pass filter (320) having a symmetric second sequence of filter coefficients (520).

[0213] E15. The method according to E14, wherein:

[0214] The filter coefficients of the second sequence of filter coefficients (520) are set to the values of the corresponding positions in the rows of the Pascal triangle having the same number of values as the order of the high-pass filter (320), and

[0215] Each of the remaining filter coefficients in the second sequence of filter coefficients (520) is set to a value obtained by multiplying the value of the corresponding position in the row of Pascal's triangle by -1.

[0216] E16. A method according to any one of E11 to E15, wherein the (M+1)th level subband image of the (M+1)-level DWT decomposition is one of an LH subband image, an HL subband image, and an HH subband image.

[0217] E17. The method according to any one of E1 to E16, wherein generating the (M+1)th level subband image of the (M+1)th level DWT decomposition of the first image (1910) comprises generating the (M+1)th level LH subband image of the (M+1)th level DWT decomposition of the first image (1910) by the following operation:

[0218] generating a low-pass filtered LL subband image by applying a row kernel on rows of the LL subband image at the Mth level, the row kernel corresponding to a low-pass filter (310);

[0219] Downsampling the columns of the low-pass filtered LL sub-band image by a factor of 2 to generate a downsampled sub-band image;

[0220] generating a high pass filtered LL subband image by applying a column kernel on columns of the downsampled subband image, the column kernel corresponding to a high pass filter (320); and

[0221] The rows of the high-pass filtered LL sub-band image are down-sampled by a factor of 2 to generate an (M+1)th level LH sub-band image.

[0222] E18. The method according to any one of E11 to E16, wherein generating the (M+1)th level subband image of the (M+1)th level DWT decomposition of the image comprises generating the (M+1)th level LH subband image of the DWT decomposition of the image by the following operation:

[0223] generating a high-pass filtered LL subband image by applying a column kernel on columns of the LL subband image of the Mth level, the column kernel corresponding to a high-pass filter (320);

[0224] Downsampling the rows of the high-pass filtered LL subband image by a factor of 2 to generate a downsampled subband image;

[0225] generating a low-pass filtered sub-band image by applying a row kernel on the rows of the downsampled sub-band image of the Mth level, the row kernel corresponding to the low-pass filter; and

[0226] The columns of the low-pass filtered sub-band image are down-sampled by a factor of 2 to generate an LH sub-band image of the (M+1)th level.

[0227] E19. The method according to any one of E11 to E16, wherein generating the (M+1)th level subband image (930) comprises generating the (M+1)th level LH subband image of the (M+1)th level DWT decomposition of the image (910) by the following operations:

[0228] generating a filtered subband image by applying a two-dimensional kernel to the LL subband image of the Mth level, the two-dimensional kernel being divisible into a product of a row kernel and a column kernel, the row kernel defining a low-pass filter and the column kernel defining a high-pass filter; and

[0229] The rows and columns of the filtered subband image are downsampled by a factor of 2.

[0230] E20. A method according to any one of E11 to E19, wherein the road sign is a baud point, and the (M+1)th level subband image of the (M+1)th level of the (M+1)-level DWT decomposition is the (M+1)th level LH subband image of the (M+1)-level DWT decomposition.

[0231] E21. The method according to any one of E11 to E20, wherein generating the boundary data comprises:

[0232] determining pixel positions of pixels having pixel values exceeding a predetermined threshold in the (M+1)th level sub-band image; and

[0233] A contour tracing algorithm is performed using the determined pixels to identify a boundary of a pixel region (1420), wherein the boundary separates pixels of the region (1420) adjacent to the boundary and having pixel values above a predetermined threshold from pixels outside the region (1420) adjacent to the boundary and having pixel values below a predetermined threshold.

[0234] E22. A computer program (245) comprising computer-readable instructions which, when executed by a processor (220), cause the processor (220) to perform the method according to any one of E1 to E21.

[0235] E23. An apparatus (30) for determining a speed of a vehicle traveling along a road by processing a pair of images (1910, 1920) of the road captured by a camera mounted on the vehicle, each image in the pair of images (1910, 1920) having a common image coordinate system and comprising a first image (1910) of the road and a second image (1920) of the road, the first image (1910) being captured at a first moment and the second image (1920) being captured at a second moment, the second moment being a predetermined period of time later than the first moment, wherein the first image (1910) and the second image (1920) include respective road marker images (1912, 1922) of road markers on the road, the apparatus comprising:

[0236] a position determination module (31) arranged to determine a position (L1) of a road marking image (1912) in the first image (1910) in an image coordinate system;

[0237] a position prediction module (32) arranged to predict a position (L1) of a road marking image (1922) in the second image (1920) in the image coordinate system based on the determined position (L1) of the road marking image (1912), an estimate of the speed of the vehicle and the predetermined time period; p );

[0238] A road sign image detection module (33) is arranged to detect a road sign at a predicted position (L p ) is located in the portion (1930) where the road sign image (1922) is detected;

[0239] a distance estimation module (34) arranged to estimate a distance travelled by the vehicle along the road during the predetermined time period based on the determined positions (L1) of road marking images in the first image (1910) and the positions (L2) of detected road marking images (1922) in the portion (1930) of the second image (1920); and

[0240] A speed calculation module (35) is arranged to calculate the speed of the vehicle based on the estimated distance and the predetermined time period.

[0241] In the foregoing description, various aspects have been described with reference to several embodiments. Therefore, the description should be considered illustrative rather than restrictive. Similarly, the diagrams shown in the accompanying drawings, which highlight features and advantages of the embodiments, are presented for illustrative purposes only. The architecture of the embodiments is sufficiently flexible and configurable that it can be utilized in a manner other than that shown in the accompanying drawings.

[0242] In one exemplary embodiment, the software implementations presented herein may be provided as a computer program or software, such as one or more programs having instructions or instruction sequences, which are included or stored on an article of manufacture such as a machine-accessible or machine-readable medium, an instruction store, or a computer-readable storage device, each of which may be non-transitory. The program or instructions on the non-transitory machine-accessible medium, machine-readable medium, instruction store, or computer-readable storage device can be used to program a computer system or other electronic device. The machine- or computer-readable medium, instruction store, and storage device may include, but is not limited to, floppy disks, optical disks, and magneto-optical disks, or other types of media / machine-readable media / instruction stores / storage devices suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They may find applicability in any computing or processing environment. As used herein, the terms "computer-readable," "machine-accessible medium," "machine-readable medium," "instruction store," and "computer-readable storage device" shall include any medium capable of storing, encoding, or transmitting instructions or instruction sequences for execution by a machine, computer, or computer processor, causing the machine / computer / computer processor to perform any of the methods described herein. Furthermore, it is common in the art to refer to software, in one form or another (e.g., program, procedure, process, application, module, unit, logic, etc.) as taking an action or causing a result. Such expressions are merely a shorthand way of stating that execution of the software by a processing system causes the processor to perform an action to generate a result.

[0243] Some embodiments may also be implemented by the preparation of application specific integrated circuits, field programmable gate arrays, or by interconnecting an appropriate network of conventional component circuits.

[0244] Some embodiments include a computer program product. A computer program product can be a storage medium or multiple storage media, one or more instruction stores, or one or more storage devices having instructions stored thereon or therein, which instructions can be used to control or cause a computer or computer processor to perform any of the programs of the exemplary embodiments described herein. The storage medium / instruction store / storage device can include, for example, but is not limited to, an optical disk, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory, flash memory cards, magnetic cards, optical cards, nanosystems, molecular memory integrated circuits, RAID, remote data storage / archiving / warehousing, and / or any other type of device suitable for storing instructions and / or data.

[0245] Some implementations stored on any of a computer-readable medium, one or more instruction stores, or one or more storage devices include hardware for controlling the system and software for enabling the system or microprocessor to interact with a human user or other mechanism using the results of the embodiments described herein. Such software may include, but is not limited to, device drivers, operating systems, and user applications. Finally, such computer-readable media or storage devices also include software for performing the example aspects described above.

[0246] Included in the programming and / or software of the system are software modules for implementing the processes described herein. In some example embodiments herein, the modules include software, while in other example embodiments herein, the modules include hardware or a combination of hardware and software.

[0247] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example and not limitation. It will be apparent to those skilled in the relevant art that various changes in form and detail may be made therein. Therefore, the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the appended claims and their equivalents.

[0248] Furthermore, the purpose of the abstract is to enable patent offices and the public, especially scientists, engineers, and practitioners in the field who are not familiar with patent or legal terminology or wording, to quickly ascertain the nature and essence of the technical disclosure of the application from a cursory inspection. The abstract is not intended to limit the scope of the embodiments presented herein in any way. It should also be understood that any process recited in the claims need not be performed in the order presented.

[0249] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on any invention or the scope that may be claimed, but rather as descriptions of specific features of the specific implementations described herein. Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable sub-combination. Furthermore, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variant of the sub-combination.

[0250] In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0251] Now that some illustrative embodiments have been described, it will be apparent that the foregoing is illustrative and not restrictive, and has been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of devices or software elements, these elements can be combined in other ways to achieve the same objectives. Actions, elements, and features discussed in connection with only one embodiment are not intended to be excluded from similar roles in other embodiments or multiple embodiments.

[0252] The apparatus described herein may be embodied in other specific forms without departing from its characteristics. The above embodiments are illustrative rather than restrictive of the systems and methods described. The scope of the apparatus described herein is therefore indicated by the appended claims rather than the foregoing description, and changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein.

Claims

1. A method for determining a speed of a vehicle traveling along a road by processing an image pair (1910, 1920) of the road, the image pair (1910, 1920) being captured by a camera mounted on the vehicle, each of the image pairs (1910, 1920) having a common image coordinate system and comprising a first image (1910) of the road and a second image (1920) of the road, the first image being captured at a first moment in time and the second image being captured at a second moment in time, the second moment in time being a predetermined period of time later than the first moment in time, wherein The first image (1910) and the second image (1920) comprise respective road marking images of road markings on the road, the method comprising processing each of the image pairs (1910, 1920) by: Determining (S510) the position of the road marker image in the first image (1910) in the image coordinate system; predicting (S520) a position of a road marking image in the second image (1920) in the image coordinate system based on the determined position of the road marking image, the estimate of the speed of the vehicle and the predetermined time period; detecting ( S530 ) a road marking image in a portion ( 1930 ) of the second image ( 1920 ) where the predicted position is located; estimating (S540) a distance traveled by the vehicle along the road during the predetermined time period based on the determined positions of the road marker images in the first image (1910) and the positions of the detected road marker images in the portion (1930) of the second image (1920); as well as calculating (S550) the speed of the vehicle based on the estimated distance and the predetermined time period, The position of the road marker image in the first image (1910) is determined by the following operations: generating an M-th level LL subband image of a (M+1)-level discrete wavelet transform (DWT) decomposition of the first image (1910) by iteratively low-pass filtering and downsampling the first image (1910) M times, where M is an integer equal to or greater than 1; generating an (M+1)th level subband image of the (M+1)th level DWT decomposition of the first image (1910) by performing high-pass filtering on the Mth level LL subband image and downsampling the result of the high-pass filtering; generating boundary data representing a boundary of a road marking image in the first image (1910) by determining a boundary of a pixel region of the subband image of the (M+1)th level, the pixel region being surrounded by pixels having pixel values substantially different from pixel values of pixels in the pixel region; and By enlarging the boundary data of the pixel area of the (M+1)th level sub-band image by 2 M+1 times to determine the position of the road marker image in the first image (1910).

2. The method according to claim 1, wherein Perform a calculation of a cross-correlation between a portion of the first image (1910) containing the determined position and a portion of the second image (1920) containing the predicted position, and use the result of the calculation to detect and determine the position of a road marker image in the portion of the second image (1920), and estimate the distance traveled by the vehicle along the road during the predetermined time period.

3. The method according to claim 1 or 2, wherein: The estimate of the vehicle's speed used to predict the position of a road marking image in the second image during processing of one of the image pairs is one of: the speed of the vehicle as measured by the vehicle's speedometer; and The speed of the vehicle calculated during processing of one of the pairs of images previously captured by the camera.

4. The method according to claim 1, wherein The predicted position of the road marker image in the second image (1920) is predicted using a mapping (1800) between a first variable and a second variable, the first variable representing the position of a portion of the image in the image coordinate system, and the second variable representing the distance of the portion of the road represented by the portion of the image from the vehicle.

5. The method according to claim 1, wherein Estimating the distance traveled by the vehicle along the road during the predetermined time period is also based on a mapping (1800) between a first variable and a second variable, the first variable representing a position of a portion of the image in the image coordinate system, and the second variable representing a distance from the vehicle to a portion of the road represented by the portion of the image.

6. The method according to claim 5, wherein: The step of estimating the distance based on the mapping (1800) comprises: determining a first value representing a distance from the vehicle to the portion of the road represented by the determined position of the road marker image in the first image (1910) using the determined position of the road marker image in the first image (1910) and the mapping (1800); determining a second value representing a distance from the vehicle to the portion of the road represented by the determined position of the road marker image in the second image (1920) using the determined position of the road marker image in the second image (1920) and the mapping (1800); and The first value and the second value are used to estimate a distance traveled by the vehicle along the road during the predetermined time period.

7. The method according to claim 5 or 6, wherein: The predicted positions of the road marking images in the second image (1920) are predicted using the mapping (1800).

8. The method according to claim 4 or 5, wherein: The camera is arranged to capture the following images as the images: an image of a road to the side of the vehicle as the vehicle travels along the road, and the mapping is a linear mapping; or An image of a road behind or in front of the vehicle as the vehicle travels along the road, and the mapping is a non-linear mapping.

9. The method according to claim 8, wherein The mapping is one of a polynomial associating the first variable with the second variable, a lookup table associating the first variable with the second variable, and a polyline associating the first variable with the second variable.

10. The method according to claim 1, wherein A first low-pass filter having a symmetrical first sequence of filter coefficients is used in at least one iteration of the iterative process.

11. The method according to claim 10, wherein: The filter coefficients of the first sequence of filter coefficients are set to values in a row of Pascal's triangle having the same number of values as the order of the first low-pass filter.

12. The method according to claim 1, wherein The step of generating the (M+1)th level subband image of the (M+1)th level DWT decomposition of the first image (1910) comprises generating the (M+1)th level LH subband image of the (M+1)th level DWT decomposition of the first image (1910) by one of the following operations: A first process, wherein the first process comprises: generating a low-pass filtered LL subband image by applying a row kernel on rows of the LL subband image at the Mth level, the row kernel corresponding to a low-pass filter; Downsampling the columns of the low-pass filtered LL sub-band image by a factor of 2 to generate a downsampled sub-band image; generating a high pass filtered LL subband image by applying a column kernel on columns of the downsampled subband image, the column kernel corresponding to a high pass filter (320); and Downsample the rows of the high-pass filtered LL subband image by a factor of 2 to generate the (M+1)th level LH subband image; The second process includes: generating a high-pass filtered LL subband image by applying a column kernel on columns of the LL subband image of the Mth level, the column kernel corresponding to a high-pass filter (320); Downsampling the rows of the high-pass filtered LL subband image by a factor of 2 to generate a downsampled subband image; generating a low-pass filtered sub-band image by applying a row kernel on the rows of the downsampled sub-band image of the Mth level, the row kernel corresponding to the low-pass filter; and Downsampling the columns of the low-pass filtered sub-band image by a factor of 2 to generate an LH sub-band image of the (M+1)th level; and The third process includes: generating a filtered subband image by applying a two-dimensional kernel to the LL subband image of the Mth level, the two-dimensional kernel being divisible into a product of a row kernel and a column kernel, the row kernel defining a low-pass filter and the column kernel defining a high-pass filter; and The rows and columns of the filtered subband image are downsampled by a factor of 2.

13. A computer-readable storage medium storing a computer program (245) comprising computer-readable instructions which, when executed by a processor (220), cause the processor (220) to perform the method according to any one of claims 1 to 12.

14. A device (30) for determining a speed of a vehicle traveling along a road by processing an image pair (1910, 1920) of the road, the image pair (1910, 1920) being captured by a camera mounted on the vehicle, each of the image pairs (1910, 1920) having a common image coordinate system and comprising a first image (1910) of the road and a second image (1920) of the road, the first image (1910) being captured at a first moment in time and the second image (1920) being captured at a second moment in time that is a predetermined period of time later than the first moment in time, wherein The first image (1910) and the second image (1920) include respective road marking images of road markings on the road, the apparatus comprising: a position determination module (31) arranged to determine the position of the road marking image in the first image (1910) in the image coordinate system; a position prediction module (32) arranged to predict a position of a road marking image in the second image (1920) in the image coordinate system based on the determined position of the road marking image, the estimate of the speed of the vehicle and the predetermined time period; a road marking image detection module (33) arranged to detect a road marking image in the portion (1930) of the second image (1920) where the predicted position is located; a distance estimation module (34) arranged to estimate a distance travelled by the vehicle along the road during the predetermined time period based on the determined positions of road marking images in the first image (1910) and the positions of detected road marking images in the portion (1930) of the second image (1920); and a speed calculation module (35) arranged to calculate the speed of the vehicle based on the estimated distance and the predetermined time period, Wherein, when determining the position of the road marker image in the first image (1910), the position determination module is arranged to: generating an M-th level LL subband image of a (M+1)-level discrete wavelet transform (DWT) decomposition of the first image (1910) by iteratively low-pass filtering and downsampling the first image (1910) M times, where M is an integer equal to or greater than 1; generating an (M+1)th level subband image of the (M+1)th level DWT decomposition of the first image (1910) by performing high-pass filtering on the Mth level LL subband image and downsampling the result of the high-pass filtering; generating boundary data representing a boundary of a road marking image in the first image (1910) by determining a boundary of a pixel region of the subband image of the (M+1)th level, the pixel region being surrounded by pixels having pixel values substantially different from pixel values of pixels in the pixel region; and By enlarging the boundary data of the pixel area of the (M+1)th level sub-band image by 2 M+1 times to determine the position of the road marker image in the first image (1910).

Images