System and method for determining surface markings from images

Abstract

Described embodiments generally relate to a method for estimating a surface normal from a plurality of images of a surface. The method comprises obtaining a first image depicting at least a portion of the surface and obtaining a second image depicting at least the portion of the surface. The method further comprises determining a homography matrix based on an initial estimated surface normal of the portion of the surface; selecting a first pixel from the first image and determining, by homography transformation, a second pixel from the second image; determining, by applying a trained neural network to at least the first image and the second image; determining a residual between the features of the first pixel and the features of the second pixel; and adjusting the estimated surface normal based on the residual.

Description

System and method for determining surface markings from images Cross-Reference to Related Application

[0001] The present application claims priority from Australian Provisional Patent Application No. 2023902579 filed on 14 August 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] Aspects of the disclosure relate generally to systems and methods for image analysis. More specifically, aspects of the disclosure relate to the estimation of a surface normal vector for a surface depicted in images, and the determination of markings on the surface. Background

[0003] As autonomous driving technology advances, one of the fundamental tasks for an autonomous vehicle is to perceive and understand its surrounding environment, including the road structure and road markings. Road line segmentation can play an important role in this perception process, as it provides critical information for lane keeping, vehicle trajectory planning, and object detection. Accurate and reliable segmentation of road lines is desirable to ensure the safe and effective operation of autonomous vehicles.

[0004] Existing road line segmentation methods primarily rely on computer vision algorithms and traditional image processing techniques. However, these methods often struggle with complex road conditions, such as varying lighting conditions, occlusions, road marking degradation, and challenging environmental factors. Consequently, road line segmentation methods may produce unreliable or inaccurate results, compromising the overall performance and safety of autonomous driving systems.

[0005] There is a need for an improved road line segmentation approach that can effectively handle diverse road conditions, offer robustness against noise and uncertainties, and provide real-time performance for autonomous driving applications.

[0006] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is solely for the purpose of providing a context for the present invention. It is not to be taken as an admission that any or all of these matters formpart of the prior art base or were common general knowledge in the field relevant to the present invention as it existed before the priority date of each claim of this application. Summary

[0007] In accordance with an aspect of the present disclosure, there is provided a method for estimating a surface normal from a plurality of images of a surface. The method comprises: obtaining, from a camera positioned a first viewpoint of the surface, a first image depicting at least a portion of a surface; obtaining, from the camera positioned at a second viewpoint of the surface, a second image depicting at least the portion of the surface; determining an initial homography matrix based on an initial estimated surface normal of the portion of the surface; and selecting a first pixel from the first image, wherein the pixel represents a location on the portion of the surface. The method further comprises: determining, by applying a homography transformation to the first image using the homography matrix, a second pixel from the second image, the second pixel spatially corresponding to the first pixel; determining, by applying a trained neural network to at least the first image, a feature map of the first image, the feature map of the first image comprising features of the first pixel; and determining, by applying the trained neural network to at least the second image, a feature map of the second image, the feature map of the second image comprising features of the second pixel. The method further comprises: determining a residual between the features of the first pixel and the features of the second pixel; and adjusting the estimated surface normal based on the residual.

[0008] In some embodiments, the camera is attached to a vehicle in motion on the surface. In some embodiments, the first viewpoint comprises a position of the vehicle on the surface at a first point in time, and the second viewpoint comprises a position of the vehicle on the surface at a second point in time.

[0009] In some embodiments, the first image and the second image comprise sequential frames of a video captured by the camera. In some embodiments, the first pixel is selected from a focal area of the first image. In some embodiments, the focal area depicts a road surface in front of the vehicle in the direction of travel.

[0010] In some embodiments, adjusting the estimated surface normal comprises: determining a surface normal delta based on the residual; and adjusting the estimated surface normal by applying the surface normal delta to the estimated surface normal.

[0011] In some embodiments, determining a surface normal delta comprises: determining a non-linear least squares cost based on the residual; and determining the surface normal delta by applying a damped least-squares method to the non-linear least squares cost. In some embodiments, the method further comprises, in response to comparing the surface normal delta with a threshold, adjusting the estimated surface normal by applying the surface normal delta to the estimated surface normal.

[0012] In some embodiments, adjusting the estimated surface normal based on the residual comprises, performing at least one iteration of: determining an adjusted surface normal by applying the surface normal delta to the estimated surface normal; adjusting the homography matrix based on the adjusted estimated surface normal; determining, by applying the adjusted homography matrix to the first image, a third pixel from the second image, the third pixel spatially corresponding to the first pixel; determining, by applying a trained neural network to at least the first image, a feature map of the first image, the feature map of the first image comprising features of the first pixel; determining, by applying the trained neural network to at least the second image, a feature map of the second image, the feature map of the second image comprising features of the third pixel; determining a second residual between the features of the first pixel and the features of the third pixel; and determining a second surface normal delta based on the second residual. In some embodiments, the method further comprises, in response to performing a predefined number of iterations of the above-described method, adjusting the estimated surface normal by applying the second surface normal delta to the estimated surface normal.

[0013] In some embodiments, determining the residual comprises determining the residual between the low-level features of the feature map of the first pixel and the low-level features of the feature map of the second pixel. In some embodiments, the method is performed responsive to obtaining a new frame from the camera. In some embodiments, the surface normal comprises a unit vector with two degrees of freedom, the degrees of freedom comprising a pitch angle and a roll angle.

[0014] In some embodiments, the initial homography matrix is determined based on an intrinsic homography matrix associated with the camera. In some embodiments, the method further comprises determining an adjusted homography matrix based on the adjusted estimated surface normal. In some embodiments, the method further comprises determining a fused feature map by applying the adjusted homography matrix to a feature map of the first image and by applying the adjusted homography matrix to a feature map of the second image.

[0015] In some embodiments, the method further comprises computing a temporally consistent representation for a plurality of pixels of the first image by employing a temporal attention mechanism on the fused feature map. In some embodiments, the temporal attention mechanism comprises a local attention mechanism. In some embodiments, the temporal attention mechanism assigns pixel representation of the first image as a query and pixel representations of the second image as keys and values. In some embodiments, the temporal attention mechanism obtains a L2 normalization on the query and the keys. In some embodiments, the method further comprises decoding the temporally consistent representation to produce image analysis information.

[0016] In some embodiments, the image analysis information comprises, road marking analysis information. In some embodiments, the road marking analysis comprises one or more of: lane segmentation information; lane detection information; road markings information; and puddle segmentation information.

[0017] According to another aspect of the present disclosure, there is provided a computer- implemented method for estimating a surface normal from a plurality of images of a surface. The computer-implemented method comprises a processor performing the steps of: obtaining, from a camera positioned a first viewpoint of the surface, a first image depicting at least a portion of a surface; obtaining, from the camera positioned at a second viewpoint of the surface, a second image depicting at least the portion of the surface; determining an initial homography matrix based on an initial estimated surface normal of the portion of the surface; selecting a first pixel from the first image, wherein the pixel represents a location on the portion of the surface; determining, by applying a homography transformation to the first image using the homography matrix, a second pixel from the second image, the second pixel spatially corresponding to the first pixel; determining, by applying a trained neural network to at least the first image, a feature map of the first image, the feature map of the first image comprising features of the first pixel; determining, by applying the trained neural network to at least the second image, a feature map of the second image, the feature map of the second image comprising features of the second pixel; determining a residual between the features of the first pixel and the features of the second pixel; and adjusting the estimated surface normal based on the residual.

[0018] According to another aspect of the present disclosure, there is provided a method for analysing surface markings from a plurality of images of a surface. The method comprises: obtaining, from a camera positioned a first viewpoint of the surface, a first image depicting atleast a portion of the surface; obtaining, from the camera positioned at a second viewpoint of the surface, a second image depicting at least the portion of the surface; determining, by applying a trained neural network to at least the first image, a feature map of the first image, the feature map of the first image; determining, by applying the trained neural network to at least the second image, a feature map of the second image, the feature map of the second image; determining, by applying an attention mechanism to the feature map of the first image and the feature map of the second image, a fused feature map; determining, by decoding the fused feature map, the surface marking information.

[0019] In some embodiments, the camera is attached to a vehicle in motion on the surface. In some embodiments, the first viewpoint comprises a position of the vehicle on the surface at a first point in time, and the second viewpoint comprises a position of the vehicle on the surface at a second point in time. In some embodiments, the first image and the second image comprise sequential frames of a video captured by the camera.

[0020] In some embodiments, the attention mechanism comprises a local attention mechanism. In some embodiments, the attention mechanism comprises a cross-frame pixel-to- pixel attention mechanism. In some embodiments, applying the cross-frame pixel-to-pixel attention mechanism comprises: projecting one or more pixels of the first image onto the second image by applying a homography transformation. In some embodiments, the one or more pixels of the first image are selected from a focal area of the first image. In some embodiments, the focal area depicts a road surface in front of the vehicle in the direction of travel.

[0021] According to another aspect of the present disclosure, there is provided a system for estimating a surface normal from a plurality of images of a surface. The system comprises a camera attached to a vehicle in motion on a surface, one or more processors, and memory comprising computer executable instructions. When the computer executable instructions are executed by the one or more processors, cause the system to perform a method described herein.

[0022] According to another aspect of the present disclosure, there is provided machine- readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method described herein.Brief Description of Drawings

[0023] The embodiments of the disclosure will now be described with reference to the accompanying drawings, in which: Figure 1 illustrates a system for analysing road features, in accordance with an embodiment; Figure 2 illustrates the architecture of an image processing module configured to analyse road markings, in accordance with an embodiment; Figure 3 illustrates a method for processing image data, in accordance with an embodiment; Figure 4 illustrates a process for estimating a road surface normal, in accordance with an embodiment; Figure 5 comprises a representation of an ordered sequence of three frames captured by a on-vehicle camera, pointed in the direction of travel of a vehicle in motion, in accordance with an embodiment; Figure 6 illustrates the frames of Figure 5, as annotated to illustrate a focal area on the road surface, and to illustrate spatial correspondence between pixels across frames, in accordance with an embodiment; Figure 7 illustrates the effect of an implementation of the HGF module, in accordance with an embodiment; Figure 8 comprise four graphs illustrating the performance impact of the number of frames, and the sampling frame gap, in accordance with an embodiment; Figure 9 comprises a table indicating the results of a comparison of the performance of the HomoFusion module to the performance of other algorithms on the ApolloScape and ApolloScape Night datasets, in accordance with an embodiment; Figure 10 comprises a table indicating the results of an ablation study, in accordance with an embodiment; Figure 11 illustrates a table indicating the results of an ablation study regarding backbones, in accordance with an embodiment;Figure 12 illustrates a table indicating the performance results for water puddle segmentation, in accordance with an embodiment; Figure 13 comprises an image-based replication of Equations (1) to (6), in accordance with an embodiment; Figure 14 comprises an image-based replication of Equations (7) to (12), in accordance with an embodiment; Figure 15 illustrates the rotation of the road surface plane around the x-axis in accordance with the pitch angle, in accordance with an embodiment; Figure 16 illustrates the rotation of the road surface plane around the z-axis in accordance with the roll angle, in accordance with an embodiment; Figure 17 illustrates a table comprising a comparison of estimated COLMAP extrinsics and OpenCV homography on the ApolloScape Dataset, in accordance with an embodiment. Description of Embodiments

[0024] Vehicle driving systems, which provide for autonomous driving or driver assistance, utilise cameras which capture images of the surface on which the vehicle is travelling (e.g. a road). Roads a often marked with road markings, such as lane boundaries, arrow, pedestrian crossings and other signals to the driver. Autonomous driving systems process the images captured by the cameras to extract meaning from the road markings.

[0025] Reliable and accurate segmentation of road lines and markings is often fundamental to achieving safe and effective autonomous driving. The view of road lines and markings are frequently occluded in the presence of other moving vehicles on the road, shadows, and glare. Furthermore, road lines and markings are often highly structured with low intra-class shape variance and overall high appearance consistency.

[0026] Embodiments of the lightweight lane mark segmentation module provided herein may offer a superior performance with reduced model complexity compared to other approaches. The integration of the homography guided fusion module and road surface normal estimator may provide accurate classification of partially occluded, shadowed, and / or glare-affected road lines and markings by leveraging adjacent frames and pixel-to-pixel attention mechanisms.

[0027] The approach of using a road surface normal to guide spatial correspondences across frames may have significant implications for a wide range of applications in autonomous driving and related applications, including road marking analysis, road surface inspection, water puddle, and other hazard detections. System architecture

[0028] Figure 1 illustrates a system for analysing road features, in accordance with an embodiment. In particular, Figure 1 depicts a vehicle 150 which is travelling along a surface 110. The vehicle 150 comprises a camera 180 configured to capture images of a spatial area external to the vehicle. The camera is pointing forwards, in the direction of travel of the vehicle.

[0029] In embodiments, vehicle may comprise a plurality of cameras. A camera may be integral with the vehicle, affixed or attached to the vehicle. The vehicle may comprise a plurality of cameras, wherein each of the cameras are configured to capture a view of a different spatial area external to the vehicle.

[0030] Surface 110 may comprise a surface on which a vehicle may travel. Surface 110 may comprise a road surface. As used herein, the term road surface refers generally to any surface which a vehicle is positioned on, is travelling on, or is anticipated to travel on as the vehicle continues its travel motion. As used herein, a road surface may comprise, but is not limited to a road, a street, a parking lot, an indoor surface on which a vehicle travels, a highway, a driveway, an unsealed surface, an unmarked surface, or some combination thereof. A road surface may comprise a surface that is not specifically or uniquely designated for use by travelling vehicles, such as, but not limited to, a verge, a field or an off-road surface.

[0031] The vehicle further comprises a controller 102 configured to control operation of the vehicle 150. In embodiments, the controller 102 is configured to provide autonomous driving control. In embodiments, the controller is configured to provide information or assistance to a driver of the vehicle.

[0032] In embodiments, the controller comprises an image processing module 106 which processes image data (e.g. video frames, and / or still images) captured by the camera 180. In embodiments, the image processing module 106 comprises one or more processors and machine-readable instructions that when executed on the one or more processors, process the image data received from the camera, in accordance with methods described herein. In embodiments, the image processing module comprises hardware circuitry configured to process the image data received from the camera in accordance with methods described herein.Perspective effect

[0033] Images captured by vehicle cameras often have a perspective effect. The perspective effect may depend upon the position of the camera, the camera lens, and other intrinsic parameters of the camera. The perspective effect can complicate image processing by distorting the shape of objects and markings depicted in the picture. Furthermore, the perspective effect can complicate the comparison of images captured from different cameras, or by the same camera located at different positions (e.g. due to the movement of the vehicle). Consequently, it is often desirable for an image processing system to perform transformation of the captured images to remove or normalise the perspective effect of the images. Homography

[0034] Homography transformation refers to a mathematical transformation that describes the relationship between two different perspectives of the same scene or object. Homography transformations may be used in computer vision and image processing applications, such as image registration, panorama stitching, augmented reality, and object tracking. Homography transformations comprise transforming the features (e.g. the pixels) of an image by applying a homography matrix.

[0035] A homography matrix defines the mapping between corresponding points in two different images taken from different viewpoints. The homography transformation allows us to project points from one image to another, or vice versa, while preserving their spatial relationships. In image stitching, a homography matrix may be used to align and merge multiple images taken from different viewpoints to create a seamless panorama. In augmented reality, a homography matrix may be applied to overlay virtual objects onto a real-world scene, aligning them correctly with the camera view.

[0036] A homography matrix may be determined based on the position of the camera relative to the surfaces within the field of view of the camera. For a vehicle mounted camera, a homography matrix can also be determined based on the position of the camera relative to the surface on which the vehicle is travelling.

[0037] A processor may apply a homography matrix to the pixels of an input image to determine the transformed coordinates of each pixel of the input image. In some embodiments, the transform does not interpolate between pixel locations. Instead, it rounds the result to the nearest coordinate. In some examples, image processing systems apply homographic transformation to generate a bird’s eye view of a spatial area from a perspective image of thatspatial area. In some examples, the image processing systems stitches together a plurality of images to generate the bird’s eye view of the spatial area. Analysing road markings

[0038] Road mark analysis aims to detect and classify road lines and road markings. Road mark analysis is often an important process in autonomous driving, and may be an effective tool for constructing accurate high definition (HD) maps of a road network. Road mark analysis may comprise: lane detection; lane segmentation; road markings detection; puddle detection; puddle segmentation; or any combination thereof.

[0039] Lane detection aims to identify the boundaries of the drivable area. Lane segementation aims to identify the shape of the lanes. Some lane detection and lane segmentation methods may further comprise a method to detect and intepret road markings.

[0040] Traditional lane detection methods apply color detection, edge detection and / or texture detection. Traditional land detection approaches may be limited in complex scenarios. Advances in deep neural networks have led to improvements in lane detection methods, with methods like message-passing networks, and global attention-based modules.

[0041] Lane detection may be formulated as a row-wise classification task based on grid division of the input image. Such detection methods usually predict lane positions, and construct the set of lanes via post-processing. Some lane detection methods comprise parametric prediction methods, which output polynomials to represent each lane.

[0042] The identification of road elements indicating lanes is important for ensuring safety in autonomous driving systems. A method for identifying road elements includes utilising a distillation approach that transfers knowledge from a large teacher network to a smaller student model, achieving competitive performance on lane mark segmentation. Another method for identifying road elements includes using a Long Short-term Memory (LSTM) based network to segment images with DeepLabv3+ and then merged the segmented images with point clouds to assist with lane segmentation. Notably, these methods do not address the challenges of occlusion, shadows, and glare that frequently occur in real-world driving scenarios.

[0043] In some situations, it may be possible to introduce global memory information from previous frames to enhance local information for video lane segmentation. However, this method introduces attention to the entire image, making it memory-intensive and inefficient for autonomous driving scenarios.Occlusion

[0044] A major challenge in the road mark analysis task is partial occlusion of the road lines and markings within the field of view of the vehicle’s camera(s). Occlusions may be caused by the vehicle itself, nearby vehicles, shadows, glare or aspects of the surrounding environment. These occlusions can lead to false classifications of lane mark segmentation and road marking classifications. For example, a partially occluded straight arrow can be easily mistaken as a right turn & straight arrow.

[0045] The adverse impact of the occlusion of the road lines and markings may be reduced by analysing information across a plurality of frames in a sequence. In particular, the impact of occlusion of the road lines and markings may be reduced by analysing cross-frame consistency. Additionally, road lines and markings are highly structured, maintaining high consistency in intra-class shapes as well as in overall appearance. This high consistency may alleviate the requirements for learning complicated features to distinguish different classes with high intra- class variance as in semantic / scene segmentation. Camera intrinsic and extrinsic parameters

[0046] On-vehicle cameras may have a fixed (or determinable) angle and position with respect to the vehicle, and this angle and position may be defined by the camera’s extrinsic parameters. When the vehicle is positioned on a road surface, the camera’s angle and position with respect to a road surface directly below the vehicle may be determined based on the camera’s angle and position with respect to the vehicle, and based on the vehicle’s angle and position with respect to the surface. However, the orientation of a road surface can vary due to slopes, undulations, and localised surface variations. Accordingly, determining the position of the camera with respect to a road surface in front of the vehicle can be challenging.

[0047] Intrinsic parameters, such as focal length, aperture, field-of-view, resolution, may also effect the analysis of the road surface in front of the vehicle. Camera intrinsic parameters can be obtained through camera calibration. Vehicle pose may be obtained from a pose estimation framework. Road surface normal

[0048] A surface normal is a vector that is perpendicular to a surface at a particular location on that surface. A road surface normal comprises a surface normal for a road surface. A road surface normal can vary due to the unevenness of the road surface, slopes in the road (e.g. uphill or downhill roads), or a camber on the road (sloping to the left or the right). Accurate estimationof a road surface normal for a road surface is desirable for establishing spatial correspondence across frames.

[0049] The updated road surface normal can facilitate the extraction of meaning from the images captured by the vehicle camera. Accordingly, the improved determination of a road surface normal can advantageously improve one or more of lane detection, lane segmentation, road marking classification, puddle detection and puddle segmentation. Improved estimation of road surface normal

[0050] Embodiments provided herein provide an improved method for estimating a road surface normal vector for a road surface in front of a vehicle. The estimated road surface normal vector may be combined with the intrinsic and / or extrinsic parameters of an on-vehicle camera to yield a homography matrix between frames captured by the on-vehicle camera. The homography matrix may be applied, through homography transformation, to establish cross- frame spatial correspondences between the frames. Embodiments described herein apply a localised pixel-to-pixel attention mechanism to a sequence of frames, faciliated by the spatial correspondences, to determine a temporally consistent representation of occluded road lines or markings. This representation may ameliorate the effects of occlusions and facilitate the accurate detection of road markings. Image processing module

[0051] Figure 2 illustrates the architecture of an image processing module 106 configured to analyse road markings, in accordance with an embodiment. Figure 3 illustrates a method 300 for processing image data, in accordance with an embodiment. Method 300 may be performed by image processing module 106.

[0052] In operation 302, the image processing module 106 receives, as input, image data from the camera 180. The image data comprises a sequence of frames captured by the camera. The sequence of frames may comprise a video sequence, or a sequence of still images.

[0053] The sequence of frames are denoted as ^^ = { ^^^^| ^^ = { ^^ − ( ^^ − 1)Δ ^^}^^^^=1} as input. This sequence consists of a current frame ^^^^216 and ^^ − 1 previous frames sampled at a fixed time interval Δ ^^. The current frame ^^^^and ^^ − 1 previous frames are represented in system 200 as frames 214.

[0054] In operation 310, the image processing module 106 is configured to output image analysis information. In the embodiment illustrated in Figure 2, the image analysis information comprises a lane mark segmentation map ^^^^218 for the current frame ^^^^216. In otherembodiments, the image analysis information may comprise lane detection information, road markings information, puddle segmentation information or other results of the analysis of the images provided by the camera 180. Encoder

[0055] The image processing module 106 comprises an encoder (represented by both 204a and 204b, collectively 204). In operation 304, the encoder 204 is configured to encode the image data captured by the camera 180.

[0056] In one embodiment, the encoder comprises a convolutional neural network (CNN) configured to extract features from image data captured by the on-vehicle camera 180. A convolutional Neural Network (CNN) is a type of deep learning model that may be used for image classification and object recognition tasks. A CNN may be trained through a process of supervised learning, in which the algorithm is trained on a labelled dataset. CNNs may comprise a plurality of convolution layers, with the output of one layer being the input for the next layer. A feature map is the output of a convolutional layer. A feature map represents specific features in the input image or the feature map of the preceding convolutional layer. Convolutional layers can learn to recognize low-level features such as edges and corners, and then combine them to detect more complex features such as shapes and patterns.

[0057] The encoder 204 comprises a plurality of convolution layers for extracting feature maps from the frames (214 and 216) received from the camera 180. The encoder 204 is configured to encode a feature presentation (also referred to as a feature map) of the sequence of input frames ^^ including a current frame ^^^^and ^^ − 1 previous frames. The encoded feature map for the ^^^^ℎframe is denoted as ^^^^= { ^^^^^^∈ ℝ^^ ^^× ^^ ^^× ^^ ^^}^^^^=1where ^^ indicates the level of the feature map. The shallowest and deepest levels are represented by 1 and ^^, respectively. ^^^^, ^^^^, and ^^^^represent the height, width, and channel number of feature maps in level ^^. All encoded features are denoted as ^^ = { ^^^^^^| ^^ = { ^^ − ( ^^ − 1)Δ ^^}^^^^=1}. Homography guided fusion module

[0058] The image processing module 106 further comprises a homography guided fusion (HGF) module 210. The HGF module is configured to exploit temporally-adjacent video frames for complementary cues, facilitating the correct detection and classification of the partially occluded road lines or markings.

[0059] To find complementary information from adjacent frames, the HGF module uses the ground plane assumption for a road region in front of the on-vehicle camera 180 to fuse the feature maps of a plurality of frames. The HGF module uses an homography matrix to fuse the feature maps. The homography matrix is determined based on an estimate road surface normal. Accordingly, in operation 400, the HGF module estimates the road surface normal and from that estimated road surface normal, the HGF module determines a homography matrix. Operation 400 is detailed in Figure 4, in accordance with an embodiment.

[0060] In operation 306, the HGF module is configured to use the homography matrix determined in operation 400, to apply a homography transformation to fuse the feature maps of a current frame 230 with the feature maps of previous frames 240 to create fused feature maps 235. The fused feature maps comprise a temporally consistent representation of the road surface in front of the vehicle.

[0061] A temporally consistent representation of the road surface may ameliorate occlusions that hinder road markings analysis. The amelioration of occlusions relies on applying an accurate homography transformation matrix, to provide accurate cross-frame spatial correspondence which in turn faciliates fusion of the feature maps. A homography transformation matrix may be determined based on intrinsic and extrinsic parameters of the on- vehicle camera, and a road surface normal. Accordingly, to determine an accurate homography, it is desirable to determine an accurate road surface normal.

[0062] The HGF module 210 comprises a road surface normal estimator (RSNE) module 212. The RSNE module is configured to determine an estimated road surface normal based on the current frame ^^^^216 and at least one of the ^^ − 1 previous frames 214.

[0063] The HGF module 210 further comprises a homography matrix generator (HMG) 260, configured to determine a homography matrix based on the road surface normal estimated by the RSNE module 212.

[0064] The HGF module 210 is configured to apply a homography transform to the feature maps (240a and 240b) of the current frame and n − 1 previous frames. Applying a homography transformation comprises applying a homography matrix 260 to feature maps 240a to produce temporally consistent representations of the plurality of frames. The homography matrix is guided by a road surface normal 250 that has been estimated by the RSNE module 212.

[0065] The homography transformation matrix between the current frame and each of the previous frames is calculated using the estimated normal vector of the road surface ^^ determined by the RSNE module 212. Decoder

[0066] The image processing module 106 further comprises a decoder 208. In operation 308, the decoder is configured to decode the fused feature maps 235, to extract image analysis information in the form of surface marking information. Surface marking information may comprise, but is not limited to road marking information, lane detection, lane segmentation, road marking categorisation, puddle detection, puddle segmentation, or a combination thereof. Surface marking information may comprise information regarding markings painted onto the road surface, such as lane markings, turning arrows, or other traffic signals. In some embodiments, in operation 310, the image processing module 106 provides the image analysis information to the vehicle controller 108, to facilitate control of the vehicle 150.

[0067] In accordance with the embodiment of Figure 2, the decoder 208 decodes and upsamples the temporally consistent feature representations 235 to produce the lane mark segmentation prediction ( ^^ ^^ ^^ ^^ ^^) 218. Method for estimating road surface normal

[0068] Figure 4 illustrates a process 400 for estimating a road surface normal, in accordance with an embodiment. Process 400 may be performed by the RSNE module 212.

[0069] In some embodiments, the RSNE module is configured to repeat the process 400 for each frame captured by the camera 180. The RSNE module iteratively updates the estimated road surface normal with the projection error of on-road pixels from the current frame onto the previous frames.

[0070] In some embodiments, the RSNE simplifies the 8 degrees of freedom (DoF) of homography problem to a 2 DoF of normal vector problem.

[0071] In operation 402, the RSNE module 212 estimates an initial surface normal for a portion of the road in front of the vehicle, in the direction of travel. In one embodiment, the RSNE module estimates that the initial road surface normal is a vector being perpendicular to a horizontal plane and pointing upwards.

[0072] In operation 404, the RSNE module 212 determines an initial homography matrix based on the initial road surface normal determined in operation 402. The RSNE module mayalso be configured to determine the initial homography matrix based on the intrinsic parameters of the on-vehicle camera 180. The RSNE module may also be configured to determine the initial homography matrix based on the extrinsic parameters of the on-vehicle camera 180. Pixel sampling example

[0073] A frame representing an image captured by the camera 180 may depict a view of a spatial area in front of the vehicle, in the direction of travel. Accordingly, the frame may indicate pixels which represent a road surface and non-road surfaces. Non-road surfaces may comprise, but are not limited to, a road verge, a sidewalk, a road island, buildings, trees, other vehicles.

[0074] Figure 5 comprises a representation of an ordered sequence of three frames (500a, 500b and 500c) captured by a on-vehicle camera 180, pointed in the direction of travel of a vehicle in motion, in accordance with an embodiment. The three frames were captured at different points in time, with frame 500a being captured first, and frames 500b and 500c being captured sequentially thereafter. In one embodiment, the frames were captured at predefined intervals (e.g.500 milliseconds apart). In one embodiment, the frames were extracted, by the image processing module 106 from a video sequence.

[0075] With reference to frame 500c, the vehicle’s hood 502 is visible in the field of view, as is a road surface 504, and road line markings 506 and 508. The frames also depict off-road surfaces and objects, such as those representing trees 512 and the sky 514.

[0076] As illustrated in frames 500a-c, the vehicle 502 is driving forward, veering from the right hand side, to travel straight on the right hand side of the road. Sampling on-road pixels

[0077] Figure 6 illustrates the frames 500a-c of Figure 5, as annotated to illustrate a focal area on the road surface, and to illustrate spatial correspondence between pixels across frames, in accordance with an embodiment.

[0078] In some embodiments, since road markings (including road lines) are located on the road surface 110, the focal area for locating and processing road markings can be limited to an area of the captured image that depicts the road surface.

[0079] In some embodiments, the RSNE module 212 is configured to estimate the road surface normal by sampling the pixels of the current frame from a focal area in front of the car, wherein the focal area is a region of the frame that is likely to contain pixels that depict the road surface (e.g. on-road pixels).

[0080] For frames provided by a forward facing camera in the direction of travel of the vehicle, the on-road pixels are often located within a substantially triangular portion of the frame, wherein the topmost point of the triangle points towards the vanishing point within the field of view of the camera. Accordingly, in some embodiments, the focal area is triangular shaped, substantially triangularly shaped or trapezoidal. In other embodiments, the focal area may be not triangular shaped. The focal area may be irregularly shaped.

[0081] In some embodiments, a triangular focal area is determined by aligning the two bottom vertices of a triangle with the bottom of an image captured by the camera, and aligning the top vertex of the triangle with the vanishing point of the image. In embodiments, the vanishing point of an image relates to the extrinsic parameters of the camera, such as the camera pitch angle facing the road plane, and the intrinsic parameters of the camera, such as the focal length. Alternatively, or additionally, the vanishing point can be observed directly from the 2D image captured by the camera.

[0082] In some embodiments, the triangle may be skewed in the direction of travel of the vehicle. For example, if a vehicle is veering or turning right, as is the case in the example illustrated by Figure 5, the triangle may be shaped so as to capture more road surface to the right of centre of the vehicle. In some embodiments, the shape of the triangle may depend on one or more of: the position of the camera on the vehicle; the angle of view of the camera; the road surface normal; the travel speed of the vehicle; the travel direction of the vehicle; a change in direction of the vehicle; or any combination thereof.

[0083] In some embodiments, because the deep features, of the feature maps, focus on on- road information, an imprecise alignment of the focal area with the on-road surface displayed in the camera images may be acceptable. In some embodiments, the configuration of the triangle's vertices is specified in a configuration file accessible by the image processing module 106.

[0084] In frame 500c, the focal area is represented by the triangular area bounded by dashed line 602.

[0085] In operation 406, of process 400, the RSNE module samples ^^=1from a current frame. For illustration purposes, in Figure 6, frame 500c has been annotated to illustrate a plurality of pixels sampled in operation 406. Each sampled pixel is illustrated by a small hollow circle, for example a pixel is indicated by reference 604. Preferrably, each of the sampled pixels are located promixate to the triangular boundary 602. Preferrably, each of thesampled pixels are located within the triangular boundary 602. Preferrably, each of the sampled pixels is a on-road pixel. A sample pixel may depict an occlusion to the road surface. Determining spatially corresponding pixels

[0086] The road surface is substantially a plane, at least within the immediate front of the vehicle where markings are readable. Therefore, using a ground plane assumption a homography transformation may be applied to map spatially corresponding road pixels between frames.

[0087] In operation 408, for each pixel of the plurality of pixels sampled in operation 406, the RSNE module computes the spatially corresponding ^^^in the previous frames byapplying the initial homography matrix (determined in operation 404) computed with the initial road surface normal (determined in operation 402).

[0088] According to the standard inverse homography, for each sampled pixel ^^^^= ( ^^, ^^)⊤of the current frame, its spatially corresponding pixels of the reference frames { ^^^^| ^^ = { ^^ − ( ^^ − 1)Δ ^^}^^^^=1} can be computed using Equation (1). ^^^^where: ∝ represents proportional; ^^ is the intrinsic matrix of the on-vehicle camera; ^^^^and ^^^^represent the relative rotation and translation between the current frame ^^^^and the ^^^^ℎprevious frame ^^^^; ^^ is the normal of the observed road surface, which is estimated by the RSNE module V12; ^^ is the vertical distance between the on-vehicle camera and the road surface obtained from the camera calibration; and ⊕ 1 converts the point coordinates to homogeneous coordinates.

[0089] With reference to the example of Figure 6, in operation 408, the RSNE module 212 is configured to map each of the sampled pixels of frame 500c to spatially corresponding pixels in previous frames (e.g. frames 500a and 500b).

[0090] In response to applying the initial homography matrix to frame 500b, the RSNE module determines that pixel 608 (e.g a second pixel) of frame 500b spatially corresponds with pixel 604 (e.g. a first pixel). Similarly, in response to applying the initial homography matrix to frame 500a, the RSNE module determines that pixel 612 of frame 500a spatially corresponds with pixel 604.Computing a residual

[0091] In operation 410, the RSNE module 212 is configured to compute a residual between the low-level features of spatially corresponding pixels across the frames.

[0092] In one embodiment, the RSNE module 212 computes a residual between the low-level features of spatially corresponding pixels across the frames with Equation (5) ^^ ^^ ^^^^ ^^ ^^ − ^^ ^^ ^^^^where the residual is set to 0 for the sample point whose projections in previous frames fall outside the frame area.

[0093] In one embodiment, the RSNE module 212 computes an overall projection error, based on the residual, with Equation (6) ^^ ^^where ∥. ∥22 shows squared norm and ^^() is a robust cost function described in Reference [3]. Updating the road surface normal

[0094] In operation 412, the RSNE module 212 updates the surface normal based on the residual computed in operation 410.

[0095] Preferably, the RSNE module 212 iteratively minimises the non-linear least-square cost towards correct normal estimation using the Levenberg–Marquardt algorithm, as described in References

[0020] and

[0028] ;

[0096] The RSNE module 212 computes an adjusted estimated surface normal value by adding the surface normal delta given by Equation (12). Road surface normal vector

[0097] In embodiments, the surface normal vector is a unit vector with two degrees of freedom (DoF), which can be decomposed into a pitch angle ^^ and a roll angle ^^, as shown in Equation 7. ^^ = (−sin ^^cos ^^, −cos ^^cos ^^, sin ^^)⊤(7)

[0098] The Jacobian of the residual function with respect to the pitch angle and the roll angle may be defined in Equation 8, by applying the chain rule,^^^^ ^^^^ ^^^^ ^^^1^[ ^^ ^^ ^^]^^ ^^^^ ^^^^ ^^ ^^^^1[ ^ ^^ ^^ ^^^^^^^] where ^^ represents the to-be-optimized target ( is gradient from 2D interpolation.(9) defines the Jacobian (^^^

[0099] Equation^ ^^^^^^ ^^ ) of the homography transformation (Equation (1)) ^^, ^^^^ ^ ⊕ 1)⊤^ ) (9)^^ ^^

[0100] in Eq.10 and Eq.11, is the Jacobian of Eq. (7) ^^ and ^^ respectively^^ ^^ ^^ ^^where ^^^^represents the ^^-th element of ^^.

[0101] The RSNE module 212 obtains the Hessian matrices as ℍ = ^^⊤^^′ ^^, where ^^′ is the derivative of function ^^(). The RSNE module computes a surface normal delta by damping the Hessian and solving the linear system, as shown in Equation (12), △ ^^ = −(ℍ + ^^diag(ℍ))−1^^⊤^^′ ^^, (12) where ^^ is the damping factor

[0036] , balancing between the Gauss-Newton ( ^^ = 0) and gradient descent ( ^^ = ∞) and △ ^^ is the surface normal delta. Compare to threshold

[0102] In some embodiments, the RSNE module 212 is configured to, in operation 414, compare the surface normal delta to a threshold. In embodiments, the size of the surface normal delta may be indicative of the closeness of the estimated road surface normal to an actual road surface normal of the road surface.

[0103] The RSNE module may be configured to, in response to the surface normal delta △ ^^ exceeding a threshold, perform operations 408, 410, 412 and 414 again. In response to the surface normal delta △ ^^ being less than a threshold, the RSNE module may be configured toproceed to operation 416. According, in embodiments, the RSNE module 212 is configured to estimate an accurate road surface normal by applying an iterative optimization process.

[0104] In some embodiments, the RSNE module 212 is configured to iteratively optimise the estimated road surface normal through the following algorithm: Input: 1stlevel feature maps: ^^1= ^^^1^ Sample Points Coordinates: Initial Pitch and Roll Value: ;Damping Factor: ^^ ^^ ^^Hyper-parameter: ^^; Maximum loop count, empirically set to 20 ^^; Convergence threshold, empirically set to 0.0001 Output: Optimized Pitch and Roll ( ^^, ^^); function OPT ^^0, ^^0, ^^1, ^^^^, ^^ Derive point features ^^^1^[ ^^^^] from ^^^1^; for ^^ ← 1 to ^^ do Calculate point coordinates ^^^^in ^^^1^ (Eq.1); Derive point features ^^^1^ [ ^^^^] from ^^^1^ ; Calculate residual ^^^^(Eq.5); Calculate observe error ^^ (Eq.6); Calculate robust cost ^^( ^^) and its derivation ^^′; Construct ^^ and its Hessian matrices ℍ (Eq.8); Obtain △ ^^ by Cholesky decomposition (Eq.12); Update road surface normal as ( ^^,^^← ( ^^, ^^)^^−1+△ ^^; if MAX(△ ^^) < ^^ then Break; end if end for end function

[0105] In operation 416, the HGF module 210 updates the homography matrix based on the updated estimate road surface normal determined in operation 412. In particular, the HGF module calculates an adjusted homography matrix 260 by applying Equation (1) with theadjusted estimated surface normal (determined in operation and the camera intrinsic and extrinsic parameters. Re-determining spatially corresponding pixels

[0106] In embodiments, for each pixel of the plurality of pixels sampled in operation 406, the RSNE module computes the spatially corresponding ^^ ^^^in the previous frames by applying Equation (1) with the adjusted homographyin operation 416), and the adjusted surface normal (determined in operation 412).

[0107] In response to applying Equation (1) with the adjusted homography matrix to frame 500b, the RSNE module determines that pixel 606 (e.g. a third pixel) of frame 500b spatially corresponds with pixel 604. Similarly, in response to applying the adjusted homography matrix to frame 500a, the RSNE module determines that pixel 610 of frame 500a spatially corresponds with pixel 604.

[0108] In embodiments, the HGF module may compute a residual between the features of pixel 604 and pixel 606. Furthermore, the HGF module may compute a residual between the features of pixel 604 and pixel 610. Based on these residuals, the HGF module may further adjust the surface normal. Use of low level features

[0109] In embodiments, the RSNE module 212 uses low-level features ^^1240a to determine the normal vector ^^ 250. While the high-level features ^^2focus on segmentation feature expression, the low-level features ^^1are supervised to extract useful features for both segmentation and road surface normal estimation. As both tasks focus on on-road objects, they can benefit each other. Attention mechanisms

[0110] An attention mechanism may be used to improve the performance and interpretability of neural networks, particularly in tasks involving sequential or structured data. An attention mechanism enables the neural network to selectively focus on different parts of the input data, assigning different levels of importance or relevance to each element. By incorporating an attention mechanism, a deep learning model can dynamically weigh and combine information from different parts of the input, allowing it to pay more attention to the relevant features and effectively ignore the irrelevant ones. A global attention mechanism may apply attention to the entire (or a significant portion of) the feature map produced by an encoder encoding an image.

[0111] Advantageously, embodiments described herein utilise a local attention mechanism which applies attention to a feature map produced by an encoder encoding only a relevant information of the image, thereby reducing encoder complexity. In embodiments, the local attention mechanism comprises a cross-frame pixel-to-pixel attention mechanism. The cross- frame pixel-to-pixel attention mechanism may be referred to as a temporal attention mechanism. In embodiments, the relevant information comprises corresponding pixels across sequential frames, wherein the pixels are located in a focal area of the image. Temporally consistent representation

[0112] The HGF module 210 applies a local attention mechanism to determine a temporally consistent representation for each sampled pixel of the current frame. In particular, in some embodiments, the HGF module 210 employs a temporal attention mechanism on the feature presentations of spatially corresponding pixels.

[0113] In operation 306, the HGF module 210 applies an attention mechanism to fuse the feature map of the current frame 230 with the feature maps of the previous frames 240 to produce a fused feature map ( ^^^^= { ^^^^^^}^^^^=1). Preferrably, the attention mechanism comprises a cross-frame pixel-to-pixel attention mechanism, rather than a more computationally expensive global attention mechanism.

[0114] With reference to the embodiment illustrated in Figure 2, feature map 235a comprises the fused feature map for the shallowest level, and feature map 235b comprises the fused feature map for the deepest level.

[0115] In one embodiment, the attention mechanism comprises projecting the pixels of the current frame ^^^^onto the previous frames through a homography transformation using the homography matrix 260 determined in light of the surface normal 250 estimated by the RSNE module 212.

[0116] The decoder produces the road marking analysis information for the current frame based on the fused feature maps 235a and 235b. In one embodiment, the road marking analysis information comprises a segmentation prediction ( ^^^^) Queries, keys and values

[0117] In some embodiments, the attention mechanism operates using three main components: queries, keys, and values. These components are derived from the input data and play different roles in the attention process. Queries may be used to derive a compatibility scoreor weight for each element in the input sequence. Keys represent the features or attributes associated with each element in the sequence, and values contain the actual information or representation of each element.

[0118] To compute the attention weights, the queries may be compared to the keys using a similarity measure, such as dot product, cosine similarity, or a learned function. The result is a set of scores that reflect the relevance or similarity between the query and each key. These scores are then passed through a softmax function to obtain a probability distribution that sums to 1, representing the attention weights assigned to each element.

[0119] The attention weights may be used to weigh the corresponding values. The weighted values may then be combined or aggregated to form a context vector, which captures the attended information. This context vector may be fed into subsequent layers of the neural network, allowing the model to focus on the most relevant parts of the input during different stages of processing.

[0120] The use of queries, keys, and values in the attention mechanism allows the HGF module 210 to perform selective information processing, where the importance of each element in the input sequence is dynamically determined based on the context and the task at hand.

[0121] In one embodiment, the attention mechanism applied by the HGF module 210 comprises mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.

[0122] The attention mechanism assigns the pixel representation of current frame ( ^^^^[ ^^^^]) as a query and the pixel representation of all frames ({ ^^^^[ ^^^^]}) as keys and values.

[0123] Advantageously, the attention mechanism omits the spatial encoding in the presence of the spatial correspondence across frames determined by appying the homography matrix 260 determined based on the estimated surface normal 250.

[0124] Advantageously, the HGF module 210 is configured to apply an L2 normalization on the query and the keys to obtain more robust (deep) feature similarities across various lighting environments. L2^^^^=∥ ^^^^[ ^^^^]∥2⋅∥ ^^^^[ ^^^^]∥2where ∥. ∥2indicates an ^^2normalization.

[0125] The HGF module 210 normalises the feature similarities by applying the softmax function, as given in Equation (3): exp ^^ ^^^^^^=

[0126] The HGF module 210 uses the result of Equation (3) as a weight to fuse the spatially corresponding pixels across the frames, as shown in Equation (4): ^^^^[ ^^^^] = ^^^^[ ^^^^] + ∑^^^^^^^^^^[ ^^^^]. (4)

[0127] The result produced by Equation (4) comprises the fused features maps 235a and 235b. Example implementation

[0128] An implementation of the present technology was developed using a pre-trained EfficientNet (Reference

[0042] ) convolutional neural network to extract image features at two different scales, 4 × and 16 × down-scaling, with 64 and 128 channels, respectively.

[0129] To focus on the bottom region of the input image, where road marks are typically visible, the bottom 40% of the input image was cropped. The processed image size is set to 272 × 848 pixels. The decoder consisted of bi-linear up-sampling layers and convolution layers, which up-sample the high-level features and increase the resolution by a factor of 4 in each level. The model using the AdamW optimizer

[0050] for 30,000 iterations on two NVIDIA RTX 3090 GPUs, with a learning rate of 4 × 10−3.

[0130] Experiments were conducted using two datasets, ApolloScape

[0017] and ApolloScape Night. For this example implementation, other datasets were deemed unsuitable due to missing camera extrinsics, insufficient frame overlap, or the absence of segmentation labels.

[0131] The ApolloScape dataset is a large-scale dataset that can be used for localization and segmentation tasks. It consists of 38 distinct classes and poses various challenges including occlusions and tiny road markings. To provide accurate camera poses, the vehicle is equipped with customer-grade global positioning system (GPS) and inertial measurement unit (IMU). However, for the lane mark segmentation task, only 41,201 annotated images out of more than 110,000 + are associated with camera pose information. These 41,201 images were used as the testset images for the experiments since the approach uses on camera poses. The textset images were divided into 35,173 training images and 6,028 validation images. Because the methods described herein uses sequentially captured image information, the validation set with completely different trajectories were selected to ensure a fair comparison.

[0132] Since there is a lack of datasets for night time lane mark segmentations, an artificial dataset was created called ApolloScape Night from the ApolloScape dataset using a cross- domain generation network (Reference [1]). The ApolloScape Night dataset allowed embodiments of the image processing module 106 to be evaluated on a challenging dataset with poor lighting conditions, road reflection, and glare.

[0133] In accordance with the guidelines of the ApolloScape benchmark

[0017] , a mean intersection-over-union (mIoU) was used as the evaluation criterion. The ApolloScape dataset contains 38 different types of lane markers, including two ignorable labels (noise and ignored). Of these labels, 18 categories are used in the official evaluation metrics. To provide a comprehensive evaluation of the experiemental approach, the evaluated mIoU results for both the selected 18 categories, denoted as 18 mIoU, and all 36 categories, denoted as 36 mIoU are reported. By reporting both sets of results, a more complete picture of the performance of the method on the ApolloScape and ApolloScape Night datasets is provided. Example results

[0134] Figure 7 illustrates the effect of an implementation of the HGF module 210 on frames captured from a travelling vehicle 820, in accordance with an embodiment. As indicated by lines 850, the HGF module explores the adjacent frames for cues, facilitating the correct classification of (1) a Straight Arrow (as illustrated in frame 806), which, with its bottom half occluded by a vehicle 830, is mistakenly classified as a Right Turn & Straight Arrow by an embodiment without the HGF module (as illustrated in frame 808), and (2) a partially occluded Dotted Line, which is incorrectly classified as a Solid Line without the HGF module.

[0135] The fused frame 802 in the 4^^ℎrow and the 2^^ ^^column demonstrates the recovered road lines and markings after projecting the previous frames onto the current frame with the estimated homography matrices. Boxes 804, 806 and 808 each enlarge the area where mistake classifications are corrected. Linked boxes 810a, 810b, 810c, 810d indicate the spatially corresponding area across the frames. Performance

[0136] Experiments on ApolloScape, a large-scale lane mark segmentation dataset, and ApolloScape Night with artificial simulated night-time road conditions, demonstrate that embodiments of the image processing module 106 outperform other existing SOTA lane mark segmentation models with less than 9% of their parameters and computational complexity.

[0137] Embodiments provided herein exploit available camera intrinsic data and ground plane assumption for cross-frame correspondence that can lead to a light-weight network with significantly improved performances in speed and accuracy.

[0138] These design decisions allow embodiments of the image processing module 106 to outperform the existing lane mark segmentation models with less than 9% of their parameters and Giga Floating Point Operations (GFLOPs). Qualitative comparison

[0139] The performance of an embodiment of the HomoFusion module was compared with the performance of other algorithms, including (a) IntRA-KD

[0016] , (b) SegFormer

[0047] , (c) CFFM

[0038] , and (d) MMA-Net

[0051] , on the ApolloScape

[0017] and ApolloScape Night datasets. To ensure a fair comparison, each model was retrained on the same subset of the training set with the same input resolution. The results of the comparison are presented in Table 1200 of Figure 9. Notably, the embodiment outperforms the existing methods on both the ApolloScape

[0017] and ApolloScape Night datasets while having less than 9% of their parameters or computational overhead.

[0140] In addition, an embodiment of the HomoFusion module was compared with CFFM

[0038] which also uses adjacent frames to enhance the representation of the current frame. The embodiment was more efficient in terms of model complexity, with a complexity of ^^( ^^ ^^ ^^) for feature extraction and ^^( ^^ ^^ ^^2) for cross-frame feature fusion. In contrast, the complexity of CFFM is ^^( ^^2^^2^^) + ^^( ^^ ^^ ^^2) for feature extraction and ^^( ^^ ^^ ^^ ^^) + ^^( ^^ ^^ ^^2) for cross-frame feature fusion, where ^^ is calculated by their receptive field and pooling kernel size. This explains why embodiments described herein have a lower computational requirement for the cross-frame pixel-to-pixel attention mechanism compared to the global attention mechanism used by CFFM.

[0141] The qualitative results of both the existing models and embodiments described herein indicate that the embodiments described herein achieve superior segmentation performance on road lines and markings, even under adverse conditions such as occlusion, road reflection, and poor lighting. Additional visualization examples for various categories can be found in the appendix. Study on Hyper-parameters

[0142] Figure 8 comprise four graphs illustrating the performance impact of the number of frames, and the sampling frame gap, in accordance with an embodiment. In particular, theimpact of the number of frames ^^ on ApolloScape (AS) and ApolloScape Night (ASN) datasets is shown in graphs (a) and (b), while graphs (c) and (d) show the effect of the sampling frame gap △ ^^.

[0143] Number of Frames ( ^^). A study was conducted on the impact of the number of frames, while setting the sampling frame gap △ ^^ = 2. The results, presented in Figure 8 (a) and (b), demonstrate that the performance of an embodiment of method disclosed herein improves on both ApolloScape and ApolloScape Night datasets with an increase in the number of explored adjacent frames. However, the performance improvement becomes negligible when the number of frames exceeds 4. Considering the monotonically increasing relationship between the model complexity and the number of frames, ^^ was set to 4 to strike a balance between model performance and complexity.

[0144] Sampling Frame Gap (△ ^^). In the study on the effect of the sampling frame gap, the number of frames was set to ^^ = 4. Figure 8 (c)-(d) indicates that the optimal performance is achieved at △ ^^ = 2. This can be attributed to the balance between sufficient overlap of road area across the frames and a wide enough temporal gap that allowing the displacement of source of occlusion, e.g. another moving vehicle. Ablation studies

[0145] Figure 10 comprises a table 1300 indicating the results of an ablation study, in accordance with an embodiment. In particular, table 1300 indicates the results of an ablation study of an embodiment of the HGF module and an embodiment of the RSNE module. Ablation studies were conducted on the ApolloScape dataset and run on an NVIDIA RTX 3090 GPU. The number of frames was set to ^^ = 4 and the sampling frame gap was set to △ ^^ = 2. “18 mIoU" represents the mean IoU of 18 types of lane markers selected by ApolloScape official metrics, and “36 mIoU" represents the mean IoU of all unignorable 36 types.

[0146] The HomoFusion module (e.g. HGF module 210) allows the use of complementary information from sequential frames. Table 1300 demonstrates that an embodiment with the HomoFusion module (4 frames) achieves better performance than an embodiment without the HomoFusion module (1 frame), illustrating the effectiveness of the temporally consistent feature representation obtained with HomoFusion module in addressing the partial occlusion issues.

[0147] The effect of directly associating the pixels across frames based on their coordinates was also studied by using identity matrix as the homography matrix, denoted as w / oHomoGuide. It shows that such an incorrect cross-frame spatial correspondence, although leveraging additional frames, provides limited help in improving the model performance.

[0148] The impact of the RSNE module 212 was studied in the following model variants each comprising the HomoFusion modules: (a) without RSNE (HomoFusion w / o RSNE), where the homography matrix is estimated with the initial surface normal ( ^^0), (b) with RSNE using RGB (Red Green Blue) information (HomoFusion w / RGB RSNE) and (c) with RSNE with features (HomoFusion Full).

[0149] Results in Table 1300 show that HomoFusion Full achieves better performance than HomoFusion w / o RSNE, demonstrating the effectiveness of the iterative optimization in estimating a more accurate homography matrix that subsequently provides more accurate guidance on cross-frame spatial correspondences. It can be also observed that HomoFusion Full significantly outperforms HomoFusion w / RGB RSNE indicating that deep features are more robust than RGB information in estimating an accurate road surface normal. This performance improvement may be attributed to the fact the feature representations are less prone to adverse noises caused by environmental factors, e.g. brightness.

[0150] The image processing module may be configured to produce a fused frame sequence with the homography transformation between the frame pairs, in accordance with an embodiment. In embodiments, the road surface normal estimated with feature representation enables more accurate homography transformation than with RGB information, resulting in better alignment of road lines and markings across frames. In embodiments, a fused frame with road surface normal estimated with features has better alignment in road lines and markings than that with RGB information.

[0151] The robustness of an embodiment to extrinsic noise was investigated by introducing random translation errors of less than 1 meter and rotation errors of up to 30∘. The results are shown in Noisy Extrinsic row 1302 of Table 1300, which demonstrates that an embodiment of the image processing module 106 is capable of mitigating extrinsic noise and remains robust.

[0152] Figure 11 illustrates a table 1400 indicating the results of an ablation study regarding backbones, in accordance with an embodiment. In particular, Table 1400 indicates the results of a comparison of an embodiment of the image processing module 106 with different backbones on the ApolloScape Dataset.

[0153] In order to ensure a fair comparison with CFFM, Reference

[0038] , and MMA-Net, Reference

[0051] , an embodiment of the image processing module 106 was implemented with‘MiT-B1’ and ’ResNet-50’ backbone. The results of table 1400 indicate that an embodiment of the image processing module 106 outperforms CFFM

[0038] and MMA-Net

[0051] . Additionally, an embodiment of the image processing mdoule was tested with ‘EfficientNet-B4 / 5’ backbones, and the results in Table 1400 indicate that the embodiment has relative stable performance even when using smaller backbones.

[0154] Figure 12 illustrates a table 1500 indicating the performance results for water puddle segmentation, in accordance with an embodiment. An embodiment of the image processing module 106 was tested with the task of detecting water hazards on a road surface. Specifically, the test methodology aimed to determine the performance of the embodiment in performing binary segmentation to identify water puddles in the road. The Puddle-1000 dataset

[0013] was used. This dataset includes On-road and Off-road datasets. The frames obtained from an onboard camera on bumpy roads are not stable and contain small extrinsic noise. The intrinsic information of the camera is provided by the dataset, while the global extrinsic information of the camera is obtained using ORB-SLAM

[0030] . The experiments were conducted using the current and seven previously consecutive frames with a 240 × 320 image size. The results are presented in Table 1500. Derivation of Jacobian

[0155] Provided herein are the detailed derivations for Equation (9), Equation (10) and Equation (11), which calculate the Jacobian of the homography transformation, the pitch ^ and the roll ^ angle, respectively. The pitch angle^and the roll angle ^ represent the rotation of the road surface with respect to the plane constructed with the x-axis and z-axis of camera coordinates. Figure 15 illustrates the rotation of the road surface plane around the x-axis in accordance with the pitch angle 1802, in accordance with an embodiment. Figure 16 illustrates the rotation of the road surface plane around the z-axis in accordance with the roll angle 1902, in accordance with an embodiment.

[0156] In embodiments, the derivation of Equation (9) is the Jacobian of the Homography Transformation Equation (1), with respect to the road surface normal n .t n • i − 1 ^ ^ − ^

[0157] In embodiments, the derivation of Equation (10) is the Jacobian of road surface normal with respect to the the pitch angle^. ^ n ^ (sin ^ ) , ) ^ ^^ ^ = (sin ^ sin ^ ,cos ^ sin ^ ,cos ^ ) sin ^ cos ^ sin ^ cos ^ cos ^ sin ^ = ( , ,cos ^ ) cos ^ cos ^

[0158] In embodiments, the derivation of Equation (11) is the Jacobian of road surface normal with respect to the roll angle ^ . ^ ^Performance with estimated camera extrinsic

[0159] Camera extrinsic information may be readily obtained in a real vehicle system; however, camera extrinsic information may not be available for some datasets. To explore the performance of embodiments under extrinsic-unknown settings, the camera extrinsic information was estimated using COLMAP (as described in Reference

[0052] ) and used the estimated extrinsic to evaluate the performance of embodiments. In some test scenarios, the ApolloScape dataset’s larger camera elevation angle caused the key points obtained by COLMAP to be further away from the vehicle, resulting in noisy camera poses. Despite theseless accurate poses, embodiments described herein achieved favourable performances (as shown in row 2002 “w / COLMAP Extrinsic’ of Table 2000 of Figure 17).

[0160] Figure 17 illustrates a table 2000 comprising a comparison of estimated COLMAP extrinsics and OpenCV homography on the ApolloScape Dataset, in accordance with an embodiment. Performance with estimated homography

[0161] The FindHomography function from OpenCV package was used to directly estimate the homography transformation matrix. The estimations are less accurate than the HomoGuide, leading to inferior lane segmentation results, as shown in row‘w / OpenCV Homography’ of Table 2000). Challenging scenarios

[0162] The performance of an embodiment described herein was evaluated on challenging scenarios. Because the LM algorithm was used for road surface normal vector estimation, there is a convex range of the algorithm that can result in failure due to excessive re-projection errors. To further investigate this scenario, the top 100 inputs was selected with the highest residuals after optimization. This approach was compared with and without RSNE, and found that incorporating the RSNE led to improved performance (18 mIoU: 53.3 and 52.5, respectively), even in cases where the optimization has a high possibility of being non-convex.

[0163] Additionally, it was investigated whether an embodiment successfully estimates the road surface normal vector under uphill / downhill road scenarios. In an embodiment, the image processing module accurately estimates the surface normal in a scenario in which a the road surface comprises an uphill surface, resulting in good alignment of road marks across multiple frames. Visualization on real night / rain scenes

[0164] The effectiveness of an embodiment was verified in challenging real-world environments, dark and rainy situations.

[0165] In particular, segmentation results of real night frames were obtained using an embodiment of the HomoFusion model, which was trained on the artificial ApolloScape Night dataset, in accordance with an embodiment. The model accurately segments various road lines and markings, such as dot lines, solid lines, crosswalks, and straight arrows.

[0166] Segmentation results of real rainy day frames were also obtained using an embodiment of the HomoFusion model, which was trained on the artificial ApolloScape Night dataset, in accordance with an embodiment. The model accurately segments various road lines and markings, including dot lines, solid lines, double yellow lines, stop lines, and crosswalks.

[0167] Despite being trained only on the artificial ApolloScape Night dataset, which simulates road conditions at night, the embodiment under test successfully segments various road lines and markings in real images. Visualization on HomoFusion

[0168] In embodiments, incorporating HomoFusion (e.g. with the HGF module 210, also referred to as HomoGuide) enables the image processing module to accurately classify road lines and markings even under adverse conditions such as occlusion, road reflection, and poor light conditions. In embodiments, HomoFusion facilitates the correct classification of partially occluded road markings. Visualization of Water Hazard Detection

[0169] Embodiments of the image processing module can align ground surface to improve the performance of detection of flat objects (such as water puddles) on the ground.

[0170] Embodiments of the image processing module are configured to transform input frames to provide predicted segmentations of water puddles that are depicted in the input frames.

[0171] The predicted segments may be visually compared with ground truth masks, which illustrate actual water puddle segmentations, to determine the performance of the image processing module Extra Qualitative Examples

[0172] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. Furthermore, it will be appreciated by persons skilled in the art that embodiments disclosed herein can be combined with one or more other embodiment disclosed herein, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

[0173] It will be appreciated by persons skilled in the art that any suitable distribution of functionality between different functional units may be used without detracting from the invention. For example, functionality illustrated to be performed by separate computing devices may be performed by the same computing device. Likewise, functionality illustrated to be performed by a single computing device may be distributed amongst several computing devices. Hence, references to specific functional units are only to be seen as references to suitable means for providing the described functionality, rather than indicative of a strict logical or physical structure or organization.

[0174] It will be appreciated by persons skilled in the art that, for processes and methods disclosed herein, the operations performed in the processes and methods may be implemented in differing order. Furthermore, the outlined steps and operations are only provided as examples, and some of the steps and operations can be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments.

[0175] References herein to software or executable instructions are to be understood as referring to executable instructions stored in volatile or non-volatile memory. The memory can include any data storage device that can store data which can thereafter be read by a processor. Examples of memory include read-only memory (ROM), random-access memory (RAM), magnetic tape, optical data storage device, flash storage devices, or any other suitable storage devices.

[0176] Throughout this specification the word ‘comprise’, or variations such as ‘comprises’ or ‘comprising’, will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

[0177] As used herein, any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment. Similarly, use of “a” or “an” preceding an element or component is done merely for convenience. This description should be understood to mean that one or more of the element or component is present unless it is obvious that it is meant otherwise.

[0178] Unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present). References [1] Vinicius F Arruda, Thiago M Paixao, Rodrigo F Berriel, Alberto F De Souza, Claudine Badue, Nicu Sebe, and Thiago Oliveira-Santos. Cross-domain car detection using unsupervised image-to-image translation: From day to night. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2019. [3] Jonathan T. Barron. A general and adaptive robust loss function. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.

[0013] Xiaofeng Han, Chuong Nguyen, Shaodi You, and Jianfeng Lu. Single image water hazard detection using fcn with reflection attention units. In Proceedings of the European Conference on Computer Vision (ECCV), pages 105–120, 2018.

[0016] Yuenan Hou, Zheng Ma, Chunxiao Liu, Tak-Wai Hui, and Chen Change Loy. Inter- region affinity distillation for road marking segmentation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 12486–12495, 2020.

[0017] Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, PengWang, Yuanqing Lin, and Ruigang Yang. The apolloscape dataset for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 954–960, 2018.

[0020] Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics, 2(2):164–168, 1944.

[0028] Donald W Marquardt. An algorithm for least-squares estimation of nonlinear parameters. Journal of the society for Industrial and Applied Mathematics, 11(2):431–441, 1963.

[0030] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):1147–1163, 2015.

[0036] Paul-Edouard Sarlin, Ajaykumar Unagar, M˚ans Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, and Torsten Sattler. Back to the Feature: Learning robust camera localization from pixels to pose. In CVPR, 2021.

[0038] Guolei Sun, Yun Liu, Henghui Ding, Thomas Probst, and Luc Van Gool. Coarse-to-fine feature mining for video semantic segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

[0042] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR, 2019.

[0047] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077–12090, 2021.

[0050] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019.

[0051] Yujun Zhang, Lei Zhu, Wei Feng, Huazhu Fu, Mingqian Wang, Qingxia Li, Cheng Li, and Song Wang. Vil-100: A new dataset and a baseline model for video instance lane detection. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 15681–15690, 2021.

[0052] Johannes Lutz Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

Claims

CLAIMS:

1. A method for estimating a surface normal from a plurality of images of a surface, the method comprising: obtaining, from a camera positioned at a first viewpoint of the surface, a first image depicting at least a portion of the surface; obtaining, from the camera positioned at a second viewpoint of the surface, a second image depicting at least the portion of the surface; determining a homography matrix based on an initial estimated surface normal of the portion of the surface; selecting a first pixel from the first image, wherein the pixel represents a location on the portion of the surface; determining, by applying a homography transformation to the first image using the homography matrix, a second pixel from the second image, the second pixel spatially corresponding to the first pixel; determining, by applying a trained neural network to at least the first image, a feature map of the first image, the feature map of the first image comprising features of the first pixel; determining, by applying the trained neural network to at least the second image, a feature map of the second image, the feature map of the second image comprising features of the second pixel; determining a residual between the features of the first pixel and the features of the second pixel; and adjusting the estimated surface normal based on the residual.

2. The method of claim 1, wherein the camera is attached to a vehicle in motion on the surface.

3. The method of any one of claims 1 or 2, wherein the first viewpoint comprises a position of the vehicle on the surface at a first point in time, and the second viewpoint comprises a position of the vehicle on the surface at a second point in time.

4. The method of any one of claims 1 to 3, wherein the first image and the second image comprise sequential frames of a video captured by the camera.

5. The method of any one of claims 1 to 4, wherein the first pixel is selected from a focal area of the first image.

6. The method of claim 5, wherein the focal area depicts a road surface in front of the vehicle in the direction of travel.

7. The method of any one of claims 1 to 6, wherein adjusting the estimated surface normal comprises: determining a surface normal delta based on the residual; and adjusting the estimated surface normal by applying the surface normal delta to the estimated surface normal.

8. The method of claim 7, wherein determining a surface normal delta comprises: determining a non-linear least squares cost based on the residual; and determining the surface normal delta by applying a damped least-squares method to the non-linear least squares cost.

9. The method of claim 7, further comprising: in response to comparing the surface normal delta with a threshold, adjusting the estimated surface normal by applying the surface normal delta to the estimated surface normal.

10. The method of claim 7, wherein adjusting the estimated surface normal based on the residual comprises, performing at least one iteration of: determining an adjusted surface normal by applying the surface normal delta to the estimated surface normal; adjusting the homography matrix based on the adjusted estimated surface normal; determining, by applying the adjusted homography matrix to the first image, a third pixel from the second image, the third pixel spatially corresponding to the first pixel; determining, by applying a trained neural network to at least the first image, a feature map of the first image, the feature map of the first image comprising features of the first pixel;determining, by applying the trained neural network to at least the second image, a feature map of the second image, the feature map of the second image comprising features of the third pixel; determining a second residual between the features of the first pixel and the features of the third pixel; and determining a second surface normal delta based on the second residual.

11. The method of claim 10, further comprising: in response to performing a predefined number of iterations of the method of claim 10: adjusting the estimated surface normal by applying the second surface normal delta to the estimated surface normal.

12. The method of any one of claims 1 to 11, wherein determining the residual comprises determining the residual between the low-level features of the feature map of the first pixel and the low-level features of the feature map of the second pixel.

13. The method of any one of claims 1 to 12, wherein the method is performed responsive to obtaining a new frame from the camera.

14. The method of any one of claims 1 to 13, wherein the surface normal comprises a unit vector with two degrees of freedom, the degrees of freedom comprising a pitch angle and a roll angle.

15. The method of any one of claims 1 to 14, wherein the homography matrix is determined based on an intrinsic homography matrix associated with the camera.

16. The method of any one of claims 1 to 15, further comprising: determining an adjusted homography matrix based on the adjusted estimated surface normal.

17. The method of claim 16, further comprising: determining a fused feature map by applying the adjusted homography matrix to a feature map of the first image and by applying the adjusted homography matrix to a feature map of the second image.

18. The method of any one of claims 1 to 17, further comprising computing a temporally consistent representation for a plurality of pixels of the first image by employing a temporal attention mechanism on the fused feature map.

19. The method of claim 18, wherein the temporal attention mechanism comprises a local attention mechanism.

20. The method of claim 18, wherein the temporal attention mechanism assigns pixel representation of the first image as a query and pixel representations of the second image as keys and values.

21. The method of claim 18, wherein the temporal attention mechanism obtains a L2 normalization on the query and the keys.

22. The method of any one of claims 18 to 21, further comprising decoding the temporally consistent representation to produce image analysis information.

23. The method of claim 22, wherein the image analysis information comprises, road marking analysis information.

24. The method of claim 23, wherein the road marking analysis comprises one or more of: lane segmentation information; lane detection information; road markings information; and puddle segmentation information.

25. A computer-implemented method for estimating a surface normal from a plurality of images of a surface, the method comprising a processor performing the steps of: obtaining, from a camera positioned at a first viewpoint of the surface, a first image depicting at least a portion of the surface; obtaining, from the camera positioned at a second viewpoint of the surface, a second image depicting at least the portion of the surface;determining a homography matrix based on an initial estimated surface normal of the portion of the surface; selecting a first pixel from the first image, wherein the pixel represents a location on the portion of the surface; determining, by applying a homography transformation to the first image using the homography matrix, a second pixel from the second image, the second pixel spatially corresponding to the first pixel; determining, by applying a trained neural network to at least the first image, a feature map of the first image, the feature map of the first image comprising features of the first pixel; determining, by applying the trained neural network to at least the second image, a feature map of the second image, the feature map of the second image comprising features of the second pixel; determining a residual between the features of the first pixel and the features of the second pixel; and adjusting the estimated surface normal based on the residual.

26. A system for estimating a surface normal from a plurality of images of a surface, the system comprising: a camera attached to a vehicle in motion on a surface; one or more processors; and memory comprising computer executable instructions, which when executed by the one or more processors, cause the system to perform the method of any one of claims 1 to 24.

27. A machine-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 24.

28. A method for analysing surface markings from a plurality of images of a surface, the method comprising: obtaining, from a camera positioned a first viewpoint of the surface, a first image depicting at least a portion of the surface;obtaining, from the camera positioned at a second viewpoint of the surface, a second image depicting at least the portion of the surface; determining, by applying a trained neural network to at least the first image, a feature map of the first image; determining, by applying the trained neural network to at least the second image, a feature map of the second image; determining, by applying an attention mechanism to the feature map of the first image and the feature map of the second image, a fused feature map; determining, by decoding the fused feature map, the surface marking information.

29. The method of claim 28, wherein the camera is attached to a vehicle in motion on the surface.

30. The method of any one of claims 28 or 29, wherein the first viewpoint comprises a position of the vehicle on the surface at a first point in time, and the second viewpoint comprises a position of the vehicle on the surface at a second point in time.

31. The method of any one of claims 28 to 30, wherein the first image and the second image comprise sequential frames of a video captured by the camera.

32. The method of any one of claims 28 to 31, wherein the attention mechanism comprises a local attention mechanism.

33. The method of any one of claims 28 to 32, wherein the attention mechanism comprises a cross-frame pixel-to-pixel attention mechanism.

34. The method of claim 33, further comprising applying the cross-frame pixel-to-pixel attention mechanism, wherein applying the cross-frame pixel-to-pixel attention mechanism comprises: projecting one or more pixels of the first image onto the second image by applying a homography transformation.

35. The method of any one of claims 28 to 34, wherein the one or more pixels of the first image are selected from a focal area of the first image.

36. The method of any one of claims 28 to 35, wherein the focal area depicts a road surface in front of the vehicle in the direction of travel.

37. A system for analysing surface markings from a plurality of images of a surface, the system comprising: a camera attached to a vehicle in motion on a surface; and one or more processors; and memory comprising computer executable instructions, which when executed by the one or more processors, cause the system to perform the method of any one of claims 28 to 36.

38. A machine-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 28 to 36.

Images