Generating training data for a machine learning model

Abstract

The present invention relates to a computer-implemented method of generating a training data pair for training a machine learning model, using contrastive learning, to annotate an image obtained from a camera with interest points and descriptors. The computer-implemented method comprises: receiving a first image of a scene and a second image of the same scene; warping the second image to conform dimensionally to the first image; generating a first mask for the first image and a second mask for the warped second image; and creating a training data pair including the first image and the first mask as first data of the training data pair and including the warped second image and the second mask as second data of the training data pair

Description

[0001] GENERATING TRAINING DATA FOR A MACHINE LEARNING MODEL

[0002] FIELD

[0003]

[0001] The subject-matter of the present disclosure relates to training a feature detection network, including generating training data for a feature detection network.

[0004] BACKGROUND

[0005]

[0002] When an autonomous vehicle traverses a route, it needs to localise itself. An important step in visual localisation is keypoint detection and matching.

[0006]

[0003] Figure 1A shows a map image and Figure 1 B shows a map image. Dots in the respective images relate to keypoints. The lines between respective dots illustrate keypoint matches where the matches keypoints have corresponding descriptors. A keypoint is obtained as an output from a decoder in the form of the neural network. A descriptor is obtained as an output from a different decoder, which may also be in the form of a neural network.

[0007]

[0004] Here, keypoints, i.e., visual features, are identified in each image, and then matched between the two images. Correctly matched keypoints can be used to triangulate the position of a camera used to capture the image in Figure 1A, relative to the position of the camera used to capture the image in Figure 1 B.

[0008]

[0005] There are a number of machine learning techniques that train neural networkbased feature detectors, but these techniques are not geared towards producing illumination-invariant features.

[0009]

[0006] With reference to Figures 2A to 2C, current machine learning models often fail to produce features that can be matched between day and night images. For instance, Figure 2A shows an image of a scene during daylight, Figure 2B shows an image of a scene during daylight with regions that change visually in certain lighting conditions, and Figure 2C shows an image of a scene during night time with those regions having changed due to different lighting conditions.

[0010]

[0007] It is an aim of the present invention to address such problems and improve on the prior art.

[0011] SUMMARY

[0012]

[0008] The subject-matter of the present disclosure aims to provide a new training technique that substantially improves the learned feature network’s ability to produce features that are matchable despite severe illumination changes existing in the images. Therefore, if the map has images that were captured during bright daylight, the network will perform well even if the autonomous vehicle is used during substantially different illumination conditions, e.g. during nighttime.

[0013]

[0009] According to an aspect of the present disclosure, there is provided a computer- implemented method of generating a training data pair for training a machine learning model, using contrastive learning, to annotate an image obtained from a camera with interest points and descriptors, the computer-implemented method comprising: receiving a first image of a scene and a second image of the same scene; warping the second image to conform dimensionally to the first image; generating a first mask for the first image and a second mask for the warped second image; and creating a training data pair including the first image and the first mask as first data of the training data pair and including the warped second image and the second mask as second data of the training data pair.

[0014]

[0010] Generating a training data pair including the corresponding first and second masks enables images taken at different times along the route to be used to train the machine learning model. Therefore, images taken under different conditions can be used and the model will be tolerant of different conditions during inference. In this way, the training data pair enables an AV to localise itself more accurately.

[0015]

[0011] The term pair may be used herein to mean two training data examples of the same scene that can be used to train the machine learning model using contrastive loss.

[0016]

[0012] The term same scene may be used herein to mean that a pose of the cameras used to capture each of the first image and the second image is the same, or similar. In other words, the pose of the cameras is the same to within an acceptable degree of tolerance. Therefore, the scene in each of the first and second images will be similar enough that a majority of keypoints from one image can be identified in the other image.

[0017]

[0013] In an embodiment, warping the second image to conform dimensionally to the first image comprises: inputting the first image to a trained feature point detector model to identify a plurality of feature points in the first image; and inputting the second image to the trained feature point detector model to identify a plurality of feature points in the second image; and matching corresponding feature points between the first and second images.

[0018]

[0014] In an embodiment, warping the second image to conform dimensionally to the first image comprises: warping the second image so that the feature points in the second image that match feature points of the first image positionally coincide. Positionally coinciding may mean that they occupy corresponding pixels.

[0019]

[0015] In an embodiment, generating the first mask for the first image and the second mask for the warped second image comprises: generating a first semantic segmentation map for the first image using a trained semantic classifier; generating a second semantic segmentation map for the warped second image using a trained semantic classifier; and identifying one or more regions in each of the first and second semantic segmentation maps that correspond to a transient feature in the scene.

[0020]

[0016] In an embodiment, generating the first mask for the first image and the second mask for the warped second image comprises: generating the first mask by identifying the or each region in the first semantic segmentation map as a transient region, and by identifying a remainder of the first semantic segmentation map as a permanent region; and generating the second mask by identifying the or each region in the second semantic segmentation map as a transient region, and by identifying a remainder of the second semantic segmentation map as a permanent region.

[0021]

[0017] In an embodiment, the transient feature includes one or more of: sky, a vehicle, an animal, and a pedestrian.

[0022]

[0018] In an embodiment, the interest points include features of a map used by an autonomous vehicle when traversing a route including the scene.

[0023]

[0019] In an embodiment, the features of the map include a feature of at least one of: a road marking, a building, a road sign, an actor in the scene.

[0024]

[0020] In an embodiment, the machine learning model includes a shared encoder for reducing a dimensionality of an input image, an interest point decoder for generating interest points from the reduced dimensionality input image, and a descriptor decoder for generating descriptors for the image from the reduced dimensionality input image.

[0025]

[0021] In an embodiment, receiving a first image of a scene and a second image of the same scene comprises: capturing, by a camera of an autonomous vehicle traversing a route, the first image of the scene at a point along the route; and capturing, by the camera of the autonomous vehicle traversing the route, the second image of the scene at the point along the route.

[0026]

[0022] In an embodiment, the first image and the second image have a different condition.

[0023] In an embodiment, the different condition is an amount of light.

[0027]

[0024] According to an aspect of the present disclosure, there is provided a transitory, or non-transitory, computer-readable medium, having instructions stored thereon that when executed by at least one processor, cause the at least one processor to perform the computer-implemented method of any preceding aspect or embodiment.

[0028]

[0025] BRIEF DESCRIPTION OF DRAWINGS

[0029]

[0026] The subject-matter of the present disclosure is best described with reference to the accompanying figures, in which:

[0030]

[0027] Figures 1A and 1B show a live image of a scene during daylight and a map image of the same scene during night time, respectively, both being annotated with interest points for localising an autonomous vehicle traversing a route;

[0031]

[0028] Figures 2A to 2C respectively show images of a scene during different conditions, namely Figure 2A being during daylight, Figure 2B being during daylight with certain regions highlighted due to the lighting conditions, and Figure 2C being during nighttime with those same regions highlighted due to the lighting conditions;

[0032]

[0029] Figure 3 shows an autonomous vehicle, according to at least one embodiment;

[0033]

[0030] Figure 4 shows a flow chart summarising a method of localising the autonomous vehicle of Figure 3, according to at least one embodiment;

[0034]

[0031] Figure 5 shows a block diagram illustrating a machine learning model used for annotating an input image with interest points and descriptors for localising the autonomous vehicle, according to at least one embodiment;

[0035]

[0032] Figures 6A and 6B show a first image of a scene during daylight and a second image of the scene during nighttime, respectively, for training the machine learning model of Figure 5 using contrastive loss, according to at least one embodiment;

[0036]

[0033] Figures 7A and 7B show a warped second image and the warped second image overlayed onto the first image, respectively;

[0037]

[0034] Figures 8A and 8B show a semantic map for the first image and a semantic map for the warped second image, respectively;

[0038]

[0035] Figures 9A and 9B show a first mask for the first image and a second mask for the second image, respectively;

[0036] Figures 10A and 10B show an annotated image obtained using the machine learning model of Figure 5, and an annotated second image obtained using the machine learning of Figure 5, respectively, for training the machine learning model using contrastive loss according to at least one embodiment;

[0039]

[0037] Figure 11 shows a flow chart summarising a computer-implemented method of training the machine learning model to annotate an image of a scene obtained from a camera with interest points and descriptors, according to at least one embodiment;

[0040]

[0038] Figure 12 shows a flow chart summarising a computer-implemented method of generating training data for training the machine learning model, according to at least one embodiment; and

[0041]

[0039] Figure 13 shows a flow chart summarising a computer-implemented method of localising an autonomous vehicle, according to at least one embodiment.

[0042] DESCRIPTION OF EMBODIMENTS

[0043]

[0040] Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination. In particular, the features of any one example embodiment may be combined with features of any other embodiment, as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the term “comprising” or “comprises” means including the component(s) specified but not to the exclusion of the presence of others.

[0044]

[0041] The embodiments described herein may be embodied as sets of instructions stored as electronic data in one or more storage media. Specifically, the instructions may be provided on a transitory or non-transitory computer-readable media. When executed by the processor, the processor is configured to perform the various methods described in the following embodiments. In this way, the methods may be computer-implemented methods. In particular, the processor and a storage including the instructions may be incorporated into a vehicle. The vehicle may be an autonomous vehicle (AV).

[0045]

[0042] Whilst the following embodiments provide specific illustrative examples, those illustrative examples should not be taken as limiting, and the scope of protection is defined by the claims. Features from specific embodiments may be used in combination with features from other embodiments without extending the subject-matter beyond the content of the present disclosure.

[0046]

[0043] With reference to Figure 3, an AV 10 may include a plurality of sensors 12. The sensors 12 may be mounted on a roof of the AV 10, or integrated into the bumpers, grill, bodywork, etc. The sensors 12 may be communicatively connected to a computer 14. The computer 14 may be onboard the AV 10. The computer 14 may include a processor 16 and storage 18. The memory may include the non-transitory computer-readable media described above. Alternatively, the non-transitory computer-readable media may be located remotely and may be communicatively linked to the computer 14 via the cloud 20. The computer 14 may be communicatively linked to one or more actuators 22 for control thereof to move the AV 10. The actuators may include, for example, a motor, a braking system, a power steering system, etc.

[0047]

[0044] The computer 14 includes an autonomy stack for controlling the AV 10 stored the storage 18. The autonomy stack may control the AV 10 in response to the sensor data. To achieve this, the autonomy stack may include one or more machine learning models. The one or more machine learning models may include an end-to-end model that is trained to provide control commands to actuators of the AV 10 in response to the sensor data. The one or more machine learning models may include machine learning models respectively responsible for perception, planning, and control. This may be in addition to the end-to-end model or as an alternative to the end-to-end model. Perception functions may include object detection and classification based on sensor data. Planning functions may include object tracking and trajectory generation. Control functions including setting control instructions for one or more actuators 22 of the AV 10 to move the AV 10 according to the trajectory.

[0048]

[0045] The sensors 12 may include various sensor types. Examples of sensor types include LiDAR sensors, RADAR sensors, and cameras. Each sensor type may be referred to as a sensor modality. Each sensor type may record data associated with the sensor modality. For example, the LiDAR sensor may record LiDAR modality data.

[0049]

[0046] The data may capture various scenes that the AV 10 encounters. For example, a scene may be a visible scene around the AV 10 and may include roads, buildings, weather, objects (e.g. other vehicles, pedestrians, animals, etc.), etc.

[0050]

[0047] With reference to Figure 4, steps performed by the autonomy stack may be summarised in a flow chart representing a method of localising an autonomous vehicle. For example, the method comprises providing a machine learning model 26, capturing an image 28 using a camera 12, inputting the captured image to the machine learning model 26 to annotate the captured image with interest points and descriptors, retrieving map images 30, and inputting the map images to the machine learning model 26 to annotate the map image with interest points and descriptors, including landmarks 32. The method also comprises comparing the interest points and descriptors from the annotated captured image to the interest points and descriptors of the map image. The method also comprises localising the AV 10 using a localisation module 34. The localisation may be a pose of the AV 10 on the map.

[0051]

[0048] Interest points include features of in scenes forming a map used by the AV 10 when traversing a route including the scene. For example, the interest points include a feature of a road marking, a building, a road sign, an actor in the scene, etc. More specifically, the interest points may not be an entire road marking, but may be a feature of a road marking, e.g. an apex of an arrow painted on a road, a corner of an opposite end of the arrow, etc.

[0052]

[0049] With reference to Figure 5, the machine learning model 26 is a fully-convolutional neural network architecture called which operates on a full-sized image and produces interest point detections accompanied by fixed length descriptors in a single forward pass. The model has a single, shared encoder 35 to process and reduce the input image 37 dimensionality. After the encoder 35, the architecture splits into two decoder “heads” or decoders 40_1 , 40_2), which learn task specific weights - one for interest point detection (interest point decoder 40_1 for generating interest points from the reduced dimensionality input image) and the other for interest point description (descriptor decoder 40_2 for generating descriptors for the image from the reduced dimensionality input image). Most of the network’s parameters are shared between the two tasks.

[0053]

[0050] It will be appreciated that the interest point decoder 40_1 is a neural network and the descriptor decoder 40_2 is a different neural network.

[0054]

[0051] The machine learning model 26 may use a VGG-style encoder 35 to reduce the dimensionality of the input image 37. The encoder 35 consists of convolutional layers, spatial downsampling via pooling and non-linear activation functions. The encoder 35 uses a plurality of max-pooling layers (e.g. three). Therefore, it is possible to define Hc= H / 8 and Wc= W / 8 for an image sized H x W . The pixels may be referred to in the lower dimensional output as “cells,” where three 2 x 2 non-overlapping max pooling operations in the encoder result in 8 x 8 pixel cells. The encoder 35 maps the input image I e RHXW to an intermediate tensor B E RH^WCXFsma||erspatial dimension and greater channel depth (, Hc< H, Wc< W and F > 1).

[0055]

[0052] For interest point detection, each pixel of the output corresponds to a probability of “point-ness” for that pixel in the input.

[0056]

[0053] The interest point detector head computes X E HCXWCX6Sanc|outpUtSa tensor sized RHXW. The 65 channels correspond to local, non-overlapping 8 x 8 grid regions of pixels plus an extra “no interest point” dustbin. After a channel-wise softmax, the dustbin dimension is removed and a RHCXWCX64=> RH*Wreshape is performed.

[0057]

[0054] The descriptor head computes D E RIICWC*Danc|0 utpUtsa tensor sized RI!XWXD. To output a dense map of L2-normalized fixed length descriptors, the machine learning model first outputs a semi-dense grid of descriptors (one every 8 pixels). Learning descriptors semi-densely rather than densely reduces training memory and keeps the run-time tractable. The decoder then performs bi-cubic interpolation of the descriptor and then L2-normalizes the activations to be unit length. This fixed, non-learned descriptor decoder is shown in Figure 3.

[0058]

[0055] Both decoders 40 1 , 40_2, operate on a shared and spatially reduced representation of the input image. To keep the model fast and easy to train, both decoders use non-learned upsampling to bring the representation back to RHxw.

[0059]

[0056] The final loss is the sum of two intermediate losses: one for the interest point decoder 40 1 , Lp, and one for the descriptor decoder 40_2, Ld. Optimising a parameterisation of the machine learning model to minimise the contrastive loss comprises optimising weights of the descriptor decoder 40_2 to minimise the contrastive loss.

[0060]

[0057] Pairs of warped images (e.g. as shown in Figures 1A and 1 B) can be used for training which have pseudo-ground truth interest point locations. The two losses can then be optimised simultaneously, given a pair of images, as shown in Figures 1A and 1 B. 2 can be used to balance the final loss:

[0059] The interest point detector loss function Lpis a fully-convolutional cross-entropy loss over the cells xhwe X. The set of corresponding ground-truth interest point labels are called Y and individual entries are called yhw. The loss is:

[0061]

[0064] The descriptor loss is applied to all pairs of descriptor cells, dhwE D from the first image and d'hlw, E D' from the second (aligned) image. With the homography-aligned image pairs, the correspondence between the (h, iv) cell and the (h'.w") cell can be written as follows:

[0062] Shwh' ’ = {1 , if h=h’ and w=w’ and {phw, Ph } are permanent features (non-transient)}

[0063]

[0065] where phwdenotes the location of the center pixel in the ( , iv) cell in the first image, and ph,w, denotes the location of the center pixel in the (h’, w’) in the second (aligned) image. The entire set of correspondences for a pair of images can be denoted with S.

[0064]

[0066] We also add a weighting termdto help balance the fact that there are more negative correspondences than positive ones. We use a hinge loss with positive margin mpand negative margin mn. The descriptor loss is defined as:

[0065]

[0071] With reference to Figures 6A and 6B, a first image 36A and a second image 36B may be received which show the same scene. For instance, they both show a scene captured from a same point along the route. In other words, a capture capturing the first image 36A has a first pose. A camera capturing a second image 36B has a second pose. The first and the second pose may be the same or to within an acceptable degree of tolerance so that a majority of features shown in the first image 36A are present in the second image 36B. The tolerance for a difference in camera positions is to within 10 cm. The tolerance for a difference in camera orientations / rotations is 2 degrees.

[0066]

[0072] In other words, the method comprises capturing, by a camera 12 of the AV 10 traversing a route, the first image 36A of the scene at a point along the route. The method also comprises capturing, by the camera 12 of the AV 10 traversing a route, the second image 36B of the scene at the point along the route. In other words, the point the camera is in when capturing both images 36A and 36B is the same, or to within an acceptable degree of tolerance as explained above.

[0067]

[0073] The first image 36A may be collected at a different time to the second image 36B. For example, the first image 36A and the second image 36B may have been collected at different times of day and / or year. The first and second images 36A, 36B, may approximately correspond to the same viewpoint as described above. However, the first and second images 36A, 36B, show the scene under different conditions. The different conditions may be different lighting conditions, or different illumination conditions. For example, the first image 36A may have been captured during daylight, whilst the second image 36B may have been captured during nighttime.

[0068]

[0074] The first and second images 36A, 36B, when captured using the tolerances outlined above, will be aligned to within about 30 pixels. However, further alignment is required to produce the dense (1 :1 ) correspondences that the contrastive loss requires. The first image 36A and second image 36B are thus coarsely aligned day and night images.

[0069]

[0075] With reference to Figures 7A and 7B, the method then comprises warping the second image to conform dimensionally to the first image.

[0070]

[0076] This may comprise inputting the first image to a trained feature point detector model to identify a plurality of feature points in the first image. Similarly, the second image may be input to a trained feature point detector model to identify a plurality of feature points in the second image. The trained feature point detector model used for each of the first and second images may be the same. The model may be the same or a pretrained version of the machine learning model 26 being trained herein, but that has been trained on illumination conditions specific to the image being processed.

[0077] The method of warping may comprise matching corresponding feature points between the first image and the second image. Matching may mean that descriptors are generated for each of the detected feature points. The descriptors from one image may be compared with descriptors from another image. Descriptors having a minimum distance, or shortest distance, may be considered to be matched feature points. The distance may be a Euclidean distance, for example, although other metrics may also be used in other embodiments.

[0071]

[0078] Once the feature points have been matched, the second image may be warped to conform dimensionally to the first image. This may mean that the second image is warped so that the feature points in the second image that match feature points of the first image positionally coincide. For example, the pixel for any matching feature points may be the same between images.

[0072]

[0079] In other words, one image is aligned onto another using a homography warping technique. This enables us to sample positive correspondences by selecting the same pixel coordinate from each image. The homography warping requires a set of initial feature correspondences, which can be acquired by using a heavyweight feature matcher (as described above). The warped second image 38B (night image) produced from homography alignment is shown in Figure 7A. The two images (the first, daytime, image 36A and the warped second, nighttime, image 38B, are overlaid in Figure 7B. As a result, their alignment can be observed.

[0073]

[0080] In the overlaid image in Figure 7B, it can be seen that the road, buildings, signs, and trees, are well aligned. Therefore, positive correspondences can be easily sampled. However, positive correspondences should not be sampled from regions where vehicles / pedestrians exist in one image, but not in the other image, since these descriptors should not be similar to one another. Additionally, since the sky is substantially different between night and day images, we do not want to sample positive correspondences from the sky region. Therefore, such positive correspondences are ignored. The method of ignoring those positive correspondences is described below and uses semantic segmentation, and mask generation.

[0074]

[0081] With reference to Figures 8A and 8B, the method may comprise generating a first semantic segmentation map 40A and a second semantic segmentation map 40B corresponding to the first image 36A and the second image 36B, respectively. The semantic map may be constructed using any method commonly used in the field. For example, a trained semantic classifier may be used to label each pixel into a semantic class. The semantic classes for features in the scene may include a road 42, other actors 44, buildings 46, lane boundaries, road boundaries 48, sidewalk 50, vegetation 52, road signs 54, sky 56, weather, etc. Those semantic classes may be grouped as either a permanent class and a transient class. The permanent class may correspond to permanent features. The transient class may correspond to transient features. For example, transient features such as other actors, e.g. a vehicle, an animal, a pedestrian, and also transient features such as the sky, some vegetation, etc., may be grouped into a transient group. Permanent features such as a road 42, buildings 46, road boundaries 48, a sidewalk, road signs, and some vegetation 52 (e.g. evergreen trees), may be grouped into a permanent group. An area covered by pixels in the transient group may be called a transient region. An area covered by pixels in the non-transient group may be called a non-transient region. In this way, the method may comprise identifying one or more regions in each of the first and second semantic segmentation maps 40A, 40B, that correspond to a transient feature in the scene.

[0075]

[0082] With reference to Figures 9A and 9B, the method comprises generating a first mask 58A (Figure 9A) for the first image and a second mask 58B (Figure 9B) for the warped second image. This may involve generating the first mask 58A by identifying the or each region in the first semantic segmentation map 40A as the transient region 60, and by identifying a remainder of the first semantic segmentation map 40A as a permanent region 62. Similarly, this may also involve generating the second mask 58B by identifying the or each region in the second semantic segmentation map 40B as the transient region 60, and by identifying a remainder of the second semantic segmentation map 40B as the permanent region 62. For example, in the first mask 58A, the transient region 60 includes pixels showing sky and other actors, e.g. cars.

[0076]

[0083] The first image 36A and the first mask 58A may be considered to be first data for the training data. The warped second image 38B and the second mask 58B may be considered second data of the training data. Together, the first data and the second data may be considered a training data pair. Therefore, the method comprises creating a training data pair including the first image and the first mask as first data of the training data pair and including the warped second image and the second mask as second data of the training data pair.

[0077]

[0084] There is also provided a computer-implemented method of training the machine learning model 26 to annotate an image of a scene obtained from a camera 12 with interest points and descriptors. The method comprises receiving the training data pair including the first image and the first mask as first data and including the second image and the second mask as second data.

[0078]

[0085] The method also comprises inputting the first image to the machine learning model 26 to annotate the first image with interest points and generate descriptors for the first image. Similarly, the method comprises inputting the second image to the machine learning model 26 to annotate the second image with interest points and generate descriptors for the second image. More specifically, a reduced dimension version of the first image 36A is obtained using the encoder 35. The interest point decoder 40_1 reproduces the image annotated with interest points. The descriptor decoder 40_2 generates descriptors. The descriptors can be matched to the interest points by pixel correspondence between the reproduced images.

[0079]

[0086] The second image described in the preceding paragraph may be considered the warped second image 38B. Similarly, a reduced dimension version of the warped second image 38B may be obtained using the encoder 35. The interest points decoder 40_1 reproduces the image annotated with interest points. The descriptor decoder 40_2 generates descriptors. Again, the descriptors can be matched with the interest points by pixel correspondence between the reproduced images.

[0080]

[0087] In Figure 10A, the first annotated image 65A annotated with interest points 66 is shown. In Figure 10B, the second annotated image 65B annotated with interest points 66 is shown.

[0081]

[0088] The method comprises reducing the number of interest points 66 and descriptors of the first image using the first mask 58A. Similarly, the method comprises reducing the number of interest points 66 and descriptors of the second image using the second mask 58B.

[0082]

[0089] More specifically, the method may comprise overlaying the second mask 58B on the second annotated image 65B. Interest points and descriptors that overlap with the permanent region are retained. Interest points 66 and descriptors that overlap with the transient region are discarded.

[0083]

[0090] Similarly, the method comprises overlaying the first mask 58A on the first annotated image 65A. Interest points 66 and descriptors that overlap with the permanent region are retained. Interest points 66 and descriptors that overlap with the transient region are discarded.

[0091] With reference to Figures 10A and 10B, interest point 66_1 is discarded, and the other interest points are retained. Therefore, arrow 68_1 is an invalid correspondence because the interest point 66_1 in Figure 10A overlaps with a vehicle, which is a transient feature and thus forms part of the transient region of the first mask. The retained interest point 66 in the first annotated image 65A may have four valid interest point correspondences, shown illustratively by arrows 68_2. During live localisation on the vehicle, the descriptors are used to described match retained the interest points 66 between the first and second annotated images 65A, 65B. More specifically, a degree of similarity is used and the descriptors with the smallest similarity correspond to a match. In other words, a distance between descriptors is obtained, and the descriptor pair with the smallest distance is taken to be the matched descriptors. The distance may be obtained using Euclidian distance, for example.

[0084]

[0092] During training, the match is known (the points connected by 68_2). Therefore, the match is used as the positive sample in the contrastive loss (see below). The goal is to minimise the descriptor distance at these two locations. The other three lines are examples of “negative” samples. The goal is the maximise the descriptor distance at these two locations.

[0085]

[0093] The method also comprises calculating a contrastive loss between the reduced number of interest points and descriptors of the respective first and second images. More specifically, it is the matched interest points that is used. The contrastive loss is obtained as described hereinbefore. Finally, the method comprises optimising a parameterisation of the machine learning model to minimise the contrastive loss, as described hereinbefore.

[0086]

[0094] With reference to Figure 11 , a computer-implemented method of training a machine learning model to annotate an image of a scene from a camera with interest points and descriptors may be summarised as comprising: receiving S100 a training data pair, the training data pair including a first image and a first mask as first data and including a second image and a second mask as second data; inputting S102 the first image to the machine learning model to annotate the first image with interest points and generate descriptors for the first image; inputting S104 the second image to the machine learning model to annotate the second image with interest points and generate descriptors for the second image; reducing S106 a number of interest points and descriptors of the first image using the first mask; reducing S108 a number of interest points and descriptors of the second image using the second mask; calculating S110 a contrastive loss between the reduced number of interest points and descriptors of the respective first and second images; and optimising S112 a parameterisation of the machine learning model to minimise the contrastive loss.

[0087]

[0095] With reference to Figure 12, a computer-implemented method of generating training data for training the machine learning model may be summarised as including: receiving S200 the first image of the scene and a second image of the same scene; warping S202 the second image to conform dimensionally to the first image; generate S204 a first mask for the first image and a second mask for the warped second image; and creating S206 the training data pair including the first image and the first mask as first data of the training data pair and including the warped second image and the second mask as second data of the training data pair.

[0088]

[0096] With reference to Figure 13, a computer-implemented method of localising an autonomous vehicle may be summarised as comprising: providing S300 a machine learning model trained as described above; capturing S302 an image using a camera of the autonomous vehicle; inputting S304 the captured image to the machine learning model to annotate the captured image with interest points and descriptors for the captured image; comparing S306 the interest points and descriptors from the annotated image to interest points and descriptors of a map used by the autonomous vehicle when traversing a route associated with the map; and localising S308 the autonomous vehicle based on the comparison.

[0089]

[0097] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0090]

[0098] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope. One of more clauses follow which may help understand the present disclosure.

[0091] Clause 1 . A computer-implemented method of training a machine learning model to annotate an image of a scene obtained from a camera with interest points and descriptors, the computer-implemented method comprising: receiving a training data pair, the training data pair including a first image and a first mask as first data and including a second image and a second mask as second data; inputting the first image to the machine learning model to annotate the first image with interest points and generate descriptors for the first image; inputting the second image to the machine learning model to annotate the second image with interest points and generate descriptors for the second image; reducing a number of interest points and descriptors of the first image using the first mask; reducing a number of interest points and descriptors of the second image using the second mask; calculating a contrastive loss between the reduced number of interest points and descriptors of the respective first and second images; and optimising a parameterisation of the machine learning model to minimise the contrastive loss.

[0092] Clause 2. The computer-implemented method of Clause 1 , wherein the first mask and the second mask each include one or more transient regions and a permanent region, the or each transient region corresponding positionally to a transient feature in the scene, wherein a remainder of the image forming the permanent region.

[0093] Clause 3. The computer-implemented method of Clause 2, wherein the transient feature includes one or more of: sky, a vehicle, an animal, and a pedestrian.

[0094] Clause 4. The computer-implemented method of Clause 2 or Clause 3, wherein reducing a number of interest points and descriptors of the first image using the first mask comprises: overlaying the first mask on the first image; retaining any interest points and descriptors that overlap with the permanent region; and discarding any interest points and descriptors that overlap with the transient region.

[0095] Clause 5. The computer-implemented method of any of Clauses 2 to 4, wherein reducing a number of interest points and descriptors of the second image using the second mask comprises: overlaying the second mask on the second image; retaining any interest points and descriptors that overlap with the permanent region; and discarding any interest points and descriptors that overlap with the transient region.

[0096] Clause 6. The computer-implemented method of any preceding clause, wherein the second image has been warped to conform dimensionally to the first image.

[0097] Clause 7. The computer-implemented method of any preceding clause, wherein the interest points include features of a map used by an autonomous vehicle when traversing a route including the scene, the autonomous vehicle including the camera.

[0098] Clause 8. The computer-implemented method of Clause 7, wherein the features of the map include a feature of at least one of: a road marking, a building, and a road sign

[0099] Clause 9. The computer-implemented method of Clause 7 or Clause 8, wherein the first image of the scene corresponds to a point on the route, and wherein the second image of the scene corresponds to the point along the route.

[0100] Clause 10. The computer-implemented method of any preceding clause, wherein the first image and the second image have a different condition.

[0101] Clause 11. The computer-implemented method of Clause 10, wherein the different condition is an amount of light.

[0102] Clause 12. The computer-implemented method of any preceding clause, wherein the machine learning model includes an encoder for reducing a dimensionality of an input image, an interest point decoder for generating interest points from the reduced dimensionality input image, and a descriptor decoder for generating descriptors for the image from the reduced dimensionality input image.

[0103] Clause 13. The computer-implemented method of Clause 12, wherein the interest point decoder is a neural network and the descriptor decoder is a different neural network, wherein optimising a parameterisation of the machine learning model to minimise the contrastive loss comprises: optimising weights of the descriptor decoder to minimise the contrastive loss.

[0104] Clause 14. The computer-implemented method of any preceding clause, further comprising generating the training data pair by: receiving the first image of the scene and a second image of the same scene; warping the second image to conform dimensionally to the first image; generating a first mask for the first image and a second mask for the warped second image; and creating the training data pair including the first image and the first mask as first data of the training data pair and including the warped second image and the second mask as second data of the training data pair.

[0105] Clause 15. The computer-implemented method of Clause 14, wherein warping the second image to conform dimensionally to the first image comprises: inputting the first image to a trained feature point detector model to identify a plurality of feature points in the first image; and inputting the second image to the trained feature point detector model to identify a plurality of feature points in the second image; and matching corresponding feature points between the first and second images.

[0106] Clause 16. The computer-implemented method of Clause 15, wherein warping the second image to conform dimensionally to the first image comprises: warping the second image so that the feature points in the second image that match feature points of the first image positionally coincide.

[0107] Clause 17. The computer-implemented method of any of Clauses 14 to 16, wherein generating the first mask for the first image and the second mask for the warped second image comprises: generating a first semantic segmentation map for the first image using a trained semantic classifier; generating a second semantic segmentation map for the warped second image using a trained semantic classifier; and identifying one or more regions in each of the first and second semantic segmentation maps that correspond to a transient feature in the scene.

[0108] Clause 18. The computer-implemented method of Clause 17, wherein generating the first mask for the first image and the second mask for the warped second image comprises: generating the first mask by identifying the or each region in the first semantic segmentation map as the transient region, and by identifying a remainder of the first semantic segmentation map as the permanent region; and generating the second mask by identifying the or each region in the second semantic segmentation map as the transient region, and by identifying a remainder of the second semantic segmentation map as the permanent region.

[0109] Clause 19. The computer-implemented method of Clause 17 or Clause 18, wherein the transient feature includes one or more of: sky, a vehicle, an animal, and a pedestrian.

[0110] Clause 20. The computer-implemented method of any of Clauses 14 to 19, wherein the interest points include features of a map used by an autonomous vehicle when traversing a route including the scene.

[0111] Clause 21 . The computer-implemented method of Clause 20, wherein the features of the map include a feature of at least one of: a road marking, a building, and a road sign.

[0112] Clause 22. The computer-implemented method of any of Clauses 14 to 21 , wherein receiving a first image of a scene and a second image of the same scene comprises: capturing, by a camera of an autonomous vehicle traversing a route, the first image of the scene at a point along the route; and capturing, by the camera of the autonomous vehicle traversing the route, the second image of the scene at the point along the route. Clause 23. A computer-implemented method of localising an autonomous vehicle, the computer-implemented method comprising: providing a machine learning model trained the computer-implemented method of any preceding clause; capturing an image using a camera of the autonomous vehicle; inputting the captured image to the machine learning model to annotate the captured image with interest points and descriptors for the captured image; comparing the interest points and descriptors from the annotated image to interest points and descriptors of a map used by the autonomous vehicle when traversing a route associated with the map; and localising the autonomous vehicle based on the comparison.

[0113] Clause 24. A transitory, or non-transitory, computer-readable medium, having instructions stored thereon that when executed by at least one processor, cause the at least one processor to perform the computer-implemented method of any preceding clause.

Claims

CLAIMS1 . A computer-implemented method of generating a training data pair for training a machine learning model, using contrastive learning, to annotate an image obtained from a camera with interest points and descriptors, the computer- implemented method comprising: receiving a first image of a scene and a second image of the same scene; warping the second image to conform dimensionally to the first image; generating a first mask for the first image and a second mask for the warped second image; and creating a training data pair including the first image and the first mask as first data of the training data pair and including the warped second image and the second mask as second data of the training data pair.

2. The computer-implemented method of Claim 1 , wherein warping the second image to conform dimensionally to the first image comprises: inputting the first image to a trained feature point detector model to identify a plurality of feature points in the first image; and inputting the second image to the trained feature point detector model to identify a plurality of feature points in the second image; and matching corresponding feature points between the first and second images.

3. The computer-implemented method of Claim 2, wherein warping the second image to conform dimensionally to the first image comprises: warping the second image so that the feature points in the second image that match feature points of the first image positionally coincide.

4. The computer-implemented method of any preceding claim, wherein generating the first mask for the first image and the second mask for the warped second image comprises: generating a first semantic segmentation map for the first image using a trained semantic classifier; generating a second semantic segmentation map for the warped second image using a trained semantic classifier; andidentifying one or more regions in each of the first and second semantic segmentation maps that correspond to a transient feature in the scene.

5. The computer-implemented method of Claim 4, wherein generating the first mask for the first image and the second mask for the warped second image comprises: generating the first mask by identifying the or each region in the first semantic segmentation map as a transient region, and by identifying a remainder of the first semantic segmentation map as a permanent region; and generating the second mask by identifying the or each region in the second semantic segmentation map as a transient region, and by identifying a remainder of the second semantic segmentation map as a permanent region.

6. The computer-implemented method of Claim 4 or Claim 5, wherein the transient feature includes one or more of: sky, a vehicle, an animal, and a pedestrian.

7. The computer-implemented method of any preceding claim, wherein the interest points include features of a map used by an autonomous vehicle when traversing a route including the scene.

8. The computer-implemented method of Claim 7, wherein the features of the map include a feature of at least one of: a road marking, a building, a road sign, an actor in the scene.

9. The computer-implemented method of any preceding claim, wherein the machine learning model includes a shared encoder for reducing a dimensionality of an input image, an interest point decoder for generating interest points from the reduced dimensionality input image, and a descriptor decoder for generating descriptors for the image from the reduced dimensionality input image.

10. The computer-implemented method of any preceding claim, wherein receiving a first image of a scene and a second image of the same scene comprises: capturing, by a camera of an autonomous vehicle traversing a route, the first image of the scene at a point along the route; andcapturing, by the camera of the autonomous vehicle traversing the route, the second image of the scene at the point along the route.11 . The computer-implemented method of any preceding claim, wherein the first image and the second image have a different condition.

12. The computer-implemented method of Claim 12, wherein the different condition is an amount of light.

13. A transitory, or non-transitory, computer-readable medium, having instructions stored thereon that when executed by at least one processor, cause the at least one processor to perform the computer-implemented method of any preceding claim.

Images