System and method for processing image data

Abstract

Certain examples described herein relate to a system and method for processing image data. In such examples, the system includes an input interface for receiving the image data, the image data representing at least one view of a scene. The system also includes an initialization engine to generate a first latent representation associated with a first segmentation of at least a first view of the scene, wherein the first segmentation is a semantic segmentation. The initialization engine is further arranged to generate a second latent representation associated with at least a second view of the scene. The system also includes an optimization engine for jointly optimizing the first latent representation and the second latent representation in a latent space to obtain an optimized first latent representation and an optimized second latent representation.

Description

Technical Field

[0001] The present invention relates to methods and systems for obtaining a representation of a scene using image processing. The invention is particularly, but not exclusively, related to obtaining a latent representation of a scene that can be used, for example, by a robotic device to navigate and / or interact with its environment. Background Art

[0002] In the fields of computer vision and robotics, there is often a need to construct a representation of an environment, such as a three-dimensional space that can be navigated using a robotic device. Constructing a representation of a three-dimensional space allows a real-world environment to be mapped into a virtual or digital realm, where the map of the environment can be used and manipulated by an electronic device. For example, a mobile robotic device may require a representation of a three-dimensional space to allow simultaneous localization and mapping (commonly referred to as "SLAM") to navigate its environment. The robotic device may operate in an indoor home or commercial environment or in an outdoor natural environment. The representation of the environment can enable models of objects within that space to be identified and / or extracted. These can be used to perform measurements on the real-world environment and / or to produce three-dimensional replicas, for example via additive manufacturing systems. Similarly, detecting human body parts in three-dimensional space can enable new human-computer interactions, making it possible to manipulate virtual representations of objects using actions in the physical world.

[0003] There are a variety of techniques that can be used to construct a representation of an environment. For example, structure from motion and multi-view stereo are two techniques that can be used to achieve this. Many techniques extract features from images of the environment, which are then correlated from image to image to construct a three-dimensional representation. Some techniques that use a reduced number of points or features to generate a representation are called "sparse" techniques. For example, these techniques may use ten to one hundred features and / or points to generate a representation. This can be contrasted with "dense" techniques that utilize thousands or millions of points to generate a representation. The advantage of "sparse" techniques is that they are easier to implement in real time, for example at a frame rate of around 30 frames per second; the use of a limited number of points or features limits the scope of processing required to construct the three-dimensional representation. In contrast, performing a real-time "dense" mapping of an environment is more difficult due to the computational requirements. For example, it is often preferable to perform "dense" mapping offline; for example, generating a "dense" representation from 30 minutes of provided image data may take 10 hours.

[0004] Once a three-dimensional (3D) representation of space has been generated, there is the further question of the utility of the representation. For example, many robotics applications require not only a definition of the geometry of space, but also useful information about what exists in that space. This is known in the field of computer vision as “semantic” knowledge of space. Understanding what exists in a space is a process that occurs subconsciously in the human brain; therefore, it is easy to underestimate the difficulty of building machines with equivalent capabilities. For example, when a human observes an object such as a cup in 3D space, in addition to the core visual processing network, many different areas of the brain are activated, including areas related to proprioception (such as movement towards the object) and language processing. However, many computer vision systems have a very naive understanding of space, and these systems only “know” the geometry of the space.

[0005] In computer vision and robotics, including rich semantic information within representations of space enables a wider range of capabilities than geometry alone. For example, in the context of home robotics, simple picking tasks require knowing what something is and where it is located. Similarly, the ability to query semantic information within representations is directly useful to humans, for example by providing a database to answer spoken queries about the semantics of previously generated representations: "How many chairs do we have in this conference room? What is the distance between the podium and its nearest chair?"

[0006] McCormac et al., in their paper “SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks,” published in the 2017 proceedings of the IEEE International Conference on Robotics and Automation (ICRA), describe the use of a system that includes a convolutional neural network (CNN) and a SLAM system. The CNN receives a 2D image (e.g., a video frame) and returns a set of per-pixel class probabilities. The SLAM system produces a globally consistent map of surface elements (which can be called “surfaces” and which allow the geometry of a space to be modeled using surfaces defined in a 3D coordinate system). Furthermore, the SLAM system provides long-term, dense correspondences between video frames, even during “multi-turn” motions where parts of the scene are viewed from multiple different positions and / or orientations, rather than simple finite rotations of the camera. These correspondences allow the CNN’s semantic predictions from multiple viewpoints to be probabilistically fused with the surface element map to produce a semantic 3D map. This approach is relatively computationally expensive and can suffer from inaccuracies or inconsistencies in object labeling.

[0007] At the other end of the scale are approaches that explicitly identify object instances and construct a scene model as a 3D object graph. McCormac et al., in their paper “Fusion++: Volumetric Object-Level SLAM,” presented at the 2018 International Conference on 3D Vision, describe an object-level SLAM system that constructs a persistent 3D graph map of arbitrary reconstructed objects. However, this approach can leave large portions of the scene undescribed. Therefore, approaches such as these may not be well suited for navigating or interacting with an environment.

[0008] Given the state of the art, efficient representations of scenes that provide information about what is visible in the scene are still desirable. For example, such representations could enable artificial systems to reason about space and shape in a way that is similar to human intuition. Summary of the Invention

[0009] According to a first aspect of the present invention, a system for processing image data is provided, the system comprising: an input interface for receiving the image data, wherein the image data represents at least one view of a scene; an initialization engine for generating: a first latent representation associated with a first segmentation of at least a first view of the scene, wherein the first segmentation is a semantic segmentation; and a second latent representation associated with at least a second view of the scene; and an optimization engine for jointly optimizing the first latent representation and the second latent representation in a latent space to obtain an optimized first latent representation and an optimized second latent representation.

[0010] In some examples, the system includes a decoder system to at least one of: decode the optimized first latent representation to obtain a decoded first representation of the first view of the scene, wherein the optimized first segmentation of the first view of the scene is derivable from the decoded first representation; and decode the optimized second latent representation to obtain a decoded second representation of the second view of the scene, wherein the optimized second segmentation of the second view of the scene is derivable from the decoded second representation. In these examples, the system may include a feature recognition engine to recognize image features of the image data, wherein the decoder system includes at least one decoder conditioned on the image features. In these examples, the feature recognition engine may be arranged to recognize image features at each of a plurality of different resolutions, and a decoder in the at least one decoder may be arranged to: generate a decoded output at each of the plurality of different resolutions; and, for each of the plurality of different resolutions, combine the image features for a given resolution with the decoded output for the given resolution.

[0011] In some examples where the system includes a decoder system, the decoder system may be arranged to at least one of: normalize the decoded first representation to obtain the optimized first segmentation; and normalize the decoded second representation to obtain the optimized second segmentation.

[0012] In certain examples where the system includes a decoder system, the system may further include: a tracking system for determining a pose of a camera during observation of the scene; and a mapping system arranged to populate a map of the scene using at least one of: the optimized first segmentation of the first view of the scene and first pose data representing a first pose of the camera during capture of the first view of the scene; and the optimized second segmentation of the second view of the scene and second pose data representing the pose of the camera during capture of the second view of the scene.

[0013] In some examples, the image data includes a first frame representing the first view of the scene and a second frame representing the second view of the scene, the first segmentation is a semantic segmentation of the first view of the scene, and the second latent representation is associated with the second segmentation which is a semantic segmentation of the second view of the scene. In these examples, the optimization engine can be arranged to jointly optimize the first latent representation and the second latent representation by: determining a semantic error term indicating a difference between the first latent representation and the second latent representation; and determining a value of the first latent representation and a value of the second latent representation that minimize the semantic error term. In such examples, the system may further include a decoder system arranged to: decode the first latent representation to obtain a decoded first representation; and decode the second latent representation to obtain a decoded second representation, wherein the optimization engine is arranged to determine the semantic error term using the decoded first representation and the decoded second representation. The decoder system can be trained on multiple pairs of input image data and ground truth semantic segmentations. In these examples, the initialization engine can be arranged to generate: a third latent representation associated with a depth map of the first view of the scene; and a fourth latent representation associated with a depth map of the second view of the scene, wherein the optimization engine is arranged to jointly optimize the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation in the latent space to obtain the optimized first latent representation and the second latent representation and the optimized third latent representation and the fourth latent representation. The optimization engine can be arranged to jointly optimize the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation by: determining a semantic error term indicating a difference between the first latent representation and the second latent representation; determining a geometric error term indicating a difference between the third latent representation and the fourth latent representation; and determining values of the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation that jointly minimize the semantic error term and the geometric error term to obtain the optimized first latent representation, the second latent representation, the third latent representation, and the fourth latent representation, respectively.In these examples, the system may include a decoder system comprising: a first decoder arranged to at least one of: decode the optimized first latent representation to obtain an optimized semantic segmentation of the first view of the scene; and decode the optimized second latent representation to obtain an optimized semantic segmentation of the second view of the scene; and a second decoder arranged to at least one of: decode the optimized third latent representation to obtain an optimized depth map for the first view of the scene; and decode the optimized fourth latent representation to obtain an optimized depth map for the second view of the scene. The optimization engine may be arranged to: determine a photometric error term indicating photometric consistency between the first view of the scene and the second view of the scene; and determine a respective value of at least one of the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation that minimizes the photometric error term.

[0014] In some examples, the first segmentation is a semantic segmentation of the first view of the scene, and the second latent representation is associated with a depth map of the second view of the scene.

[0015] In some examples, at least one of the first potential representation and the second potential representation is a predetermined representation.

[0016] According to a second aspect of the present invention, a robotic device is provided, comprising: a system according to any one of the above examples; one or more actuators for enabling the robotic device to interact with a surrounding three-dimensional environment, wherein at least a portion of the surrounding three-dimensional environment is shown in the scene; and an interaction engine comprising at least one processor to control the one or more actuators.

[0017] According to a third aspect of the present invention, a method for processing image data is provided, the method comprising: obtaining a first latent representation associated with a first segmentation of a first view of a scene represented by the image data, wherein the first segmentation is a semantic segmentation; obtaining a second latent representation associated with a second view of the scene; and jointly optimizing the first latent representation and the second latent representation in a latent space to obtain an optimized first latent representation and an optimized second latent representation.

[0018] In some examples, the image data includes a first frame representing the first view of a scene and a second frame representing the second view of the scene, the first segmentation is a semantic segmentation of the first view of the scene, and the second latent representation is associated with the second segmentation being the semantic segmentation of the second view of the scene. In these examples, the method can include obtaining a third latent representation associated with depth data of the scene; obtaining a fourth latent representation associated with the depth data of the scene; and jointly optimizing the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation in the latent space to obtain the optimized first and second latent representations and the optimized third and fourth latent representations.

[0019] According to a fourth aspect of the present invention, a method for training a latent representation prediction engine to predict semantic segmentation of an input image is provided, the method comprising: detecting image features of an image; encoding the ground truth semantic segmentation of the image using an encoder of an autoencoder to obtain a latent representation of the ground truth semantic segmentation; decoding the latent representation of the ground truth semantic segmentation using a decoder of the autoencoder to obtain a predicted semantic segmentation of the image, wherein the autoencoder is adjusted using the image features; and updating the latent representation prediction engine using a loss function based on a comparison between the predicted semantic segmentation of the image and the ground truth semantic segmentation of the image.

[0020] In some examples, the decoder uses the image features for adjustment; or the encoder uses the image features for adjustment and the decoder uses the image features for adjustment.

[0021] In some examples, the method includes training the latent representation prediction engine to predict the semantic segmentation and depth map associated with the input image. In these examples, the encoder can be a first encoder, the decoder can be a first decoder, the autoencoder can be a first autoencoder, the loss function can be a first loss function, and the method can include: encoding a ground truth depth map associated with the image using a second encoder of a second autoencoder to obtain a latent representation of the ground truth depth map; decoding the latent representation of the ground truth depth map using a second decoder of the second autoencoder to obtain a predicted depth map for the image, wherein the second autoencoder is conditioned using the image features; and updating the latent representation prediction engine using a second loss function based on a comparison between the predicted depth map and the ground truth depth map. The second decoder is conditioned using the image features; or the second encoder can be conditioned using the image features and the second decoder can be conditioned using the image features.

[0022] In some examples, training the latent representation prediction engine includes training the encoder and the decoder to perform variational autoencoding of an input semantic segmentation of the input image.

[0023] In some examples, the decoder comprises a linear decoder.

[0024] In some examples, the encoder is arranged to produce an encoded output at each of a plurality of different resolutions, and the method includes: detecting the image features of the image at each of the plurality of different resolutions; and using the image features to adjust the encoder by combining the image features for a given resolution with the encoded output for the given resolution for each of the plurality of different resolutions.

[0025] In some examples, the decoder is arranged to produce a decoded output at each of a plurality of different resolutions, and the method includes: detecting the image characteristics of the image at each of the plurality of different resolutions; and using the image characteristics to adjust the decoder by combining the image characteristics for a given resolution with the decoded output for the given resolution for each of the plurality of different resolutions.

[0026] In some examples, the image is a color image.

[0027] In some examples, the loss function includes a regularization term.

[0028] According to a fifth aspect of the present invention, there is provided a non-transitory computer-readable storage medium comprising computer-executable instructions which, when executed by a processor, cause a computing device to perform any of the methods described above.

[0029] Further features will become apparent from the following description which proceeds with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A is a schematic diagram showing an example of an environment to be mapped;

[0031] Figure 1B is a schematic diagram illustrating available degrees of freedom of an exemplary apparatus;

[0032] Figure 1C is a schematic diagram illustrating video data generated by an exemplary capture device;

[0033] Figure 2 is a flowchart illustrating a method of processing image data according to an example;

[0034] Figure 3 is a flowchart illustrating a method of processing image data according to another example;

[0035] Figure 4 is a schematic diagram of a system for processing image data according to an example;

[0036] Figure 5 is a schematic diagram of a system for processing image data according to another example;

[0037] Figure 6 is a schematic diagram of an autoencoder conditioned on image features according to an example;

[0038] Figure 7 is a flow chart illustrating an optimization method according to an example;

[0039] Figure 8 is a flowchart illustrating an optimization method according to another example;

[0040] Figure 9 is a flowchart illustrating an optimization method according to yet another example;

[0041] Figure 10A is a schematic diagram of components of a computing system according to an example;

[0042] Figure 10B is a schematic diagram illustrating components of a robotic apparatus according to an example;

[0043] Figure 11 is a schematic diagram illustrating a non-transitory computer-readable medium according to an example;

[0044] Figure 12 is a schematic diagram of a latent representation prediction engine during training according to an example;

[0045] Figure 13 is a flowchart illustrating a method of training a latent representation prediction engine according to an example;

[0046] Figure 14 is a flowchart illustrating a method of training a latent representation prediction engine according to another example. DETAILED DESCRIPTION

[0047] Certain examples described herein enable obtaining a latent representation of a scene. A latent representation is, for example, a representation inferred from measurements. A latent representation is sometimes referred to as a "hidden" set of variable values because they may not be directly measurable from the environment. In this case, the measurement of the scene can be, for example, an image of the scene, which can be a two-dimensional (2D) color image, such as an RGB (red, green, blue) image, or an image including depth information, such as an RGB-D image (including depth (D) data). Typically, the latent representation is more compact than the direct measurement, e.g., having a lower dimensionality. Therefore, such a latent representation can be more efficiently processed and stored. For example, the latent representation can be generated using one or more "hidden" layers of a probabilistic model or a neural network architecture.

[0048] In some cases, the examples described herein can be used to obtain semantic segmentation of a scene from a latent representation of the scene. Semantic segmentation can be thought of as object segmentation, such as labeling of portions of an image, where each label includes an association with a particular object or class of objects. An object can refer to any visible thing or entity with a physical presence that a robot can interact with, for example. Objects are therefore considered broadly and include many entities such as walls, doors, floors, and people, as well as furniture, other devices, and regular objects in a home, office, and / or exterior space. Using semantic segmentation, mapping of an environment can be improved, such as enabling improved interaction between a robotic device and the environment. For example, if a home robot's map includes semantic segmentation that identifies regions of space associated with particular objects, the robot can distinguish between a "door" and a "wall."

[0049] In some cases, the examples described herein can be used to obtain a depth map of a scene from a latent representation of the scene. For example, a depth map indicates the depth associated with spatial elements (e.g., pixels or image portions) of an image of the scene. The depth value of a pixel or image portion can represent the distance to a surface in the environment along the line of sight from a camera that is observing the scene. Thus, the mapping of the scene can be improved by using the depth map, which can also enhance or improve the interaction between the robotic device and the scene. For example, the depth map can be used to control the robotic device to accurately grasp an object by moving a gripping mechanism of the robotic device to a position within the environment that more closely corresponds to the depth of the object to be grasped.

[0050] In some examples described herein, the latent representation can be an optimized latent representation, e.g., a more refined or accurate estimate of the latent representation compared to the initial estimate of the latent representation. In this case, a first latent representation associated with a first segmentation of a first view of a scene and a second latent representation associated with a second view of the scene can be obtained. Segmentation can be generally considered to refer to labeling image portions with appropriate labels that represent the characteristics of a given image portion. For example, the first segmentation can be a semantic segmentation, where image portions are associated with specific objects or object classes. In these examples, the first latent representation and the second latent representation are jointly optimized in the latent space to obtain an optimized first latent representation and an optimized second latent representation. Jointly optimizing the first and second latent representations preserves the correlation between the first and second views and improves the consistency of the latent representations. The optimized first and second latent representations can therefore more accurately represent the characteristics of the scene. Furthermore, a segmentation or mapping can be obtained from at least one of the first and second latent representations that can be more internally consistent. For example, an optimized first segmentation (e.g., a semantic segmentation) can be derived from the optimized first latent representation. The distribution of semantic labels for the optimized first segmentation may be smoother than otherwise.

[0051] As an example, in previous approaches, the semantic label associated with one pixel can be independent of the semantic labels associated with neighboring pixels. Therefore, using such previous approaches can cause semantic segmentation to change rapidly and dramatically across an image. For example, if an image has a table, three neighboring pixels of the table may each be associated with a different corresponding label (e.g., "table," "bed," "chair"), despite the fact that each of these pixels should be associated with the same label ("table").

[0052] In contrast, jointly optimizing a first latent representation and a second latent representation (as in the examples described herein) can improve the smoothness of a segmentation (e.g., a semantic segmentation) obtained from the optimized latent representations. For example, for an image of a table, three adjacent pixels can each be associated with the same label ("table") using the methods herein due to the correlations preserved by the joint optimization. In examples, the methods described herein can therefore be used to obtain an optimized semantic segmentation (e.g., from the optimized latent representation) with improved semantic consistency, such that any given part of the scene has the same semantic label regardless of the viewpoint. Different representations, such as a semantic segmentation and a depth map and / or different frames of one or more of the semantic segmentation and the depth map can be jointly optimized, for example, different modalities and / or different times of data representing views of a scene (e.g., data from related or shared camera poses) can be jointly optimized. Thus, the interaction between a robotic device and its environment can be improved by using such a segmentation.

[0053] Figure 1A and Figure 1B An example of a 3D space and the capture of image data associated with that space is schematically shown. Figure 1C A capture device configured to generate image data while viewing a space is shown. These examples are presented to better explain certain features described herein and should not be considered limiting; certain features have been omitted and simplified for ease of explanation.

[0054] Figure 1A An example 100 of a three-dimensional space 110 is shown. The 3D space 110 may be an interior and / or exterior physical space, such as at least a portion of a room or a geographic location. The 3D space 110 in this example 100 includes a plurality of physical objects 115 located within the 3D space. These objects 115 may include one or more of the following: people, electronic devices, furniture, animals, building parts, and equipment. Although Figure 1A The 3D space 110 in FIG. 1 is shown as having a bottom surface, but this need not be the case in all implementations, for example the environment may be airborne or in outer space.

[0055] Example 100 also illustrates various example capture devices 120-A, 120-B, 120-C (collectively referenced by reference numeral 120) that may be used to capture video data associated with 3D space 110. A capture device such as Figure 1ACapture device 120-A) may include a camera arranged to record data generated from observing 3D space 110 in digital or analog form. For example, capture device 120-A may be a monocular capture device, such as a monocular camera. Monocular cameras typically capture images of a scene from one position at a time and may have a single lens or lens system. In contrast, a stereo camera generally includes at least two lenses, each of which has a separate image sensor. A monocular capture device that can be used as capture device 120-A may be a monocular multi-directional camera device that is arranged to capture images of 3D space 110 from multiple angular positions. In use, multiple images can be captured one after another. In some cases, the multiple angular positions cover a wide field of view. In certain cases, capture device 120-A may include an omnidirectional camera, such as a device arranged to capture a substantially 360-degree field of view. In this case, the omnidirectional camera may include a device with a panoramic annular lens, such as a lens that can be mounted with respect to a charge coupled array.

[0056] In order to capture multiple images of the 3D space from multiple different positions, the capture device 120-A can be movable. For example, the capture device 120-A can be arranged to capture different frames corresponding to different observed portions of the 3D space 110. The capture device 120-A can be movable relative to a static mount and can, for example, include an actuator to change the position and / or orientation of the camera relative to the 3D space 110. In another case, the capture device 120-A can be a handheld device that is operated and moved by a human user. In one case, the capture device 120-A can include a static image device configured to capture a sequence of images; in another case, the capture device 120-A can include a video device to capture video data including a sequence of images in the form of video frames. For example, the capture device 120-A can be a monocular camera or a monocular capture device to capture or otherwise obtain frames of video data.

[0057] exist Figure 1A, multiple capture devices 120-B, 120-C are also shown as being coupled to a robotic device 130 that is arranged to move within the 3D space 110. The robotic device 130 may include autonomous aerial and / or ground mobility devices. In this example 100, the robotic device 130 includes actuators 135 that enable the device to navigate within the 3D space 110. These actuators 135 include wheels in the illustration; in other cases, they may include tracks, digging mechanisms, rotors, etc. One or more capture devices 120-B, 120-C may be statically or movably mounted on such a device. In some cases, the robotic device may be statically mounted within the 3D space 110, but a portion of the device (such as an arm or other actuator) may be arranged to move within the space and interact with objects within the space. Each capture device 120-B, 120-C may capture different types of image data, video data, and / or may include a stereoscopic image source. In one embodiment, at least one of the capture devices 120-B, 120-C is configured to capture photometric data, such as color or grayscale images. In one embodiment, one or more of the capture devices 120-B, 120-C can be moved independently of the robotic device 130. In one embodiment, one or more of the capture devices 120-B, 120-C can be mounted on a rotating mechanism, such as to rotate in an angled arc and / or to rotate 360 degrees, and / or be arranged with suitable optics to capture a panoramic view of a scene (e.g., up to a full 360 degree panorama). It should be understood that in some cases, a capture device similar or identical to capture device 120-A can be used as Figure 1A One or both of the capture devices 120-B, 120-C.

[0058] Figure 1B Examples 140 of the degrees of freedom available to the capture device 120 and / or robotic device 130 are shown. In the case of a capture device such as 120-A, the orientation 150 of the device can be co-linear with the axis of a lens or other imaging device. As an example of rotation about one of three axes, a normal axis 155 is shown. Similarly, in the case of the robotic device 130, an alignment direction 145 of the robotic device 130 can be defined. This can indicate the facing and / or direction of travel of the robotic device. The normal axis 155 is also shown. Although only a single normal axis is shown with reference to the capture device 120 or robotic device 130, these devices can rotate about any one or more of the axes schematically shown as 140 as described below.

[0059] More generally, the orientation and position of a capture device may be defined in three dimensions with reference to six degrees of freedom (6DOF): the position may be defined in each of the three dimensions, e.g., by [x, y, z] coordinates, and the orientation may be defined by an angular vector representing a rotation about each of the three axes, e.g., [θ x ,θ y ,θ z ]. Position and orientation can be viewed as transformations in three dimensions, for example relative to an origin defined in a 3D coordinate system. For example, an [x, y, z] coordinate can represent a translation from the origin to a specific position in the 3D coordinate system, and an angle vector [θ x ,θ y ,θ z ] can define rotations within a 3D coordinate system. A transformation with 6 DOF can be defined as a matrix, such that multiplication by the matrix applies the transformation. In some implementations, a capture device can be defined with reference to a restricted set of these six degrees of freedom, e.g., the y dimension can be constant for a capture device on a ground vehicle. In some implementations, such as implementations of robotic device 130, the orientation and position of a capture device coupled to another device can be defined with reference to the orientation and position of the other device, e.g., the orientation and position of robotic device 130.

[0060] In the examples described herein, the orientation and position of a capture device, such as listed in a 6DOF transformation matrix, can be defined as the pose of the capture device. Similarly, the orientation and position of an object representation, such as listed in a 6DOF transformation matrix, can be defined as the pose of the object representation. The pose of the capture device can change over time, such as as video data or a series of still images are recorded, such that the capture device can have a different pose at time t+1 than at time t. In the case of a handheld mobile computing device that includes a capture device, the pose can change as a user moves the handheld device within the 3D space 110.

[0061] Figure 1C An example of a capture device configuration is schematically shown. Figure 1C In example 160, capture device 165 is configured to generate image data 170. Figure 1C In the example, image data 170 includes a plurality of frames 175. Each frame 175 may be associated with a capture such as Figure 1A The image of the 3D space in 110 is related to a specific time t in the time period (ie, F t ). Frame 175 typically includes a 2D representation of the measured data. For example, frame 175 may include a 2D array or matrix of pixel values recorded at time t. Figure 1CIn the example of , all frames 175 within the image data have the same size, although this need not be the case in all examples. The pixel values within the frame 175 represent measurements of a particular portion of 3D space. Figure 1C In

[0015] , the image data represents multiple views of a scene from a monocular capture device, each view being captured at a different respective time t. However, in other cases, the image data captured by the capture device (or image or video capture system) may represent multiple views of a scene captured at the same or at least partially overlapping times. This may be the case if the capture device is a stereoscopic capture system.

[0062] exist Figure 1C In the example of , each frame 175 includes photometric data. Photometric data generally represents a photometric characteristic of an image, such as brightness, intensity, or color. Figure 1C In , each frame 175 includes an intensity value for each pixel of the frame 175, which may be stored, for example, in grayscale or brightness levels from 0 to 255 for each color band or color channel. For example, a grayscale of 0 corresponds to the darkest intensity (e.g., black), and a grayscale of 255 corresponds to the brightest intensity (e.g., white), with grayscale levels between 0 and 255 corresponding to intermediate intensities between black and white. Figure 1C In

[0045] , the photometric data represents red, green, and blue pixel intensity values at a given resolution. Thus, each frame 175 represents a color image, where each [x, y] pixel value in the frame comprises an RGB vector [R, G, B]. As an example, the resolution of the color data can be 640x480 pixels. In other examples, other color spaces can be used and / or the photometric data can represent other photometric characteristics.

[0063] The capture device 165 can be arranged to store the image data 170 in a coupled data storage device. In another embodiment, the capture device 165 can transmit the image data 170 to a coupled computing device, for example, as a data stream or on a frame-by-frame basis. The coupled computing devices can be directly coupled, for example, via a universal serial bus (USB) connection, or indirectly coupled, for example, where the image data 170 can be transmitted across one or more computer networks. In yet another embodiment, the capture device 165 can be configured to transmit the image data 170 across one or more computer networks for storage on a network attached storage device. The image data 170 can be stored and / or transmitted on a frame-by-frame basis or in batches (e.g., multiple frames can be bundled together).

[0064] One or more pre-processing operations may also be performed on the image data 170 before it is used in the examples described later. In one case, pre-processing may be applied so that the two sets of frames have a common size and resolution.

[0065] In some cases, capture device 165 may be configured to generate video data as image data. The video data may similarly represent multiple frames captured at different corresponding times. In one embodiment, the video data captured by capture device 165 may include a compressed video stream or file. In this case, the frames of the video data may be reconstructed from the stream or file, for example, as output from a video decoder. After pre-processing the video stream or file, the video data may be retrieved from a memory location.

[0066] supply Figure 1C As an example, and as will be appreciated, image data 170 may be generated using configurations other than those shown in the figures for use in the methods and systems described below. Image data 170 may also include any measured sensory input arranged in two dimensions, representing a captured or recorded view of a 3D space. For example, this may include photometric data, depth data, electromagnetic imaging, ultrasound imaging, and radar output. In these cases, only imaging devices associated with a specific form of data may be required, such as an RGB device without depth data.

[0067] Figure 2 is a flow chart illustrating a method 200 of processing image data according to examples herein. Figure 2 At item 202 of , a first latent representation L1 is obtained. The first latent representation is associated with a first segmentation of a first view of a scene represented by the image data. If the segmentation is derivable from the latent representation (e.g., by processing the latent representation), the latent representation may be considered to be associated with a segmentation of a view of the scene, as described with reference to Figure 3 As discussed further.

[0068] exist Figure 2 , the first segmentation is a semantic segmentation. The first latent representation may be lower dimensional, more compact, or otherwise representable using fewer parameters than the semantic segmentation. In this way, the first latent representation may more efficiently represent the semantic segmentation without sacrificing much, if any, detail. The first latent representation may, for example, be a code that is, for example, a sequence or other arrangement of numbers, letters, and / or symbols used to represent the semantic segmentation. As an example, the code may be a tensor of numeric or other values, such as a vector. The first latent representation may have a predetermined or fixed data size. In this way, the size of the first latent representation may be selected to store a sufficient amount of detail to allow accurate reconstruction of the semantic segmentation, while being small enough to be efficiently processed or stored. Reference Figure 3 An example of obtaining the first latent representation is further discussed.

[0069] exist Figure 2At item 204 of , a second latent representation L2 is obtained. The second latent representation is associated with a second view of the scene. The first view and the second view may be identical to each other. In other words, the capture device used to capture the image data may not have moved between the capture of the first view and the second view. However, in other examples, the first view and the second view may be different from each other. However, the first view and the second view may partially overlap with each other such that at least a portion of the first view also appears in the second view (although in a different position). The second latent representation may be associated with the same segmentation as the first latent representation. For example, the second latent representation may be associated with a semantic segmentation of the second view of the scene. However, in other examples, the second latent representation may be associated with a segmentation or other mapping that is different from the first latent representation. For example, while the first latent representation is associated with a semantic segmentation, the second latent representation may be associated with a depth map.

[0070] exist Figure 2 At item 206 of , the first latent representation and the second latent representation are jointly optimized in the latent space to obtain an optimized first latent representation at item 208 and an optimized second latent representation at item 210. It should be understood that the optimized first latent representation and the optimized second latent representation can be obtained simultaneously, such as at the same time as each other, or one can be obtained before the other. An optimized semantic segmentation of the first view of the scene can be derived from the optimized first latent representation. However, the optimized first latent representation can be much smaller than the optimized semantic segmentation itself and can therefore be more efficiently stored and processed while allowing reproduction of the optimized semantic segmentation.

[0071] For example, jointly optimizing the first latent representation and the second latent representation in the latent space refers to an optimization process that iteratively updates the values of the first latent representation and the second latent representation to minimize the residual. Since the values of the first latent representation and the second latent representation themselves are iteratively updated, this optimization can be considered to be in the latent space. In contrast, an optimization process that iteratively updates the values of other variables and then derives the first latent representation and the second latent representation from these other variables may not be considered to be in the latent space. Figure 2 In the example of

[0045] , optimization within the latent space can include correlation between the first latent representation and the second latent representation, which can improve the accuracy with which the optimized first latent representation and the second latent representation obtained by the optimization capture details of the scene. This can improve, for example, the accuracy with which an optimized semantic segmentation derived from the optimized first latent representation represents the semantic content of the scene.

[0072] The optimized latent representation can be obtained from the image data itself, which is typically readily accessible in a robot mapping system, rather than from other data that needs to be captured and stored separately. For example, image data can be used during the optimization process. The image data, for example, represents at least a first view of a scene (which is associated with a first latent representation) and at least a second view of the scene (which is associated with a second latent representation). In this case, the image data can be used to identify a correspondence between a portion of the first view of the scene and a portion of the second view of the scene. For example, the image data can be used to identify a portion of the second view that corresponds to a given portion of the first view (in other words, it shows the same portion of the scene). Since the corresponding portions of the first and second views are the same portion of the scene, a given characteristic (e.g., semantic label or depth) of these portions of the first and second views should be the same. This can be exploited during the optimization process, which can, for example, be arranged to identify values of the first and second latent representations that minimize the difference between a characteristic derived from the first latent representation and the same characteristic derived from the second latent representation for the portions of the first and second views that are identified as showing the same portion of the scene.

[0073] Figure 3 is a flow chart illustrating a method 300 of processing image data according to further examples herein. Figure 3 The method 300 is similar to Figure 2 However, Figure 3 The method 300 is shown in more detail as how it can be implemented Figure 2 However, it should be understood that Figure 3 The method 300 can only be performed Figure 2 This is one way of describing method 200 , and other methods are possible in other examples.

[0074] At item 302, an initial value of a first latent representation L1 is obtained, which may be referred to as L 1初始 Similarly, at item 304, an initial value of the first latent representation L2 is obtained, which may be referred to as L 2初始 In this example, the first latent representation is associated with a semantic segmentation of a first view of a scene captured in the image data to be processed. Figure 2 As shown, the second latent representation is associated with a second view of the scene and can be a semantic segmentation or a different segmentation or mapping, such as a depth map.

[0075] In an example, at least one of the first potential representation and the second potential representation may be a predetermined representation. For example, the first potential representation and / or the second potential representation may be a default value or other predetermined value, such as a zero value. In this way, the predetermined representation may be an initial estimate of the first potential representation and / or the second potential representation, which is independent of the scene (and image data). This initial estimate is then used by the first potential representation and / or the second potential representation in this example. Figure 3 The optimization process is refined.

[0076] At item 306, a determination is made as to whether this is the first pass of the optimization process. If so, then at item 308, the initial values of the first latent representation and the initial values of the second latent representation are used as inputs L for the optimization. 1输入 、L 2输入 .

[0077] At item 308, the first latent representation L of the input 1输入 and the second latent representation L 2输入 is decoded to obtain a decoded first latent representation L 1d and the second latent representation L 2d In this case, the first latent representation and the second latent representation of the input are decoded using a decoder system trained to obtain a given segmentation or mapping from the input. For example, the decoder system may include a first decoder trained to obtain semantic segmentation from the first input latent representation and a second decoder trained to obtain a depth map from the second input latent representation. This will refer to Figure 4 Further described. In this case, the decoder system may include a normalization engine to obtain a segmentation or mapping normalized to a predetermined value range. In these cases, the decoded first latent representation and the second latent representation may be those obtained before normalization.

[0078] Using the decoded first latent representation and the second latent representation, an optimization process may be performed to identify optimized first latent representation and second latent representation. The optimization may be performed using any optimization process. Figure 3 In the example of , the optimization process involves using the Gauss-Newton algorithm, which can be used to solve nonlinear least squares problems (although this is just an example). Given n variables β = (β1, ..., β m ) of m residuals r=(r1,...,r m ), where m ≥ n, the Gauss–Newton algorithm can be used to iteratively find the values of the variables that minimize the sum of squares:

[0079]

[0080] The value of the variable is calculated iteratively as:

[0081]

[0082] Where T represents the matrix transpose, and J is the Jacobian matrix, which can be expressed as:

[0083]

[0084] This is Figure 3 As shown in Figure 3 At item 312 of , at least one residual is determined using the decoded first latent representation and the second latent representation. The residual is a measure of the difference between, for example, a measured value and a predicted value, and corresponds to, for example, an error term. Typically, the greater the difference between the measured value and the predicted value, the greater the residual. Thus, minimizing the difference between the measured value and the predicted value (e.g., by minimizing the residual or a function based on the residual) can be used to determine optimal parameters for a model from which the predicted value can be obtained. As an example, item 312 can involve calculating a residual that represents the difference between the decoded first latent representation and the decoded second latent representation.

[0085] As reference Figure 3 As explained, the image data may include a first view and a second view of a scene. The image data may be used during the optimization process to identify a portion of the second view of the scene (e.g., a pixel or other spatial element) that corresponds to a given portion of the first view of the scene (which may also be a pixel or other spatial element). In this case, the residual may represent the difference between a decoded first latent representation of the pixels of the first view of the scene and a decoded second latent representation of the pixels of the second view of the scene, the decoded second latent representation being identified as showing the same portion of the scene as the pixels of the first view of the scene.

[0086] At item 314, at least one Jacobian is determined using the at least one residual, for example, using the equation above. Thus, in examples such as this, the residual can be differentiable, such that the corresponding Jacobian can be computed. At item 316, the Jacobian is used to determine a first latent representation L that minimizes a function of the residuals (e.g., the sum of the squares of the residuals in examples where there is more than one residual). 1输出 and the second latent representation L 2输出 In this way, the first latent representation and the second latent representation are jointly optimized in the latent space.

[0087] In some cases, one or more Jacobians may be pre-computed to increase the speed with which the value of the Jacobian may be evaluated. For example, the Jacobian may depend on the image represented by the image data and not on other features. In this case, the Jacobian may be computed once for each input image and does not need to be recomputed for each iteration of the optimization process. In this way, the pre-computed Jacobian may be reused in subsequent iterations of the optimization process and may be reused in subsequent optimizations based on the same input images. For example, the Jacobian may be computed for a keyframe of a video and then stored for use in future optimizations involving the same keyframe. A keyframe may be a keyframe specified by an external system, such as an external SLAM system. In other cases, a frame obtained after a capture device observing a scene moves a distance exceeding a threshold distance may be a keyframe. In Figure 3 At item 318 of , it is determined whether the optimization is complete. For example, when the cost function (such as the sum of squares S(β) discussed above) meets or is less than a predetermined value, or after a predetermined number of iterations, the optimization can be considered complete. If the optimization is not complete, then Figure 3 The method involves using the output L of the previous iteration 1输出 、L 2输出 As the input L for subsequent iterations 1输入 、L 2输入 ,like Figure 3 Optimization is performed as described above via items 310, 312, 314, 316, and 318. Figure 7 、 Figure 8 and Figure 9 Further details of exemplary optimizations are described.

[0088] If the optimization is determined to be complete at item 318, the first latent representation L output by the optimization process is 1输出 and the second latent representation L 2输出 The values of can be regarded as the optimized first potential representation and the second potential representation, respectively. In some cases, the optimized first potential representation and the second potential representation can be stored in, for example, a Figure 3 In this way, the optimized first potential representation and the second potential representation can be retained for subsequent use. However, in Figure 3In the example of , the optimized first latent representation is decoded at item 322 to obtain a decoded first representation of a first view of the scene. The optimized second latent representation is decoded at item 324 to obtain a decoded second representation of a second view of the scene. The optimized first latent representation and the second latent representation may be decoded by a decoder system, such as the decoder system used at item 310 to decode the first latent representation and the second latent representation that were input to the optimization process. Figure 3 In the example of , both the optimized first latent representation and the second latent representation are decoded. However, it should be understood that in some cases, decoding of one or both of the optimized first latent representation and the second latent representation may be omitted.

[0089] The decoded first representation and the second representation may be considered to represent observable or measurable characteristics of the scene in a less compact manner than the optimized first latent representation and the second latent representation. The decoded second representation may be similar to the decoded first representation but represents a second view of the scene (which may be the same or different than the first view of the scene) and may represent different characteristics than the decoded first representation. For example, while the values of the decoded first representation may indicate the semantic content of a portion of the first view of the scene associated with those values, the values of the decoded second representation may indicate the depth of a portion of the second view of the scene associated with those values. However, in other cases, the decoded first representation and the second representation may represent the same characteristics as each other, but for different views of the same scene.

[0090] The decoded values of the first representation and the second representation may be unnormalized values. In this way, the decoded values of the first representation and the second representation may each be internally consistent but have different scalings from each other. Figure 3 This situation is shown in Figure 3 In the example of , at item 326, the decoded first representation is normalized to obtain an optimized first segmentation, which is, for example, a semantic segmentation of the first view of the scene. Figure 3 At item 328 of , the decoded second representation is normalized to obtain an optimized second segmentation, which is, for example, a segmentation of the second view of the scene, such as a semantic segmentation. The normalization of at least one of the decoded first representation and the second representation can be performed by a decoder system, such as a decoder system for decoding the first latent representation and the second latent representation during optimization (which itself can be a decoder system for decoding the optimized first latent representation and the second latent representation). Figure 3 In the example of , both the decoded first representation and the second representation are normalized. However, it should be understood that in some cases, the normalization of one or both of the decoded first representation and the second representation may be omitted.

[0091] By normalizing the decoded first and second representations, a more meaningful or intuitive representation can be obtained. For example, a softmax function can be used. The softmax function maps an n-dimensional vector of real values to an n-dimensional vector of real values in the range of 0 to 1. This can be summarized mathematically as:

[0092]

[0093] The real-valued K-dimensional vector is mapped to a real-valued K-dimensional vector σ(z), each value of which is in the range (0,1) and makes all values of the vector σ(z) add up to 1. indicates the set of all real-valued tuples with dimension K. However, other normalization functions may be used in other examples.

[0094] As an example, the decoded first representation can be normalized to obtain an optimized semantic segmentation of the first view of the scene. Such a semantic segmentation can be an array of values, each value in the range of (0,1), such that a pixel of the first view of the scene has a corresponding semantic segmentation value. In this case, a particular range of semantic segmentation values can be considered to correspond to a particular class of objects (such as "table"), and different ranges can be considered to correspond to different classes of objects (such as "bed"). In this way, the optimized semantic segmentation can be used to identify regions of the first view of the scene that correspond to a particular class of objects (or a particular object).

[0095] Figure 4 According to examples (such as Figure 2 and Figure 3Schematic diagram of a system 400 for processing image data (e.g., an example of a system 400). System 400 includes an input interface 402 for receiving image data 404 and transmitting the image data 404 for processing. Image data 404 represents at least one view of a scene. Input interface 402 can be an image acquisition interface that can be coupled to a capture device such as a camera. The image acquisition interface can include a hardware interface, such as a USB or network interface, and computer program code that implements a software driver. In one embodiment, system 400 can be configured to operate on streaming data (e.g., live video data) that can be received by the image acquisition interface of system 400. In another embodiment, system 400 can be communicatively coupled to a camera and arranged to store image data 404 received from the camera device in one or more persistent and non-persistent data storage devices, for example, data frames can be copied to memory and / or can be stored on a hard drive or solid-state storage device. In another embodiment, image data 404 can be stored external to system 400 in a given file format, for example, in one or more files accessible in the data storage device. In this case, system 400 can use or implement a portion of a file system to at least read data from the one or more files. In this case, input interface 402 may be any suitable interface to allow system 400 to receive image data 404 from a data storage device. System 400 may include dedicated processing electronics and / or may be implemented by computer program code executed by a processor of at least one computing device.

[0096] System 400 may include one or more embedded computing devices. This may include at least one processor operating in association with memory to execute computer program code loaded onto a computer-readable medium. The medium may include a solid-state storage device, such as an erasable programmable read-only memory, and the computer program code may include firmware. In other cases, system 400 may include a suitably configured system-on-chip, an application-specific integrated circuit, and / or one or more suitably programmed field programmable gate arrays. In one case, system 400 may be implemented by computer program code and / or dedicated processing electronics in a mobile computing device and / or a desktop computing device; in this case, system 400 may be configured to receive images transmitted from a robotic device and / or transmit the determined potential representation (or a segmentation or mapping derived from the determined potential representation) back to the robotic device. Other implementations of distributed computing may also be used without departing from the examples described herein. In one case, as with or in lieu of the previous case, system 400 may be implemented in whole or in part by one or more GPUs executing computer program code. In some cases, system 400 may be implemented by, for example, one or more functions being implemented in parallel on the cores of multiple processors and / or GPUs.

[0097] Figure 4 The system 400 further includes an initialization engine 406 configured to generate a first latent representation L1 408 associated with a first segmentation of at least a first view of a scene captured in the image data 404. In this example, the first segmentation is a semantic segmentation. The initialization engine 406 is further configured to generate a second latent representation L2 410 associated with at least a second view of the scene. Figure 3 As explained, the first view and the second view can be different from or the same as each other. The second latent representation can represent the same or different characteristics as the first latent representation. For example, the second latent representation can be associated with a semantic segmentation of at least the second view of the scene or a depth map of at least the second view of the scene. The term "generated" can be used broadly in this context to refer to newly generated first latent representations and second latent representations, or to refer to first latent representations and second latent representations retrieved by initialization engine 406 or otherwise obtained (e.g., from a storage device of system 400 or a storage device accessible to system 400).

[0098] The system 400 is arranged to input the image data 404 and the first latent representation and the second latent representation to an optimization engine 412, which is arranged to jointly optimize the first latent representation 408 and the second latent representation 410 in the latent space to obtain an optimized first latent representation 414 and an optimized second latent representation 416.

[0099] exist Figure 4 In the example of FIG. 4 , the optimization engine 412 is arranged to implement reference Figure 3 The optimization process described above is described (although this is merely an example and other optimization processes are possible). The optimization engine 412 includes a decoder system 418, which in this example includes a first decoder 420 configured to decode the first latent representation 408 and a second decoder 422 configured to decode the second latent representation 410. In this case, the first latent representation 408 and the second latent representation 410 are decoded during the optimization, as described with reference to FIG. Figure 3 As explained in item 310.

[0100] exist Figure 4 In the example, the first decoder 420 has been trained on multiple pairs of input image data and ground truth semantic segmentation to decode the first latent representation of the semantic segmentation to obtain the semantic segmentation itself. The first decoder 420 is, for example, a decoder of an autoencoder trained using multiple pairs of input image data and ground truth semantic segmentation, as shown in FIG. Figures 12 to 14 Further discussion.

[0101] Figure 4The second decoder 422 in the example of FIG has been trained on multiple pairs of input image data and ground truth segmentations or mappings corresponding to features that the second latent representation is intended to represent. For example, the second decoder 422 may have been trained on multiple pairs of input image data and ground truth depth maps to decode the second latent representation of the depth map to obtain the depth map itself. The second decoder 422 may also be a decoder of an autoencoder, which is typically a different autoencoder than the autoencoder of which the first decoder 420 may form a part.

[0102] It should be understood that in some cases, the first latent representation and the second latent representation may each represent a semantic segmentation (of the first and second views of the scene, respectively). In this case, the decoder system 418 may not include the second decoder 422. In these cases, both the first latent representation and the second latent representation may be decoded by the same decoder (e.g., the first decoder 420).

[0103] The first decoder 420 can be trained to output a predetermined latent representation as the most probable latent representation. For example, the first decoder 420 can be trained to output a zero code as the most probable latent representation for semantic segmentation. This behavior of the first decoder 420 can be imposed by training the first decoder 420 using a multivariate Gaussian prior centered at zero. With the first decoder 420 trained in this manner, the initialization engine 406 can be arranged to generate a predetermined representation as a first latent representation, which is, for example, the most probable latent representation. For example, the initialization engine 406 can be arranged to generate a zero latent representation as the first latent representation. The second decoder 422 can be trained similarly. Therefore, the initialization engine 406 can also or alternatively be arranged to generate a predetermined representation as a second latent representation, which is, for example, the most probable latent representation, such as a zero latent representation.

[0104] In addition to outputting the optimized first latent representation 414 and the second latent representation 416, the optimization engine 412 is also arranged to output the decoded first representation 424 and the second representation 426 and the optimized first segmentation 428 and the second segmentation 430, e.g., as described with reference to FIG. Figure 3 322, 324, 326, 328 of . It should be understood that in some cases, at least one of the optimized first and second latent representations 414, 416, the decoded first and second representations 424, 426, and the optimized first and second partitions 428, 430 may not be output by the optimization engine 412 or may be output at different times relative to each other. Furthermore, in some cases, the decoder system 418 may not form part of the optimization engine 412, but may instead be a separate or otherwise distinct component with which the optimization engine 412 may still communicate or between which data may be transferred.

[0105] Figure 5 is a schematic diagram of a system 500 for processing image data 504 according to another example. Figure 5 of and Figure 4 Similar or corresponding features are marked with the same reference numerals increased by 100. A corresponding description applies.

[0106] Figure 5 System 500 with Figure 4 The system 500 is the same as the system 400, except that the system 500 additionally includes a feature recognition engine 532, which is arranged to recognize image features 534 of the image data 504. The feature recognition engine 532 can be any engine that allows for the recognition of image features 534 such as corners, edges, shapes, colors, etc. For example, the feature recognition engine 532 can be implemented as a neural network architecture, which can be implemented in software, hardware, or a combination of software and hardware. Such a neural network architecture can be a convolutional neural network (CNN), such as a fully convolutional network. In an example, the feature recognition engine 532 includes a so-called "U-Net", which is an example of a fully convolutional network. The U-Net architecture is described in the 2015 paper "U-Net: Convolutional networks for biomedical image segmentation" by Ronneberger et al. (incorporated by reference where applicable).

[0107] exist Figure 5 In the example of , decoder system 512 is conditioned on image feature 534. One or both of first decoder 520 and second decoder 522 may be conditioned on image feature 534.

[0108] Figure 6 FIG2 is a schematic diagram of an autoencoder 600 conditioned on image features according to an example. An autoencoder can be considered conditioned on image features where at least one of the encoder or decoder is conditioned on the image features. For example, in some cases, the decoder of the autoencoder may be conditioned on the image features while the encoder of the autoencoder is not.

[0109] exist Figure 6 In , image features are obtained using a U-Net 601, which is arranged to decompose the input image into convolutional features. The U-Net 601 includes a contraction path (sometimes referred to as a downsampling path) schematically indicated by reference numeral 604. Figure 6In FIG, the contraction path 604 includes a set of downsampling blocks 602a-602c, each of which is arranged to obtain image features with a given resolution. Each downsampling block may include a convolution followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction path 604, spatial information is reduced. However, feature information is increased, for example by increasing the number of feature maps at each stage. In this way, an image representation with increasing coarseness but increasing the dimensionality of the image features can be obtained.

[0110] U-Net 601 also includes an expansion path 606, which is sometimes referred to as an upsampling path. Figure 6 In the example, the expansion path 606 includes a set of upsampling blocks 603a-603c. Each upsampling block combines the features and spatial information obtained by the contraction path 604, for example using upconvolution and concatenation (by Figure 6 (Indicated by the arrows between 602a and 603a, 602b and 603b, and 602c and 603c in FIG. 3 ). In this way, the upsampling blocks 603a-603c are arranged to obtain image features of multiple different resolutions.

[0111] pass Figure 6 The image features obtained by the upsampling blocks 603a-603c in are used to adjust the autoencoder 600. For example, the autoencoder is an artificial neural network that is trained to learn an identity mapping while having a bottleneck that forces the autoencoder to learn a compact representation of the data to be automatically encoded. For example, the autoencoder 600 can be trained to learn a compact representation of the semantic segmentation of the image (which may be referred to as a latent representation in this article). The autoencoder 600 includes an encoder 608 and a decoder 610. Figure 6 The decoder 610 can be used as Figure 5 at least one of the first decoder 520 and the second decoder 522 .

[0112] exist Figure 6 In the example of FIG6 , the autoencoder 600 is a conditional variational autoencoder having a variational part 612. The use of a variational autoencoder can increase the smoothness of the mapping between the latent representation and the segmentation (e.g., semantic segmentation) that can be derived from the latent representation (e.g., by decoding the latent representation using an autoencoder). In this case, small changes in the latent representation tend to result in small changes in the segmentation output by the autoencoder.

[0113] The encoder 608 includes a series of encoding components, which include a set of encoding blocks 614a-614c to encode the received data, such as the ground truth segmentation that the autoencoder 600 will be trained to perform autoencoding. The encoder 608 may also include a first component, which is arranged to perform convolution and subsampling of the input data, for example, before the encoding blocks 614a-614c. The encoding blocks 614a-614c can be considered to implement a downsampling operation. Downsampling can be achieved by changing the stride of a series of convolutions between the filter (sometimes called a kernel) associated with a given stage of the encoder 608 and the input of the encoder 608. The encoder 608 can be a convolutional neural network, such as a fully convolutional recognition model, such as a convolutional network described in the paper "Very Deep Convolutional Networks for Large-Scale Image Recognition" by K. Simonyan and A. Zisserman, which was published as a conference paper at ICLR 2015 (incorporated by reference where applicable).

[0114] In this case, the encoder 608 forms part of the variational autoencoder 600 rather than a vanilla encoder. Thus, in this case, the encoder 608 is trained to output the mean and uncertainty of the feature that the encoder 608 has been trained to encode. In an example, the encoder 608 can be trained using an input segmentation or mapping with multiple spatial elements. For example, the encoder 608 can be trained using a ground truth semantic segmentation, where the pixel array corresponds to the pixels of the input image. However, the pixels of the ground truth semantic segmentation may include semantic values (e.g., values indicating or otherwise representing the semantic label associated with the corresponding pixel of the input image) rather than photometric values. In such a case, the encoder 608 outputs, for example, an average semantic value and an uncertainty associated with the average semantic value (or a vector of mean and associated uncertainty) for each pixel of the ground truth semantic segmentation, rather than directly outputting a potential representation of the semantic segmentation for a given pixel. In these cases, the variational part 612 samples from a distribution having a mean corresponding to the average semantic value and an uncertainty corresponding to the uncertainty associated with the average semantic value to obtain a potential representation for a particular pixel. The distribution is, for example, a Gaussian distribution. This can be thought of as corresponding to sampling from the latent space associated with the features that the autoencoder 600 was trained to autoencode.

[0115] The latent representation obtained by the variational part 612 (e.g., a dimensionality reduction encoding of the input data) can then be decoded by the decoder 610 to obtain an estimated reconstruction of the data input to the encoder 608 (e.g., a semantic segmentation or a depth map). The decoder 610 includes a set of decoding blocks 615a-615c. The decoder 610 can be considered to implement an upsampling operation. For example, bilinear interpolation or deconvolution can be used to implement upsampling. During decoding, the decoder 610 outputs multiple feature maps at multiple different resolutions (which can, for example, be considered to correspond to corresponding decoded outputs). For example, each decoding block 615a-615c can output a decoded output at a different corresponding resolution. In Figure 6 In the example of FIG, the resolution of the decoded output generated by the decoding blocks 615a-615c decreases from decoding block 615a to decoding block 615c. The output of the decoder 610 is, for example, a decoded latent representation, which can be normalized, for example, with reference to Figure 3 As discussed in items 326 and 328.

[0116] exist Figure 6 , the encoder 608 and decoder 610 use image features obtained by the upsampling blocks 603a-603c of the U-Net 601 (which is an example of a feature recognition engine) for conditioning. However, in other cases, the decoder 610 uses image features for conditioning, while the encoder 608 does not use image features for conditioning (or vice versa). The image features have multiple resolutions. In this case, the image features obtained by the feature recognition engine (in this case, by the upsampling blocks 603a-603c) are used to condition the encoder 608 and decoder 610. In this case, the encoder 608 produces an encoded output (e.g., a feature map) at each of a plurality of different resolutions. For each of the plurality of different resolutions, the encoder 608 can be conditioned by combining the image features of the given resolution with the encoded output of the given resolution. Similarly, in this case, the decoder 610 produces a decoded output (e.g., a feature map) at each of the plurality of different resolutions. For each of the plurality of different resolutions, the encoder 610 can be conditioned by combining the image features of the given resolution with the decoded output of the given resolution. This uses Figure 6As shown by the arrows in , the lowest resolution image features output by the first upsampling block 603a of the U-Net 601a are combined with the lowest resolution encoded output generated by the third encoding block 614c and the lowest resolution decoded output generated by the first decoding block 615a. Similarly, the highest resolution image features output by the third upsampling block 603c are combined with the highest resolution encoded input generated by the first encoding block 614a and the highest resolution decoded output generated by the third decoding block 615c. Combining the image features with the encoded or decoded output can be performed, for example, by concatenating the image features (e.g., in the form of a tensor) with the encoded or decoded output (which can also be in the form of a tensor).

[0117] It should be understood that Figure 6 The arrangement of is simplified for ease of illustration and other layers may be included in other arrangements. For example, the autoencoder 600 may include at least one fully connected layer between the encoder 608 and the variational part 612.

[0118] As an example, something like Figure 6 The arrangement of the arrangement can be Figure 5For example, the system 500 may include a U-shaped multi-task network comprising a shared encoder and two separate decoders (one for conditioning the first decoder 520 and the other for conditioning the second decoder 522, although in some cases at least one of the first decoder and the second decoder may not be conditioned on image features). The encoder may be based on a "ResNet" model (e.g., ResNet101), as described in Kaiming He et al., "Deep Residual Learning for Image Recognition," 2015 (incorporated by reference where applicable). The encoder may be trained on one or more image datasets, such as ImageNet (as described in Deng et al., "ImageNet: A Large-Scale Hierarchical Image Database," 2009, incorporated by reference where applicable). The encoder may be trained as part of the implementation and / or use a set of pre-trained parameter values. For example, the decoder can include a RefineNet unit, as described in G. Lin et al., “RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation,” in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), incorporated by reference where applicable. The system 500 can also include two variational autoencoders (to provide a first decoder 520 and a second decoder 522). The encoder portion of these autoencoders can only be used during training, but can be omitted during inference when the ground truth values of the features to be determined (e.g., semantic segmentation or depth map) are generally not available. In these cases, the encoder portion can be, for example, a fully convolutional recognition model like VGG, and the decoder portion can be, for example, a linear generative model.

[0119] In examples such as these, the first decoder 520 or the second decoder 522 can be a linear decoder. This approach can be used to obtain a linear relationship between the latent representation and the segmentation associated with the latent representation, which is conditioned on the input image in a nonlinear manner. For example, this linearity allows the Jacobian to be pre-computed, such as for use during optimization. Optimization can therefore be performed more quickly than would otherwise be possible.

[0120] Figure 7 is a flow chart illustrating an optimization method 700 according to an example. Figure 7 At item 702, image data is received. The image data in this case includes a first frame representing a first view of a scene and a second frame representing a second view of the scene. This can be done, for example, using a Figure 4 and Figure 5 The input interfaces 402 and 502 are used to receive image data.

[0121] exist Figure 7 At item 704, for example, using Figure 4 and Figure 5 The initialization engine of initialization engine 406, 506 obtains a first latent representation L1 and a second latent representation L2. In this example, the first latent representation is associated with a first segmentation, which is a semantic segmentation of a first view of the scene. In this case, the second latent representation is associated with a second segmentation, which is a semantic segmentation of a second view of the scene. The first view and the second view of the scene can be the same or different from each other. In this example, the first latent representation and the second latent representation are predetermined representations, such as a zero representation.

[0122] exist Figure 7 At item 706, for example, using Figure 4 and Figure 5 The optimization engines 412 and 512 jointly optimize the first potential representation and the second potential representation, which for example achieves Figure 2 or Figure 3 Optimization method. Figure 7 In an example of

[0015] , the optimization includes determining a semantic error term that indicates the difference between the first latent representation and the second latent representation. This difference can be represented in the form of various functions that measure the distance or dissimilarity between two elements.

[0123] In one example, the first view I1 and the second view I2 of the scene partially overlap and therefore share a common field of view. In this example, a method such as Figure 4 and Figure 5 The decoder of the first decoder 420, 520 decodes the first latent representation L1 and the second latent representation L2 to obtain a decoded first latent representation L 1d and the second latent representation L 2d The decoded first latent representation L 1d and the second latent representation L 2d For example, it is not normalized and can therefore be a pre-softmax prediction obtained from the decoder. The semantic error term r s This can be determined using the decoded first and second latent representations, for example using:

[0124] r s =DS(L1d i , L 2d j )

[0125] Where DS represents a difference function, such as the Euclidean distance function, L 1d i represents the decoded first latent representation of the image region i in the first view of the scene, and L 2d j A decoded second latent representation of image region j in the second view of the scene is represented. Image region i in the first view of the scene corresponds to image region j in the second view of the scene. In other words, the same portion of the scene appears in both image regions i and j. Image region j in the second view of the scene that corresponds to image region i in the first view of the scene can be found using a dense correspondence function, for example, based on an estimated relative rigid body transformation of the scene from the first view to the second view.

[0126] Semantic error term r s You can use Figure 4 and Figure 5 The optimization engines 412 and 512 of FIG. 5 are used to determine the residual error (e.g., as Figure 3 The semantic error term r can be used to determine the s As residuals (e.g., as in reference Figure 3 314) to determine the Jacobian, which can then be used to determine the semantic error term r s The values of the first latent representation and the second latent representation that minimize a function of (such as a sum of squares function).

[0127] This process can be performed iteratively, for example, Figure 3 On this basis, we can Figure 7 The value of the first latent representation and the value of the second latent representation that minimize the semantic error term are found at item 708 of . These values may correspond to the optimized first latent representation and the optimized second latent representation, respectively. However, in some cases, further optimization or further processing of these values may be performed to obtain the optimized first latent representation and the optimized second latent representation. The optimized first semantic segmentation and the second semantic segmentation associated with the first view and the second view of the scene, respectively, may be derived from the optimized first latent representation and the second latent representation, for example, by decoding the optimized first latent representation and the second latent representation and performing a normalization process.

[0128] Figure 8 is a flow chart illustrating an optimization method 800 according to another example. Figure 8 The optimization method 800 is similar to Figure 7 Optimization method 700. Similar to Figure 7 , Figure 8 The optimization method 800 may be used to obtain an optimized first latent representation and a second latent representation associated with an optimized first semantic segmentation and a second semantic segmentation, the optimized first semantic segmentation and the second semantic segmentation being associated with a first view and a second view of a scene, respectively. Figure 8 The optimization method 800 may be used to obtain an optimized third latent representation and a fourth latent representation associated with an optimized first depth map and a second depth map, the optimized first depth map and the second depth map being associated with a first view and a second view of a scene, respectively.

[0129] Figure 8 Item 802 is similar to Figure 7 Item 702; the corresponding description will apply.

[0130] exist Figure 8 At item 804, a first latent representation L1 and a second latent representation L2 are obtained, for example using Figure 4 and Figure 5 Initialization engines 406 and 506 are used to initialize the scene. In this example, the first latent representation is associated with a first segmentation, which is a semantic segmentation of a first view of the scene. In this case, the second latent representation is associated with a second segmentation, which is a semantic segmentation of a second view of the scene. The first view and the second view of the scene can be the same or different from each other. In an example, the first latent representation and the second latent representation are predetermined representations, such as a zero representation.

[0131] Figure 8 Item 804 also involves obtaining a third latent representation L3 associated with the depth map of the first view of the scene, and a fourth latent representation L4 associated with the depth map of the second view of the scene. The third latent representation L3 and the fourth latent representation L4 may be obtained using a method such as Figure 4 and Figure 5 The initialization engine 406, 506 obtains the initialization engine, and may be a predetermined representation, such as a zero representation.

[0132] In accordance with Figure 8 In the example of , the first latent representation, the second latent representation, the third latent representation and the fourth latent representation are jointly optimized in the latent space to obtain the optimized first latent representation and the second latent representation as well as the optimized third latent representation and the fourth latent representation. This optimization can be performed using methods such as Figure 3 The method is implemented by the method and can be used such as Figure 4 and Figure 5 The optimization engines 412 and 512 are used to implement the optimization.

[0133] In particular, Figure 8 In the example of FIG, at item 806, the joint optimization of the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation in the latent space includes determining a semantic error term indicating the difference between the first latent representation and the second latent representation. The semantic error term can be as described in reference Figure 7 Determined as described in item 706.

[0134] In this example, at item 808, the joint optimization of the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation in the latent space further includes determining a geometric error term r indicative of a difference between the third latent representation and the fourth latent representation. d . Geometric error term r d This can be determined using a first depth map D1, which can be obtained by decoding the third latent representation (and, in some cases, normalizing the decoded third latent representation). Figure 4 and Figure 5 The third latent representation is decoded by a decoder of the second decoder 422, 522, which has been trained to decode the latent representation of the depth map to obtain the depth map. In this case, the geometric error term r d The determination of also uses a second depth map D2, which may be obtained by decoding (and, in some cases, normalizing) the fourth latent representation. For an image region i in the first view of the scene, the geometric error term r d It can be expressed as:

[0135]

[0136] where d1 i represents the depth value of image region i in the first view of the scene (as obtained from the first depth map D1), and D2 j represents the depth value of image region j in the second view of the scene (as obtained from the second depth map D2). Image region i in the first view of the scene corresponds to image region j in the second view of the scene. In this example, the geometric error term r d It is D1 i and D2 j But in other cases, the geometric error term r d May be or include based on D1 i and D2 j The geometric error term r is a difference function (such as the Euclidean distance function). d You can use Figure 4 and Figure 5 The optimization engines 412 and 512 of FIG. 5 are used to determine the residual error (e.g., as Figure 3 (as determined in item 312 of ).

[0137] exist Figure 8 At item 810, the values of the first potential representation, the second potential representation, the third potential representation, and the fourth potential representation that jointly minimize the semantic error term and the geometric error term are determined. This joint optimization may include evaluating the Jacobian using at least the semantic error term and the geometric error term as residuals. Such Jacobian can then be used to determine the values of the first potential representation, the second potential representation, the third potential representation, and the fourth potential representation that minimize a function of these residuals (e.g., a sum-of-squares function). In this way, the optimized first potential representation, the second potential representation, the third potential representation, and the fourth potential representation can be obtained.

[0138] Optimized semantic segmentations and depth maps can be obtained from the optimized first, second, third, and fourth latent representations, for example, by decoding these representations. For example, the optimized first and second latent representations can be decoded to obtain optimized semantic segmentations of the first and second views of the scene, respectively. The optimized third and fourth latent representations can be decoded to obtain optimized depth maps of the first and second views of the scene, respectively. In some cases, the optimized first and second latent representations are decoded using a first decoder trained to obtain semantic segmentations from an input latent representation of semantic segmentation, and the optimized third and fourth latent representations are decoded using a second decoder trained to obtain depth maps from an input latent representation of depth maps.

[0139] Figure 9 is a flow chart illustrating an optimization method 900 according to another example. Figure 9 The optimization method 900 is similar to Figure 8 Optimization method 800. However, Figure 9 The optimization method 900 also includes optimization based on photometric data associated with the first frame and the second frame of image data.

[0140] Figure 9 Items 902 and 904 are similar to Figure 8 Items 802 and 804; corresponding descriptions will apply.

[0141] At item 906, a photometric error term is determined that indicates photometric consistency between a first view of a scene (as captured in a first frame) and a second view of the scene (as captured in a second frame). A portion of the first view of the scene can be considered photometrically consistent with a corresponding portion of the second view (which shows the same portion of the scene as in the portion of the first view) where photometric characteristics (e.g., color or intensity values) are similar or identical. In other words, the same portion of the scene should appear similar or identical regardless of the viewpoint of the camera used to obtain the image of the portion of the scene. The photometric error term, for example, provides a measure of the degree to which a given portion of the first view (e.g., image area i of the first view) is photometrically consistent with a corresponding portion of the second view (e.g., image area j of the second view).

[0142] As an example, for an image region i in the first view of the scene, the photometric error term r p It can be expressed as:

[0143]

[0144] Among them I1 i represents the intensity of image region i in the first view of the scene, and I2 j represents the intensity of image region j in the second view of the scene. Image region i in the first view of the scene corresponds to image region j in the second view of the scene. Intensity is a measure of the amount of light received from a given part of the scene, for example. In examples where image regions i and j correspond to pixels in the first and second views, respectively, the pixel values of pixels i and j in the first and second frames may be used as the intensity I1 for determining the photometric error term. i 、I2 j However, this is just one example, and in other cases the photometric error term may be based on a photometric characteristic other than intensity, such as luminance (e.g., a measure of the visual perception caused by the brightness of a scene) or color (which may be represented as a position in a color space). In this example, the photometric error term r p It is I1 i and I2 j But in other cases, the photometric error term r p May be or include based on I1 i and I2 j The photometric error term r is a difference function (such as the Euclidean distance function). p An optimization engine can be used to determine, for example, Figure 4 and Figure 5 Optimization engine 412, 512.

[0145] Figure 9 Item 908 involves determining the semantic error term r s, the semantic error term indicates the difference between the first latent representation and the second latent representation (which are associated with the semantic segmentation of the first view and the second view of the scene, respectively). The semantic error term r s You can refer to Figure 8 Determined as described in item 806.

[0146] Figure 9 Term 910 involves determining the geometric error term r d , the geometric error term indicates the difference between the third latent representation and the fourth latent representation (which are associated with the depth maps of the first view and the second view of the scene, respectively). The geometric error term r d You can refer to Figure 8 Determined as described in item 808.

[0147] Photometric error term r p , semantic error term r s and the geometric error term r d At least one of can be used as a residual (e.g., as Figure 3 One or more of these residuals may be used in an optimization process, such as one involving determining a Jacobian using the residuals and using the Jacobian to determine values of the first, second, third, and fourth latent representations that minimize a function of the residuals, as described with reference to Figure 3 314 and 316. In some cases, the optimization process involves a joint optimization of at least one of two of these error terms. The optimization may involve multiple optimizations. The multiple optimizations may include optimization of a single error term or a joint optimization of at least two of the error terms. For example, in some cases where a photometric error term is determined, the optimization may include determining a corresponding value of at least one of the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation that minimizes the photometric error term. By minimizing the photometric error term, the pose of the camera (e.g., during the capture of the first frame and the second frame) may be determined more accurately.

[0148] Figure 9 The example involves determining values of the third and fourth latent representations that jointly minimize the photometric error term and the geometric error term at item 912. Thus, Figure 9 Term 912 may include determining a Jacobian based on the photometric error term and the geometric error term, and using the Jacobian to determine values of the third latent representation and the fourth latent representation that minimize a function of the photometric and geometric error terms.

[0149] exist Figure 9 At item 914 of , values of the first latent representation and the second latent representation that minimize the semantic error term are determined, for example, using a joint optimization process. Figure 9Item 914 may be similar to Figure 7 Item 708.

[0150] Finally, in Figure 9 At item 916 of , the values of the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation that jointly minimize the semantic error term and the geometric error term are determined. Figure 9 Item 916 of can include determining a Jacobian based on the semantic error term and the geometric error term, and using the Jacobian to determine values of the first, second, third, and fourth latent representations that minimize a function of the semantic and geometric error terms. The optimized values of the first and second latent representations obtained at item 914 and the optimized values of the third and fourth latent representations obtained at item 912 can be used as inputs to the optimization of item 916. In this way, both geometry and semantics can be coupled in an optimization framework, which can improve both geometric estimates (e.g., depth maps) derived from the third and fourth latent representations and semantic estimates (e.g., semantic segmentation) derived from the first and second latent representations.

[0151] Figures 7 to 9 These are merely examples of potential representations that can be determined using the methods herein. It should be understood that these methods can be used to obtain other potential representations. For example, in some cases, a first potential representation can be associated with a semantic segmentation of a first view of a scene and a second potential representation can be associated with a depth map of a second view of the scene.

[0152] Figure 10A 1 is a schematic diagram illustrating components of a computing system 1000 that can be used to implement any of the methods described herein. The computing system 1000 can be a single computing device (e.g., a desktop, laptop, mobile, and / or embedded computing device) or can be a distributed computing system distributed across multiple discrete computing devices (e.g., certain components can be implemented by one or more server computing devices based on requests from one or more client computing devices over a network).

[0153] The computing system 1000 comprises a camera 1002, which in this case is a video camera arranged to provide video frames (which for example comprise observations of a scene). The computing system 1000 comprises an image processing system 1004, which is arranged to implement a method according to the methods described herein. Figure 10AIn the embodiment of the present invention, image processing system 1004 is arranged to process image data obtained by camera 1002 to obtain an optimized segmentation, such as a semantic segmentation, of at least one view of a scene. In one embodiment, image processing system 1004 is arranged to obtain an optimized first segmentation of a first view of the scene and an optimized second segmentation of a second view of the scene. The optimized first segmentation and second segmentation are, for example, semantic segmentations. Image processing system 1004 can also be arranged to obtain other optimized segmentations or mappings, such as an optimized depth map for at least one view of the scene.

[0154] The computing system 1000 further comprises a tracking system 1006 arranged to determine the pose of the camera 1002 during observation of the scene. The computing system 1000 comprises a mapping system 1008 arranged to populate a map of the scene with the optimized segmentation or mapping obtained by the image processing system 1004. Figure 10A , the mapping system 1008 is arranged to populate a map of the scene with at least one of an optimized first segmentation of a first view of the scene and first pose data representing a first pose of the camera during capturing the first view of the scene, and an optimized second segmentation of a second view of the scene and second pose data representing a pose of the camera during capturing the second view of the scene.

[0155] Tracking system 1006 and mapping system 1008 may form part of a simultaneous localization and mapping (SLAM) system. SLAM systems within the field of robotic mapping and navigation are used to construct and update maps of an unknown environment while simultaneously localizing a robotic device associated with the map within the environment. For example, a robotic device may be a device that constructs, updates, and / or uses a map.

[0156] Figure 10B is a schematic diagram illustrating components of a robotic device 1010 according to an example. The robotic device 1010 includes Figure 10A The robotic device 1010 also includes one or more actuators 1012 to enable the robotic device 1010 to interact with the surrounding three-dimensional environment. At least a portion of the surrounding three-dimensional environment can be shown in a scene captured by the camera 1002 of the computing system 1000. Figure 10B In the case of Figure 1AThe robotic device 1010 may capture image data, for example in the form of video data, while navigating a particular environment. However, in another scenario, the robotic device 1010 may scan the environment or operate on image data received from a third party, such as a user with a mobile device or another robotic device. When the robotic device 1010 processes the image data, it may be arranged to obtain an optimized latent representation from which segmentations and / or mappings (e.g., semantic segmentations or depth maps) may be derived, for example, to enable the robotic device 1010 to map its environment.

[0157] The robotic device 1010 also includes an interaction engine 1014 comprising at least one processor to control the one or more actuators 1012 . Figure 10B The interaction engine 1014 may be configured to use the segmentation and / or mapping obtained by the robotic device 1010 to control the robotic device 1010 to interact with the surrounding three-dimensional environment. For example, semantic segmentation may be used to identify specific objects in the environment to be grasped, or a depth map may be used to identify the location of an object to be grasped and / or to avoid collisions with obstacles such as walls.

[0158] As referenced in this article Figure 10A and Figure 10B Examples of the functional components described herein may include dedicated processing electronics and / or may be implemented by means of computer program code executed by a processor of at least one computing device. In some cases, one or more embedded computing devices may be used. The components described herein may include at least one processor operating in association with a memory to execute computer program code loaded onto a computer-readable medium. The medium may include a solid-state storage device, such as an erasable programmable read-only memory, and the computer program code may include firmware. In other cases, the components may include a suitably configured system-on-chip, an application-specific integrated circuit, and / or one or more suitably programmed field programmable gate arrays. In one case, the components may be implemented by computer program code and / or dedicated processing electronics in a mobile computing device and / or a desktop computing device. In one case, as in the previous case or as an alternative to the previous case, the components may be implemented by one or more graphics processing units executing the computer program code. In some cases, the components may be implemented by, for example, one or more functions implemented in parallel on the cores of multiple processors and / or graphics processing units.

[0159] Figure 11is a schematic diagram of an example 1100 illustrating a processor 1102 and a non-transitory computer-readable storage medium 1104 including computer-executable instructions 1104. The computer-executable instructions 1104, when executed by the processor 1102, cause a computing device (such as a computing device including the processor 1102) to obtain at least one latent representation associated with a view of a scene (e.g., as captured in image data). The instructions may result in execution of a method similar to the example method described above. For example, the computer-readable storage medium 1106 may be arranged to store image data 1108 representing an observation of a scene. The computer-executable instructions 1104, when executed by the processor 1102, may be configured to cause the computing device to process the image data 1108 to generate at least one optimized latent representation 1110, which may be stored in the computer-readable storage medium 1106. Although in Figure 11 , image data 1108 and optimized latent representation 1110 are shown as being stored on computer-readable storage medium 1106, but in other examples, at least one of image data 1108 and optimized latent representation 1110 may be stored in a storage device external to (but accessible to) computer-readable storage medium 1106.

[0160] Another example in this paper involves training a latent representation engine to predict semantic segmentation of an input image. Figures 12 to 14 Describe these examples.

[0161] Figure 12 is a schematic diagram of a latent representation prediction engine 1200 during training according to an example. The latent representation prediction engine 1200 receives image data 1202. The image data 1202 in this case is training data, which is used to train the latent representation prediction engine 1200 to predict semantic segmentations of an input image. In this example, the image data 1202 includes image data representing the input image, for example, as a 2D array of pixel values (such as pixel intensity values). For example, the image can be a color image. The image data is processed by a feature recognition engine 1204 of the latent representation prediction engine 1200. The feature recognition engine 1204 is arranged to detect image features 1206 of the input image (e.g., by processing the image data 1202). Figure 12 The feature recognition engine 1204 and the image features 1206 can be used with Figure 5 The feature recognition engine 532 and the image features 534 are similar or identical; corresponding descriptions will apply.

[0162] Returning to the reference image data 1202, Figure 12The image data 1202 in the example also includes a ground truth semantic segmentation 1208 of the input image. For example, the ground truth semantic segmentation includes a plurality of spatial elements, each spatial element being associated with a corresponding portion of the input image. Each spatial element is labeled with a semantic label, which indicates the actual content of the corresponding portion of the input image. The actual content is, for example, the type or category of objects that appear in the portion of the input image, such as "table", "bed", "chair". The semantic label does not have to be a text label (such as a string), but can be a number or other label that still indicates an object of a particular category. There does not have to be a one-to-one mapping between the spatial elements of the ground truth semantic segmentation 1208 and the pixels of the input image, although there may be.

[0163] Figure 12 The image data 1202 also includes a ground truth depth map 1210 for the input image. The ground truth depth map 1210 can be similar to the ground truth semantic segmentation 1208, but includes depth values instead of semantic labels. For the ground truth semantic segmentation 1208, a one-to-one mapping between the depth elements of the ground truth depth map 1210 and the pixels of the input image is not required, although one may exist.

[0164] The ground truth semantic segmentation 1208 and the depth map 1210 are processed by the encoder system 1212 of the latent representation prediction engine 1200. In this example, the encoder system 1212 includes a first encoder 1214 and a second encoder 1216. The first encoder 1214 will be trained to encode the ground truth semantic segmentation 1208 to generate a latent representation of the ground truth semantic segmentation 1208. The second encoder 1216 will be trained to encode the ground truth depth map 1210 to generate a latent representation of the ground truth depth map 1210. The first encoder 1214 and the second encoder 1216 in this example are both conditioned using the image features 1206 obtained by the feature recognition engine 1204 and can be used with Figure 6 12. The second encoder 1216 may be similar or identical to the encoder 608 of FIG. 12. For example, the first encoder 1214 may be arranged to produce an encoded output at each of a plurality of different resolutions and the image features 1206 may be at each of the plurality of different resolutions. In this case, the first encoder 1214 may condition on the image features 1206 by combining the image features 1206 of a given resolution with the encoded output of the given resolution for each of the plurality of different resolutions (e.g., using stitching). However, in other examples, the second encoder 1216 may not use the image features 1206 for conditioning. The second encoder 1216 may be similar or different in structure to the first encoder 1214.

[0165] Figure 6The first encoder 1214 outputs a first latent representation 1218, which is a latent representation of the ground truth semantic segmentation 1208. The second encoder 1216 outputs a second latent representation 1220, which is a latent representation of the ground truth depth map 1210.

[0166] use Figure 12 The decoder system 1222 in the decoder system 1222 processes the first latent representation 1218 and the second latent representation 1220. The decoder system 1222 includes a first decoder 1224 and a second decoder 1226. The first decoder 1224 will be trained to decode the latent representation of the ground truth semantic segmentation 1208 to obtain a predicted semantic segmentation of the input image. The second decoder 1226 will be trained to decode the latent representation of the ground truth depth map 1210 to obtain a predicted depth map of the input image. The first decoder 1224 and the second decoder 1226 are both conditioned using the image features 1206 obtained by the feature recognition engine 1204 and can be used with Figure 6 12. The decoder 610 of FIG. 12 may be similar or identical to the decoder 610 of FIG. 12. For example, the first decoder 1224 may be arranged to produce a decoded output at each of a plurality of different resolutions and the image feature 1206 may be at each of the plurality of different resolutions. In this case, the first decoder 1224 may condition on the image feature 1206 by combining the image feature 1206 of a given resolution with the decoded output of the given resolution for each of the plurality of different resolutions (e.g., using splicing). The second decoder 1226 may be similar or different in structure to the first decoder 1224. For example, in some cases, the second decoder 1226 may not use the image feature 1206 for conditioning. Furthermore, in some cases, the first decoder 1224 and / or the second decoder 1226 may use the image feature 1206 for conditioning, while the first decoder 1214 and / or the second encoder 1216 do not use the image feature 1206 for conditioning. In certain cases, each of the first decoder 1224 and the second decoder 1226 uses the image feature 1206 for conditioning, and neither the first encoder 1214 nor the second encoder 1216 uses the image feature 1206 for conditioning, but this is merely an example.

[0167] The first encoder 1214 and the first decoder 1224 in this example correspond to a first autoencoder, which is trained to automatically encode the semantic segmentation of the input image. The second encoder 1216 and the second decoder 1226 in this example correspond to a second autoencoder, which is trained to automatically encode the depth map of the input image. Figure 6As explained, an autoencoder can be considered as being conditioned using image features, where at least one of the encoder or decoder of the autoencoder is conditioned using image features. Figure 12 In the example, both the first autoencoder and the second autoencoder are conditioned using image features.

[0168] As reference Figure 6 As described, the first autoencoder and the second autoencoder can be variational autoencoders to improve the smoothness of the mapping between the latent representation and the output of the autoencoder. For example, the first encoder 1214 and the first decoder 1224 can be trained to perform variational autoencoding of the input semantic segmentation of the input image, as described in reference Figure 6 In some cases, first decoder 1224 and / or second decoder 1226 can be linear decoders, such that decoder system 1222 includes linear decoders.

[0169] The first decoder 1224 is arranged to output a predicted semantic segmentation 1228 of the input image, and the second decoder 1226 is arranged to output a predicted depth map 1230 of the input image. The predicted semantic segmentation 1228 and the predicted depth map 1230 may be normalized semantic segmentations or depth maps. Normalization may be performed by the decoder system 1222 (e.g., by the first decoder 1224 and / or the second decoder 1226) or by another component.

[0170] The predicted semantic segmentation 1228 may be used to adjust weights or other parameters associated with the first encoder 1214 and the first decoder 1224, thereby training the first encoder 1214 and the decoder 1224 to more accurately automatically encode the input semantic segmentation. For example, the latent representation prediction engine 1200 may be updated using a loss function based on a comparison between the predicted semantic segmentation 1228 and the ground truth semantic segmentation 1208.

[0171] The weights associated with the second encoder 1216 and the second decoder 1226 may be updated in a similar manner. For example, the latent representation prediction engine 1200 may be updated based on a comparison between the predicted semantic segmentation 1228 and the ground truth semantic segmentation 1208 using a loss function.

[0172] The encoder system 1212 and the decoder system 1222 can then be used to process additional input images using the updated weights, and the weights can be updated again in a similar manner. This process can be repeated using a set of training data that includes multiple pairs of input image data and ground truth semantic segmentation and / or ground truth depth data. In this way, the latent representation prediction engine 1200 can be trained to decode input latent representations associated with semantic segmentation to obtain semantic segmentation (e.g., using a first decoder 1224), and to decode input latent representations associated with depth maps to obtain depth maps (e.g., using a second decoder 1226). The first decoder 1224 and the second decoder 1226 can therefore be referred to above. Figures 3 to 11 Methods are described to obtain an optimized latent representation and / or to obtain an optimized segmentation or mapping.

[0173] It should be understood that Figure 12 This is merely an example of a latent representation prediction engine 1200. In other cases, the feature recognition engine 1204 may be omitted. In such cases, the encoder system 1212 and the decoder system 1222 do not need to be conditioned on the image features 1206. Furthermore, although Figure 12 The latent representation prediction engine 1200 is arranged to predict both a semantic segmentation and a depth map associated with an input image, but in other cases, the latent representation prediction engine may be arranged to predict only one of the semantic segmentation or the depth map (or may be arranged to predict different segmentations or maps). For example, in one case, the latent representation prediction engine 1200 is arranged to obtain a predicted semantic segmentation 1228 without obtaining a predicted depth map 1230. In this case, the image data 1202 may not include the ground truth depth map 1210, and the second encoder 1216 and the second decoder 1226 may be omitted. In this case, at least one of the first encoder 1214 or the first decoder 1224 may be conditioned using the image features 1206, such that the autoencoder for automatically encoding the semantic segmentation (which in this case includes the first encoder 1214 and the first decoder 1224) is conditioned using the image features 1206.

[0174] Figure 13 is a flow chart illustrating a method 1300 of training a latent representation prediction engine according to an example. Figure 13 The method 1300 can be used with Figure 12 The method 1300 may be used with the latent representation prediction engine 1200 or a similar latent representation prediction engine, but omitting the second encoder 1216 and the decoder 1226. The method 1300 may be performed during a configuration phase, prior to performing the method for obtaining an optimized latent representation as described herein (e.g., using a decoder system that may be trained using the method 1300).

[0175] exist Figure 13 At item 1302, image features of the image are detected. This can be done using Figure 12 The image recognition engine 1204 is used to perform the feature recognition. The image in this case is represented by image data, such as training data including samples for a plurality of different scenes. Each sample of the training data may include photometric data representing the scene (which may be processed to detect image features) and at least one ground truth segmentation or mapping of the scene.

[0176] exist Figure 13 At item 1304 of , an encoder is used to encode the ground truth semantic segmentation of the image. The encoder can be conditioned by image features such as Figure 12 The first encoder 1214 of is used, although this is not required. In this way, a latent representation of the ground truth semantic segmentation can be obtained.

[0177] exist Figure 13 At item 1306 of , a decoder is used to decode the latent representation of the ground truth semantic segmentation to obtain the predicted semantic segmentation of the image. In this case, the decoder is conditioned using image features and can be used with Figure 12 The first decoder 1224 is similar or identical to the first decoder 1224.

[0178] exist Figure 13 At item 1308, a latent representation prediction engine is updated using a loss function based on a comparison between the predicted semantic segmentation of the image and the ground truth semantic segmentation of the image. In this case, the latent representation prediction engine includes an encoder system and a decoder system and may include a neural network architecture. The loss function generally provides a measure of the difference between the predicted value and the ground truth value and can take various forms. For example, the loss function may include the squared error between the predicted semantic segmentation and the ground truth semantic segmentation.

[0179] The loss function can include a reconstruction term (sometimes called reconstruction loss), which constrains the latent representation prediction engine to learn to accurately auto-encode the input (e.g., the ground truth semantic segmentation of a given sample). As an example, the segmentation labels of the ground truth semantic segmentation (which can be discrete numbers) can be one-hot encoded before being processed by the latent representation prediction engine. In this case, a multi-class cross entropy function can be used as the reconstruction loss, R:

[0180]

[0181] Where C is the number of categories, is the cth element of the one-hot encoded semantic label of the ith pixel in the ground truth semantic segmentation, and is the predicted semantic segmentation of the ith pixel (eg, it is the output of the decoder system after normalization). However, this is just an example and other reconstruction terms may be used in other cases.

[0182] The loss function may also include a regularization term (sometimes referred to as a regularization loss) that constrains the latent representation prediction engine to learn to predict latent representations within a meaningful latent space (e.g., such that latent representations that are closer in the latent space are more similar than those that are farther apart). As an example, the regularization term may be based on the Kullback-Leibler divergence, such as explained in DP Kingma and J. Ba. Adam, “Auto-Encoding Variational Bayes,” in the proceedings of the 2014 International Conference on Learning Representations (ICLR).

[0183] Items 1302 to 1308 may be repeated for multiple samples to determine a set of parameter values (e.g., weights associated with a neural network architecture) for a latent representation prediction engine so that the latent representation prediction engine can predict a semantic segmentation from a latent representation associated with the semantic segmentation.

[0184] Figure 14 is a flow chart illustrating a method 1400 of training a latent representation prediction engine according to an example. Figure 14 The method 1400 is similar to Figure 13 Method 1300, but Figure 14 The method 1400 can be used to train a latent representation prediction engine to predict both semantic segmentation and depth maps of an input image. Figure 14 The method 1400 can be used with Figure 12 The method 1400 may be used together with the latent representation prediction engine 1200, although other engines may be used in other examples. The method 1400 may be performed during a configuration phase, prior to performing a method for obtaining an optimized latent representation as described herein (e.g., using a decoder system that may be trained using the method 1400).

[0185] Figure 14 Items 1402 to 1406 and Figure 13 Items 1302 to 1306 are the same; corresponding descriptions will apply.

[0186] Figure 14 Item 1408 is similar to Figure 13 However, in this case, the loss function of 1408 is the first loss function because Figure 14Method 1400 also involves using a second loss function (described in reference item 1414) to train the latent representation prediction engine to predict the depth map.

[0187] exist Figure 14 At item 1410 of the decoder system, a ground truth depth map associated with the image is determined (image features are detected at item 1402). The ground truth depth map is determined using an encoder system to obtain a latent representation of the ground truth depth map. In this case, the encoder system is used to obtain latent representations of both the ground truth semantic segmentation and the ground truth depth map. However, the latent representation of the ground truth semantic segmentation can be obtained using a first encoder of the encoder system, while the latent representation of the ground truth depth map can be obtained using a second encoder of the decoder system. The first encoder and / or the second encoder can be conditioned on the image features, although this is not required.

[0188] exist Figure 14 At item 1412 of

[0066] , the latent representation of the ground truth depth map is used by a decoder system to obtain a predicted depth map for the image. In this case, the decoder system is used to obtain latent representations of both the ground truth semantic segmentation and the ground truth depth map. However, the latent representation of the ground truth semantic segmentation can be obtained using a first decoder of the decoder system, while the latent representation of the ground truth depth map can be obtained using a second decoder of the decoder system. At least one of the first decoder and the second decoder can be conditioned on image features.

[0189] exist Figure 14 At item 1414 of [ 0 ], the latent representation prediction engine is updated using a second loss function based on the comparison between the predicted depth map and the ground truth depth map. As with the first loss function, the second loss function may include a reconstruction term and / or a regularization term. However, the reconstruction term, for example, represents the difference between the predicted depth map and the ground truth depth map. As an example, the following expression may be used as the reconstruction loss R2 of the second loss function:

[0190]

[0191] Where N is the number of pixels in the depth map, is the predicted depth of pixel i, p i is the ground truth depth of pixel i, and b i is the prediction uncertainty of pixel i (eg, as predicted by the second decoder). However, this is merely an example and other reconstruction terms may be used in other cases.

[0192] Items 1410 to 1414 can be repeated for multiple samples to determine a set of parameter values (e.g., weights associated with a neural network architecture) for a latent representation prediction engine so that the latent representation prediction engine can predict semantic segmentation from a latent representation associated with semantic segmentation or a depth map from a latent representation associated with depth.

[0193] In some cases, the latent representation prediction engine can be jointly trained to automatically encode semantic segmentation and depth maps. In this case, a combined loss function (which is, for example, a function of the first loss function and the second loss function) can be minimized to update the parameter values of the latent representation prediction engine.

[0194] The above embodiments should be understood to be illustrative. Other examples are contemplated. It should be understood that the optimized first potential table and second potential representation (or other representations of scenes as described herein) need not be obtained for every frame of the video, but can be performed for a subset of frames such as key frames.

[0195] It should be understood that any feature described with respect to any example may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the appended claims.

Claims

1. A system for processing image data, the system comprising: an input interface for receiving the image data, wherein the image data represents at least one view of a scene; Initialize the engine, which generates: a first latent representation associated with a first segmentation of at least a first view of the scene, wherein the first segmentation is a semantic segmentation; and a second latent representation associated with at least a second view of the scene; as well as an optimization engine for jointly optimizing the first latent representation and the second latent representation in a latent space to obtain an optimized first latent representation and an optimized second latent representation, The jointly optimizing the first latent representation and the second latent representation includes iteratively updating the values of the first latent representation and the second latent representation to minimize the residual.

2. The system of claim 1 , comprising a decoder system to perform at least one of the following: decoding the optimized first latent representation to obtain a decoded first representation of the first view of the scene, wherein the optimized first partitioning of the first view of the scene is derivable from the decoded first representation; and The optimized second latent representation is decoded to obtain a decoded second representation of the second view of the scene, wherein the optimized second partitioning of the second view of the scene is derivable from the decoded second representation.

3. The system according to claim 2, comprising: a feature recognition engine for recognizing image features of the image data, Wherein said decoder system comprises at least one decoder conditioned on said image characteristics.

4. A system according to claim 3, wherein the feature recognition engine is arranged to recognize image features at each of a plurality of different resolutions, and The decoder in the at least one decoder is arranged to: generating a decoded output at each of the plurality of different resolutions; and For each of the plurality of different resolutions, the image features of a given resolution are combined with the decoded output of the given resolution.

5. A system according to any one of claims 2 to 4, wherein the decoder system is arranged to do at least one of the following: normalizing the decoded first representation to obtain the optimized first segmentation; and The decoded second representation is normalized to obtain the optimized second segmentation.

6. The system according to any one of claims 2 to 4, comprising: a tracking system for determining a pose of a camera during observation of the scene; as well as A mapping system arranged to populate a map of the scene using at least one of: the optimized first segmentation of the first view of the scene and first pose data representing a first pose of a camera during capture of the first view of the scene; as well as The optimized second segmentation of the second view of the scene and second pose data representing the pose of the camera during capture of the second view of the scene.

7. A system according to any one of claims 1 to 4, wherein the image data includes a first frame representing the first view of the scene and a second frame representing the second view of the scene, the first segmentation is a semantic segmentation of the first view of the scene, and the second latent representation is associated with the second segmentation which is a semantic segmentation of the second view of the scene.

8. The system of claim 7, wherein the optimization engine is arranged to jointly optimize the first latent representation and the second latent representation by: determining a semantic error term indicative of a difference between the first latent representation and the second latent representation; and A value of the first latent representation and a value of the second latent representation that minimize the semantic error term are determined.

9. A system according to claim 8, comprising a decoder system arranged to: decoding the first latent representation to obtain a decoded first representation; and decoding the second latent representation to obtain a decoded second representation, wherein the optimization engine is arranged to determine the semantic error term using the decoded first representation and the decoded second representation.

10. The system of claim 9, wherein the decoder system is trained on multiple pairs of input image data and ground truth semantic segmentation.

11. The system of claim 7, wherein the initialization engine is arranged to generate: a third latent representation associated with a depth map of the first view of the scene; and a fourth latent representation associated with a depth map of the second view of the scene, Wherein the optimization engine is arranged to jointly optimize the first latent representation, the second latent representation, the third latent representation and the fourth latent representation in the latent space to obtain the optimized first latent representation and the second latent representation and the optimized third latent representation and the fourth latent representation.

12. The system of claim 11 , wherein the optimization engine is arranged to jointly optimize the first potential representation, the second potential representation, the third potential representation, and the fourth potential representation by: determining a semantic error term indicative of a difference between the first latent representation and the second latent representation; determining a geometric error term indicative of a difference between the third latent representation and the fourth latent representation; and Determine values of the first latent representation, the second latent representation, the third latent representation and the fourth latent representation respectively to jointly minimize the semantic error term and the geometric error term to obtain optimized first latent representation, second latent representation, third latent representation and fourth latent representation.

13. The system of claim 12, comprising a decoder system comprising: A first decoder is arranged to do at least one of the following: decoding the optimized first latent representation to obtain an optimized semantic segmentation of the first view of the scene; and decoding the optimized second latent representation to obtain an optimized semantic segmentation of the second view of the scene; as well as a second decoder arranged to at least one of: decoding the optimized third latent representation to obtain an optimized depth map for the first view of the scene; as well as The optimized fourth latent representation is decoded to obtain an optimized depth map for the second view of the scene.

14. The system of claim 11, wherein the optimization engine is arranged to: determining a photometric error term indicative of photometric consistency between the first view of the scene and the second view of the scene; and A respective value of at least one of the first latent representation, the second latent representation, the third latent representation, and the fourth latent representation that minimizes the photometric error term is determined.

15. The system of any one of claims 1 to 4, wherein the first segmentation is a semantic segmentation of the first view of the scene, and the second latent representation is associated with a depth map of the second view of the scene.

16. The system of any one of claims 1 to 4, wherein at least one of the first potential representation and the second potential representation is a predetermined representation.

17. A robotic device comprising: The system according to any one of claims 1 to 15; one or more actuators for enabling the robotic device to interact with a surrounding three-dimensional environment, wherein at least a portion of the surrounding three-dimensional environment is shown in the scene; as well as An interaction engine includes at least one processor to control the one or more actuators.

18. A method for processing image data, the method comprising: obtaining a first latent representation associated with a first segmentation of a first view of a scene represented by the image data, wherein the first segmentation is a semantic segmentation; obtaining a second latent representation associated with a second view of the scene; and jointly optimizing the first latent representation and the second latent representation in a latent space to obtain an optimized first latent representation and an optimized second latent representation, The jointly optimizing the first latent representation and the second latent representation includes iteratively updating the values of the first latent representation and the second latent representation to minimize the residual.

19. The method of claim 18, wherein the image data comprises a first frame representing the first view of the scene and a second frame representing the second view of the scene, the first segmentation is a semantic segmentation of the first view of the scene, and the second latent representation is associated with the second segmentation being a semantic segmentation of the second view of the scene.

20. The method according to claim 19, comprising: obtaining a third latent representation associated with depth data of the scene; obtaining a fourth latent representation associated with depth data of the scene; as well as The first latent representation, the second latent representation, the third latent representation and the fourth latent representation are jointly optimized in the latent space to obtain the optimized first and second latent representations and optimized third and fourth latent representations.

21. A method for training a latent representation prediction engine to predict semantic segmentation of an input image, the method comprising: Detecting image features of an image; encoding the ground truth semantic segmentation of the image using an encoder of an autoencoder to obtain a latent representation of the ground truth semantic segmentation; decoding the latent representation of the ground truth semantic segmentation using a decoder of the autoencoder to obtain a predicted semantic segmentation of the image, wherein the autoencoder is conditioned using the image features; as well as The latent representation prediction engine is updated using a loss function based on a comparison between the predicted semantic segmentation of the image and the ground truth semantic segmentation of the image.

22. The method of claim 21, wherein: The decoder uses the image features for adjustment; or The encoder uses the image characteristics for conditioning and the decoder uses the image characteristics for conditioning.

23. The method of claim 21 or claim 22, comprising training the latent representation prediction engine to predict the semantic segmentation and depth map associated with the input image.

24. The method of claim 23, wherein the encoder is a first encoder, the decoder is a first decoder, the autoencoder is a first autoencoder, the loss function is a first loss function, and the method comprises: encoding a ground truth depth map associated with the image using a second encoder of a second autoencoder to obtain a latent representation of the ground truth depth map; decoding the latent representation of the ground truth depth map using a second decoder of the second autoencoder to obtain a predicted depth map for the image, wherein the second autoencoder is conditioned using the image features; as well as Based on a comparison between the predicted depth map and the ground truth depth map, the latent representation prediction engine is updated using a second loss function.

25. The method of claim 24, wherein: The second decoder uses the image feature for adjustment; or The second encoder uses the image characteristics for conditioning and the second decoder uses the image characteristics for conditioning.

26. The method of claim 21 or 22, wherein training the latent representation prediction engine comprises training the encoder and the decoder to perform variational autoencoding of an input semantic segmentation of the input image.

27. The method of claim 21 or 22, wherein the decoder comprises a linear decoder.

28. A method according to claim 21 or 22, wherein the encoder is arranged to produce an encoded output at each of a plurality of different resolutions, and the method comprises: detecting the image feature of the image at each of the plurality of different resolutions; as well as The encoder is adjusted using the image characteristics by combining the image characteristics for a given resolution with the encoded output for the given resolution for each of the plurality of different resolutions.

29. A method according to claim 21 or 22, wherein the decoder is arranged to produce a decoded output at each of a plurality of different resolutions, and the method comprises: detecting the image feature of the image at each of the plurality of different resolutions; as well as The decoder is adjusted using the image characteristics by combining the image characteristics for a given resolution with the decoded output for the given resolution for each of the plurality of different resolutions.

30. The method of claim 21 or 22, wherein the image is a color image.

31. The method of claim 21 or 22, wherein the loss function includes a regularization term.

Images