System for translating images between two different heterogeneous cameras and a method therefor

Abstract

The present disclosure proposes a system (100) for translating images between two different heterogeneous cameras and a method therefor. The system (100) comprises a source camera (S), a target camera (T), two generative adversarial networks (GANs), a transform prediction module (30), and at least one warping module (40). The transform prediction module (30) is configured to derive a transformation parameter using the system (100) and the method. The transform prediction module (30) is optimized using a loss function comprising a loss between a translated source image versus a target image and a translated target image versus a source image.

Description

Technical field of the invention

[0001] The present disclosure relates to the field of image processing. In particular, the present invention discloses a system for translating images between two different heterogeneous cameras and a method therefor. Background of the invention

[0002] Image classification, object recognition, semantic segmentation, and instance segmentation are all tasks of interest to computer vision. These tasks have their ultimate application in many real-world applications, ranging from assisted and automated driving scenarios to mobile phone applications. The recent success of deep learning approaches for computer vision tasks has led to a need for large amounts of annotated image data. Consider a scenario in which a manufacturer supplies a camera-based product to various OEMs. The manufacturer has collected a large amount of data about the camera used in the products (source) and offers the OEMs numerous services in addition to the product. All research and development work demonstrates a high level of maturity with regard to the product.Now, let's say, due to improvements in camera technology as well as the requirements set by various OEMs, the manufacturer has to switch to a new camera system (target system) that offers various improvements compared to the previous camera system.

[0003] In the above scenario, migrating the various systems and processes to the new camera or target camera system can be a time-consuming process because the source camera contains vast amounts of annotated image data. The source camera annotations must be replicated for numerous captured image types for the target camera system. The process of image annotation is extremely expensive and time-consuming, and may be the most expensive process in developing any data-driven machine vision application. Image-to-image translation is a technique that converts, or translates, a source image into a target image while preserving certain visual properties of the original image.

[0004] The state of the art in this area can be divided into two types. First, image-to-image translation systems attempt to take images from the source camera and translate the image properties, such as contrast, color correction, brightness, etc., to the target camera, while maintaining the same semantics as the source. Second, there are task-aware domain adaptation systems, where the goal is to induce a downstream task system to adapt, based on source camera inputs, to also work with the target camera inputs. Furthermore, with regard to flow estimation, previous techniques are limited to computing flow for single-camera or homogeneous camera setups.

[0005] However, none of the known approaches helps to automatically transfer the annotations available from the source camera to the images captured by the target camera. This must either be done explicitly using some pre-trained networks to obtain pre-labels, or it must go through the same manual annotation process via human labelers. The present invention proposes to solve this problem through a unified and task-independent system that provides robust and accurate translation of images between the source and target cameras and enables the transfer of semantics from the source camera to the target camera system. This also enables the rapid transfer of multiple types of annotations from the source camera to the target camera, obviating the need for annotation using methods external to the target camera system (such as pre-label networks or human annotation). Brief description of the attached drawings

[0006] An embodiment of the invention will be described with reference to the following accompanying drawings: Fig. 1 shows a system (100) for translating images between two different heterogeneous cameras (S,T); Fig. Figure 2 illustrates the process steps for translating images between two different heterogeneous cameras (S,T). Detailed description of the drawings

[0007] Fig. Figure 1 shows a system (100) for translating images between two different heterogeneous cameras (S, T). The system (100) comprises a source camera (S), a target camera (T), two generative adversarial networks (GANs), a transformation prediction module (30), and at least one warping module (40). The two cameras differ in terms of lens properties and the semantics of the captured images. Both camera S and camera T capture images simultaneously. The captured images are later synchronized, i.e., images Is from S are mapped to images It from T, so that the difference in the capture timestamp is kept below a predefined threshold (e.g., 9 ms): |Timestamp(Is)−Timestamp(It)|⇐9 ms (predefined threshold)

[0008] Note that for most of these image pairs (Is, It), the timestamp difference is rarely zero due to various factors that depend on the hardware characteristics of the two cameras, namely - frames per second (fps), post-capture trigger latency of the capture pipeline, etc.

[0009] The source camera (S), which captures a source image, is in communication with a first generative adversarial network (GAN). The first GAN (12) comprises a source encoder (Es), a source generator (Gs), and a discriminator. The target camera (T), which captures target images, is in communication with a second generative adversarial network (GAN), wherein the second GAN (22) comprises a target encoder (Et) and at least one target generator (Gt), thus characterizing this system (100).

[0010] The encoder-generator pair is tasked with learning the latent representation of its respective cameras and each generating an image from the latent image. For example, the encoder Es converts the input image Is into its latent representation Zs such that Zs = Es(Is), and then Gs decodes a Zs to reconstruct the image Is'. The same applies to the second or target camera (T), using Et and Gt for the latent representation, input, and reconstruction, with Zt, It, and It', respectively. The reconstruction target serves as a supervisor for learning camera-specific features. The respective discriminators ensure that encoders and generators are pitted against each other in such a way that learning is maximized at the encoders and generators. To perform a translation of the camera properties, a joint approach to learning latent representations for the two cameras is assumed.To achieve this, weight distribution is performed between encoders and generators. The weights of the last and first layers of encoders (Es, Et) and generators (Gs and Gt) are combined.

[0011] Once latent representations are learned, performing an exchange of latent features from one encoder to another generator is enabled. Essentially, the encoders encode the semantics into the latent code, remove the camera-specific information, and add the camera-specific information to the semantics passed to it as input once it is passed to the generator. The semantics of the Is are encoded in Zs, and when passed to the generator (Gt), the target camera-specific information (T) is added to the semantics specified by Zs to obtain Is-t according to the following equation: Is−t=Gt(Zs)

[0012] Both the learning of camera-specific features and the representation of the shared latent space are supported by the adversarial loss and the cyclic consistency loss of GANs. Therefore, the source generator (Gs) is trained to generate a translated target image. The target generator (Gt) is trained to generate a translated source image.

[0013] The transform prediction module (30) is configured to receive intermediate outputs of the source encoder (Es) and the target encoder (Et) along with a source image and a target image as input. The transform prediction module (30) executes an AI model. Those skilled in the art are familiar with the neural network types of AI models and the various types of neural architectures, such as convolutional neural networks, recurrent neural networks, transformers, and the like. It is understood that this disclosure is not specialized to any one type of model executed. Those skilled in the art will also recognize that the AI ​​module may be implemented as a set of software instructions, a combination of software and hardware, or any combination thereof. For example, neural networks are specialized silicon chips containing AI technology used for machine learning.In an exemplary embodiment of the present invention, the transformation prediction module (30) is a neural network trained to optimize the loss between the translated source image and the target image and the translated target image and the source image. The transformation prediction module (30) therefore derives (using method step 200) a final transformation parameter that can be used to translate images between the two cameras.

[0014] The warping module (40) communicates with the transformation prediction module (30). The output of the warping module (40), prepared with the inputs Zt and Zs for the warping module, is fed into the source generator (Gs) and the target generator (Gt). The warping module (40) can be either a logic circuit or a software program that responds to logical instructions and processes them to obtain a meaningful result.It can be implemented in the system (100) as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, one or more microchips or integrated circuits interconnected via a motherboard, as hard-wired logic, as software stored in a memory device and executed by a microprocessor, as firmware, as an application-specific integrated circuit (ASIC) and / or as a field-programmable gate array (FPGA) and / or as any component that processes signals based on operating instructions.

[0015] As used in this application, the terms "module", "system" and "network" refer to a computer-related entity, or an entity associated with or part of an operable device having one or more specific functions, where such entities may be either hardware, a combination of hardware and software, software, or executing software.

[0016] It should be understood at the outset that, although exemplary embodiments are illustrated in the figures and described below, the present disclosure should in no way be limited to the exemplary implementations and techniques illustrated in the drawings and described below.

[0017] Fig. Figure 2 shows process steps for translating images between two different heterogeneous cameras. The system (100) comprising the heterogeneous cameras was designed according to Fig.1. For clarity, it should be noted again that the system (100) comprises a source camera (S), a target camera (T), two generative adversarial networks (GANs), a transformation prediction module (30), and at least one warping module (40). The source camera (S), which captures source images, communicates with the first generative adversarial network (GAN), and the target camera (T), which captures target images, communicates with the second generative adversarial network (GAN).

[0018] Method step 201 comprises providing an intermediate output of the source encoder (Es) and the target encoder (Et), along with a source image and a target image, as input to a transformation prediction module (30). The source encoder (Es) and the target encoder (Et) receive the source image and the target image as input from the source cameras (S) and the target cameras (T), respectively. They extract some intermediate features from the image, which are fed into the transformation prediction module (30) as input.

[0019] The transformation prediction module (30) is based on optical flow-based semantic translation, where optical flow aims to predict the movement of semantics from one image to another for each pixel in the source image to the target image. The output T, i.e., the transformation parameter, has the size [height (Is) x width (Is) x 2], such that Ti,j = [u,v] represents the movement of pixel i,j in Is in the x- and y-directions, respectively, and the new coordinates of i,j in It are given by m,n, such that m=i+u n=j+v

[0020] This transformation parameter is derived in process step 206.

[0021] Method step 202 comprises executing a warping module (40) with outputs from the transform prediction module (30), the source encoder (Es), and at least the destination encoder (Et). Method step 203 comprises feeding Zt and Zs into the warping module, and the output serves as input to both the source generator (Gs) and the destination generator (Gt). The warping module (40) is primarily a differentiable interpolation module. The warping module (40) takes the feature map as input, repositions the pixel features, and interpolates them according to the transformation parameters provided by the transform prediction block. These transformation parameters are learned by the transformation module when it is configured.

[0022] Method step 204 includes executing the source generator (Gs) and the target generator (Gt) to generate a translated target image and a translated source image, respectively. Method step 205 includes defining a loss function that includes the loss between the translated source image and the target image and the translated target image and the source image (L1-L2((Is-t), It) + L1-L2((It-s), Is)).

[0023] Method step 206 includes optimizing the loss function to configure the transformation prediction module (30). In an exemplary embodiment of the present invention, the transformation prediction module (30) is a neural network. Optimizing the loss function includes adjusting the network parameters and hyperparameters of the neural network. The network strengthens the learning of the transformation parameter using the following optimization equation. L1−L2((Is−t), It)+L1−L2((It−s), Is)

[0024] This equation leads the transformation prediction module (30) to learn the transformation so that the pixel position in translated images is as close as possible, thereby reducing the loss. Also note that the weight of the above loss in the overall optimization equation gradually increases as training progresses, allowing the system (100) to first adapt to the camera characteristics and later to the semantic translation. Based on the above transformation prediction module (30), the transformation parameter "T" is derived.

[0025] Method step 207 comprises executing the configured transformation prediction module (30) to enable the translation of images between the two different heterogeneous cameras (S, T). After configuration, the transformation prediction module (30) provides transformation parameters that can be used to easily convert a source image relative to a target camera (T) into a translated source image.

[0026] The basic idea here is to provide semantic translation via the transformation parameter. Camera translation generally translates properties, such as brightness, contrast, saturation, etc., from one camera to another. Overall, the proposed system (100) attempts to mimic both the post-capture and post-processing pipelines of its respective camera through its encoder-generator pair. Translating such properties does not translate the change in the position of objects / semantics in the image, as these properties are introduced into the post-processing pipeline and are independent of the semantics captured by the camera. The semantic translation task aims to translate the camera lens-specific properties, mainly field of view, distortion, etc., from the source camera (S) to the target.

[0027] One skilled in the art will recognize that these method steps describe only a series of steps for achieving the objectives, and that these methods can be implemented with adaptations and changes to the system (100).

[0028] This idea of ​​developing a system (100) for translating images between two different heterogeneous cameras and a method for doing so helps reuse mature systems based on the source camera (S) and its data to dramatically accelerate the development of the target camera system (T). This is very important in applications where data contributes to increasing the accuracy of the system. For example, a user switching from an iPhone camera to a Samsung camera can have all their data annotations (face recognition tags, location tags, etc.) automatically transferred. Another viable technical solution for the transformation prediction module could be 3D rendering. 3D video rendering of images, such as SFMs (Structure from Motion), is often limited by camera type and positioning.Using the proposed cross-camera flow or transformation prediction can reduce the dependence on camera type and positioning. Similarly, in the automotive industry, a car manufacturer deploying driver assistance cameras can upgrade the vehicle-mounted cameras without worrying about excessive data collection and system transmission costs.

[0029] It should be understood that the embodiments explained in the above detailed description are merely illustrative and do not limit the scope of this invention. Any modification of the system (100) for translating images between two different heterogeneous cameras and a method therefor are conceivable and form part of this invention. The scope of this invention is limited only by the claims.

Claims

[1] System (100) for translating images between two different heterogeneous cameras (S,T), the system (100) comprising: a source camera (S) capturing a source image, which is in communication with a first generative adversarial network (GAN), the first GAN (12) comprising a source encoder (Es) and at least one source generator (Gs), a target camera (T) capturing target images, which is in communication with a second generative adversarial network (GAN), the second GAN (22) comprising a target encoder (Et) and at least one target generator (Gt), the system (100) characterized by : a transformation prediction module (30), wherein the transformation prediction module (30) is configured to receive intermediate outputs of the source encoder (Es) and the destination encoder (Et) as input; a warping module (40) in communication with the transformation prediction module (30), wherein the output of the warping module (40) is fed into the source generator (Gs) and the target generator (Gt). [2] System (100) for translating images between two different heterogeneous cameras (S,T) according to claim 1, wherein the source generator (Gs) is configured to generate a translated target image. [3] System (100) for translating images between two different heterogeneous cameras (S,T) according to claim 1, wherein the target generator (Gt) is configured to generate a translated source image. [4] System (100) for translating images between two different heterogeneous cameras (S,T) according to claim 1, wherein the transformation prediction module (30) is trained to optimize the loss between the translated source image versus the target image and the translated target image versus the source image. [5] Method (200) for translating images between two different heterogeneous cameras (S,T), a source camera (S) capturing a source image, which is in communication with a first generative adversarial network (GAN), wherein the first GAN (12) comprises a source encoder (Es) and at least one source generator (Gs), a target camera (T) capturing target images, which is in communication with a second generative adversarial network (GAN), wherein the second GAN (22) comprises a target encoder (Et) and at least one target generator (Gt), the method comprising: Providing (201) the output of the source encoder (Es) and the target encoder (Et) together with the source image and the target image as input to a transformation prediction module (30); Executing (202) a warping module (40) with outputs of the transformation prediction module (30), the source encoder (Es) and at least the target encoder (Et); feeding (203) the output of the warping module (40) as input to both the source generator (Gs) and the target generator (Gt); Executing (204) the source generator (Gs) and the target generator (Gt) to generate a translated target image and a translated source image, respectively; defining (205) a loss function comprising the loss between the translated source image versus the target image and the translated target image versus the source image; Optimizing (206) the loss function to configure the transformation prediction module (30); Executing (207) the configured transformation prediction module (30) to enable the translation of images between the two different heterogeneous cameras (S,T). [6] Method (200) for translating images between two different heterogeneous cameras (S,T) according to claim 5, wherein the transformation prediction module (30) is a neural network. [7] Method (200) for translating images between two different heterogeneous cameras (S,T) according to claim 5, wherein the optimization of the loss function comprises a fine-tuning of the network parameters and hyperparameters of the neural network. [8] Method (200) for translating images between two different heterogeneous cameras (S,T) according to claim 5, wherein the transformation prediction module (30) is configured to derive a transformation parameter.

Images