Improving image segmentation using neural network translation models

Abstract

Image segmentation is improved using a neural network translation model. The neural network includes an encoder, a common decoder, and a residual decoder. The encoder encodes an input image into a latent space. The latent space separates unique features from other common features. The common decoder decodes the common features residing in the latent space to generate a translated image that lacks the unique features. The residual decoder decodes the unique features residing in the latent space to generate an image delta corresponding to the unique features. The neural network combines the translated image with the image delta to generate a combined image that can include both the common features and the unique features. The combined image can be used to drive autoencoding. Once training is complete, the residual decoder can be modified to generate segmentation masks that indicate any regions of a given input image in which the unique features reside.

Description

Background Technology

[0001] In traditional image segmentation methods, neural networks are trained to identify specific features of an image based on training data comprising a set of labeled images. For example, a neural network can be trained to identify instances of cars within an image based on training data containing images of cars and labels indicating the regions in those images where the car resides. One drawback of this approach is that the training data needs to be manually generated by humans. Specifically, humans must visually evaluate each image in the training data and then generate one or more labels indicating the location of features of interest in those images. This process can be very time-consuming. Furthermore, certain types of training data (e.g., training data associated with medical imaging) can only be generated by experts who may not be easily found. Attached Figure Description

[0002] To gain a more detailed understanding of the above-described features of the various embodiments, the inventive concept briefly outlined above can be described in more detail by referring to the various embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings only illustrate typical embodiments of the inventive concept and should not be construed as limiting the scope in any way, and that other equivalent embodiments exist.

[0003] Figure 1 A system configured to implement one or more aspects of the various embodiments is shown.

[0004] Figure 2 According to various embodiments Figure 1 A more detailed illustration of the neural network.

[0005] Figure 3 This illustrates how training can be performed according to various embodiments. Figure 1 The neural network.

[0006] Figure 4 Various embodiments are shown. Figure 1 How does a neural network generate a segmentation mask?

[0007] Figure 5 Various embodiments are shown. Figure 1 How neural networks enhance the spatial details of translated images.

[0008] Figure 6 This is a flowchart of method steps for translating an image to generate a segmentation mask according to various embodiments.

[0009] Figure 7 This is a block diagram illustrating a computer system configured to implement one or more aspects of various embodiments.

[0010] Figure 8Included according to various embodiments Figure 7 A block diagram of the parallel processing unit (PPU) in the parallel processing subsystem.

[0011] Figure 9 Included according to various embodiments Figure 8 A block diagram of the General Processing Cluster (GPC) in a Parallel Processing Unit (PPU). Detailed Implementation

[0012] In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to those skilled in the art that the inventive concept can be practiced without one or more of these specific details.

[0013] As mentioned above, traditional methods for image segmentation involve training neural networks based on manually labeled training data. Generating this training data can be very time-consuming. Furthermore, some types of training data can only be generated by experts and may not be easily manually labeled.

[0014] To address this problem, various embodiments include neural networks trained to translate an input image that may include features of interest into a translated image that does not. The input image and the translated image can be compared to reveal the regions in the input image where the features of interest are located.

[0015] In one embodiment, the neural network is trained based on weakly labeled training data. The weakly labeled training data includes a first set of images containing the feature of interest and a second set of images not containing the feature of interest. The weakly labeled training data also includes a label for each image, indicating the presence or absence of the feature of interest. The feature of interest is unique for the first set of images and is therefore referred to herein as a "unique feature." Both sets of images also include various other features present in both sets of images. These features may be referred to herein as "common features."

[0016] In one embodiment, the neural network includes an encoder, a common decoder, and a residual decoder. The encoder and decoder are trained based on input images and corresponding labels contained in weakly labeled training data. The encoder encodes the input image into a latent space. The latent space disentangles the unique features of the input image from its common features. The common decoder decodes the common features residing in the latent space to generate a translated image lacking unique features. The residual decoder decodes the unique features residing in the latent space to generate an image increment corresponding to the unique features. The neural network combines the translated image with the image increment to generate a combined image that can contain both common and unique features. The combined image can be used to drive the training of the neural network via autoencoding. Once training is complete, the residual decoder can be modified to generate a segmentation mask instead of an image increment. A given segmentation mask indicates any region in a given input image where unique features reside.

[0017] At least one technical advantage of the disclosed technique compared to existing technologies is that it can accurately segment images based solely on weakly labeled training data. Therefore, features of interest can be easily identified within an image without requiring a complex manual process to generate training data. Another technical advantage of the disclosed technique compared to existing technologies is that it requires fewer reference segments to train the neural network compared to traditional methods, thus simplifying and accelerating the training process. These technical advantages represent several technological advancements compared to existing methods.

[0018] System Overview

[0019] Figure 1 A system 100 configured to implement one or more aspects of various embodiments is shown. As illustrated, in one embodiment, system 100 includes a training engine 110 and a neural network 120.

[0020] In one embodiment, training engine 110 is configured to train neural network 120 based on weakly labeled training data 112 to generate translated images 122(0) and 122(1). The weakly labeled training data 112 includes images 114(0) and 114(1). Image 114(0) includes a feature of interest, and image 114(1) does not include a feature of interest. Images 114(0) and 114(1) are labeled to indicate the presence or absence of a feature of interest. Image 114(0) is labeled “present” (P) to indicate that the feature is present in image 114(0). Image 114(1) is labeled “absent” (A) to indicate that the feature is absent in image 114(1). A feature of interest may be referred to herein as a “unique feature” because the feature appears only in image 114(0). Images 114(0) and 114(1) share certain features. These features are referred to herein as “common features”.

[0021] In one embodiment, in Figure 1 In the example shown, images 114(0) and 114(1) are cross-sectional images of the brain. Both images 114(0) and 114(1) include features commonly associated with the brain, including, for example, cerebral hemispheres, fissures, and gyri. These features (indicated by “C”) are common to both images 114(0) and 114(1). Additionally, image 114(0) includes abnormalities such as brain tumors. This feature (indicated by “U”) appears uniquely in image 114(0). Although for clarity... Figure 1 Common and unique features are shown in the training data 112, but these features are not explicitly labeled in the weakly labeled training data 112. As mentioned above, the weakly labeled training data 112 indicates whether unique features exist in images 114(0) and 114(1).

[0022] In one embodiment, training engine 110 trains neural network 120 based on weakly labeled training data 112 to encode images 114(0) and 114(1) into a latent space. The latent space separates unique features from common features. Training engine 110 also trains neural network 120 to decode common features from the latent space to generate a translated image 122(0) lacking unique features. Training engine 110 further trains neural network 122(1) to decode unique features from the latent space (combined with common features) to generate a translated image 122(1) lacking common features. In some embodiments, the translated image 122(1) may be represented as an image difference.

[0023] In one embodiment, during training, translated images 122(0) and 122(1) can be combined to reconstruct images 114(0) and 114(1). Training engine 114 can then perform an autoencoding process based on those reconstructed images to train neural network 120. Once neural network 120 is trained in this way, a modified version of neural network 120 can generate segmentation masks instead of translated images. The segmentation masks indicate regions in the image containing unique features. The following will also combine... Figures 2-6 The above technologies will be described in more detail.

[0024] Translation Model

[0025] Figure 2 According to various embodiments Figure 1 A more detailed illustration of the neural network is shown below. As illustrated, in one embodiment, the neural network 120 includes an encoder 200, a common decoder 210, and a residual decoder 220.

[0026] In one embodiment, encoder 200 is a multi-layer convolutional neural network that encodes input images into a latent space. The latent space separates common and unique features of these input images. Common decoder 210 is a multi-layer deconvolutional neural network that decodes the common features from the latent space to generate a translated version of the input image that includes the common features but not the unique features. Residual decoder 220 is a multi-layer deconvolutional neural network that decodes the unique features from the latent space based on the common features to generate a translated image that includes the unique features but not the common features. These translated images can be combined to reconstruct the input image and trained via autoencoding.

[0027] In one embodiment, during operation, encoder 200 receives input image X. P It is marked to indicate the presence of one or more unique features. X P It can be included in, for example Figure 1 In image 114(0). Encoder 200 will X P Encoding into the latent space to separate common features from unique features, thereby generating common features C. p and unique features U p The common decoder 210 pairs common features C p Decode to generate X PA X PA It is X P The translated version no longer contains unique features. The residual decoder 220 is based on the common feature C. p Unique feature U p Decode to generate Δ PA Δ PA It is XP The translated version no longer includes common features. Δ PA Represented as a residual between images, and may include pixel values, which, when compared with X... PA When used in combination, it produces a translated image X. PP During training, the encoder 200, common decoder 210, and residual decoder 220 can be modified to reduce X. P and X PP The differences between them.

[0028] In one embodiment, Figure 1 The training engine 110 includes a generative adversarial network (GAN), which identifies X during training. P and X PP And modify at least one of encoder 200, common decoder 210 and residual decoder 220 to reduce X. P and X PP The differences between them. Those familiar with autoencoder training will understand how to implement GANs to improve upon X. P To X PP Image translation. Other training techniques can also be implemented to further train the individual elements of neural network 120, as shown below. Figure 3 To describe in more detail.

[0029] Figure 3 This illustrates how to train according to various embodiments. Figure 1 The neural network 120 is configured to operate for at least two cycles, as shown in the figure. The neural network 120 can be configured in this manner during training to improve the translation capabilities of the encoder 200, common decoder 210, and residual decoder 220.

[0030] In one embodiment, the neural network 120 operates within a first cycle 300 to process an input image X that does not include one or more unique features. A Translated into a translated image X that includes unique features. AP Specifically, encoder 200 will input image X P Encoding into the latent space to generate common features C A The unique features do not exist in the input image X. A Therefore, encoder 200 will not generate unique features U in the latent space. A The common decoder 210 pairs common features C A Decode to generate X AA X AA It is X A The translated version, with X ASimilarly, it does not contain unique features. The residual decoder 220 obtains random samples in the latent space corresponding to unique features, and then generates Δ. AP To reflect the common features C generated by these random samples through encoder 200 A The residual between them. X AA and Δ AP Combined to generate translated image X AP Thus, cycle 300 is completed.

[0031] Subsequently, the neural network 120 operates within the second cycle 310 to translate the image X. AP The translated image X no longer includes the unique features introduced through cycle 300. APA Specifically, encoder 200 will render image X... AP Encoding into the latent space to generate common features C AP Encoder 200 can also generate unique features to be input into residual decoder 220, as described above in conjunction with cycle 300, but these unique features can be ignored in cycle 310. Then, common decoder 210 processes the common features C... AP Decode to generate translated image X APA Translation image X APA It does not contain unique features, therefore it can be compared with X A Comparisons are made to drive training through an autoencoding process. Specifically, encoder 200, common decoder 210, and residual decoder 220 can be trained to reduce X. A and X APA The difference between them. This specific training method enables the encoder 200, common decoder 210 and residual decoder 220 to translate images with higher accuracy in the presence of noise, thus making the neural network 120 more robust to noise.

[0032] In one embodiment, combined Figures 2-3 The various techniques described can be applied to translating input image X. P To generate translated images X that can be used for training purposes. PAP For example, combining Figure 2 The described technique can be applied to translating input image X. P To generate the translated image X PA As mentioned before. Then, X can be... PA Input to Figure 3 The period is 300, thus generating X. PAP Through this process, X will be... P The unique features are removed and then reintroduced. Encoder 200, common decoder 210, and residual decoder 220 can be trained to reduce X. P and XPAP The differences between them.

[0033] In one embodiment, once the neural network 120 has been trained in the manner described above, various modifications can be applied to the neural network 120 to facilitate the generation of the segmentation mask. The following section combines... Figure 4 These modifications will be described in more detail.

[0034] Generate segmentation mask

[0035] Figure 4 Various embodiments are shown. Figure 1 How a neural network generates a segmentation mask. As shown in the figure, in one embodiment, neural network 120 includes a multilayer perceptron (MLP) 400 coupled between encoder 200 and residual decoder 220. MLP 400 includes two or more layers and is configured to perform scaling and shifting operations as decoder 220 decodes common and unique features from the latent space.

[0036] In one embodiment, the MLP 400 can be incorporated into the neural network 120 to implement a method known in the art as "adaptive instance normalization." When implementing adaptive instance normalization, the MLP 400 adjusts one or more normalization parameters associated with the residual decoder 220 so that the encoder 200 generates a segmentation mask instead of a residual. These normalization parameters may include, for example, one or more scaling and shifting parameters.

[0037] In one embodiment, using the example input shown, encoder 200 will input image X P Encoding into the latent space to generate unique features U P and common features C P Based on unique features U P and common features C P The MLP 400 utilizes the residual decoder 220 to perform adaptive instance normalization, enabling the residual decoder 220 to generate a segmentation mask 410. The segmentation mask 410 indicates the input image X. P The specific region where the unique characteristics of the Chinese are located.

[0038] In one embodiment, as in the following combination Figure 5 In more detail, neural network 120 can be enhanced to recover spatial details associated with the input image by including long-skip connections.

[0039] Restoring spatial details

[0040] Figure 5 Various embodiments are shown. Figure 1How a neural network enhances the spatial details of a translated image. As shown in the figure, in one embodiment, the neural network 120 includes a long hop connection 500(0) between the encoder 200 and the common decoder 210 and a long hop connection 500(1) between the encoder 200 and the residual decoder 220. The long hop connections 500(0) and 500(1) typically help the common decoder 210 and the residual decoder 220 recover and / or enhance spatial details when translating an image, respectively.

[0041] In one embodiment, long skip connections 500(0) and 500(1) perform convolution operations on common and unique features received from encoder 200 to generate compressed versions of these features. For example, a given long skip connection 500 can compress a 64x64 feature map to generate an 8x8 feature map. Long skip connections 500(0) and 500(1) then concatenate one or more of the resulting compressed features and feed the concatenated compressed features forward to common decoder 210 and / or residual decoder 220. Common decoder 210 and residual decoder 220 can then generate the translated image and / or residual with higher spatial precision.

[0042] Overall reference Figures 1-5 In one embodiment, Figure 1 The training engine 110 can use any technically feasible training method to train the encoder 200, common decoder 210, and residual decoder 220, as previously mentioned, including those methods typically associated with autoencoders. For example, the training engine 110 can use the encoder 200, common decoder 210, and residual decoder 220 to implement backpropagation to reduce the error between the input image and the translated image and / or the combined image.

[0043] In one embodiment, during training, training engine 110 implements one or more objective functions listed in Table 1 below:

[0044]

[0045] In one embodiment, the training engine 110 trains a common decoder 210 and a residual decoder 220 based on objective function 1 to improve image reconstruction. Objective function 1 includes various loss terms, which can be evaluated based on specific input and translated images. (The above is combined with...) Figures 2-3 The input and translated images were discussed.

[0046] In one embodiment, training engine 110 trains encoder 200 based on objective function 2 to improve latent space reconstruction. Objective function 2 includes various loss terms, which can be evaluated using specific common and unique features. (The above is combined with...) Figures 2-3Some of the common and unique features were discussed. Other common and unique features proposed in objective function 2 can be combined by encoder 200 based on the above. Figures 2-3 The specific reconstructed image discussed is used to generate it. For example, C PA It can be based on encoder 200 Figure 2 X PA Generate. Similarly, C AA It can be based on encoder 200 Figure 3 X AA Generate. C PP and U PP It can be based on encoder 200 Figure 2 X PP Generate. U AP It can be based on encoder 200 Figure 2 X AP generate.

[0047] In one embodiment, the training engine 110 includes the combination described above. Figure 2 The GAN mentioned above is trained to identify X. P and X AP and X A and X PA The training engine 110 can improve the discriminative ability of GANs based on objective function 3.

[0048] In one embodiment, training engine 110 trains MLP 400 based on objective function 4 to perform the above combination. Figure 4 An adaptive instance normalization method is proposed. The objective function 4 is evaluated to determine the Dice coefficients of the segmentation mask 410.

[0049] Segmentation through image-to-image translation

[0050] Figure 6 This is a flowchart of method steps for translating an image to generate a segmentation mask according to various embodiments. Although combined... Figures 1-5 The system describes the method steps, but those skilled in the art will understand that any system configured to execute the method steps in any order falls within the scope of this embodiment.

[0051] As shown in the figure, method 600 begins at step 602, where Figure 2 The encoder 200 encodes the input image into a latent space to separate common and unique features that may appear in the input image. In one embodiment, Figure 1The training engine 110 trains the encoder 200 based on weakly labeled training data 112. The weakly labeled training data 112 includes images 114(0) and 114(1). Some features appear only in image 114(0), while other features appear in both images 114(0) and 114(1). In one embodiment, the encoder 200 is a multilayer convolutional neural network.

[0052] In step 604, Figure 2 The public decoder 210 decodes common features from the latent space to generate a translated image lacking unique features. In one embodiment, Figure 1 The training engine 110 trains a common decoder 210 based on weakly labeled training data 112, and trains an encoder 200 based on the same training data. In one embodiment, the common decoder 210 is a multi-layer deconvolutional neural network. In another embodiment, the common decoder 210 receives one or more long hop connections from the encoder 200, as described above. Figure 5 The discussion continues. In some embodiments, during the training operation, method 600 proceeds to step 606. In some embodiments, during the inference operation, method 600 proceeds to step 608. In various embodiments, one or both of steps 606 and 608 may be performed, depending on, for example, one or more loss functions implemented.

[0053] In step 606, Figure 2 The residual decoder 220 decodes unique features from the latent space to generate a residual image difference. The residual image difference may include pixel values, which, when combined with the translation image generated via step 604, produce a translation image. The translation image may be a reconstructed version of the input image. During training, the residual decoder 220 can be modified to reduce the difference between the input image and the reconstructed image. In one embodiment, the residual decoder 220 is a multi-layer deconvolutional neural network. In another embodiment, the residual decoder 220 receives one or more long-span connections from the encoder 200, as described above. Figure 5 The topic under discussion. Step 606 is typically performed during training.

[0054] In step 608, the neural network 120 configures the residual decoder 220 to generate a segmentation mask indicating the location of unique features in the input image. In one embodiment, the neural network 120 implements a multilayer perceptron (MLP) to perform adaptive instance normalization using the residual decoder 220. For example, the neural network 120 may implement... Figure 4 The MLP 400 is used to perform scaling and shifting operations using common and unique features encoded into the latent space. Step 608 is typically performed during inference operations once training is complete.

[0055] Example hardware architecture

[0056] Figure 7 This is a block diagram illustrating a computer system 700 configured to implement one or more aspects of various embodiments. In some embodiments, the computer system 700 is a server operating in a data center or cloud computing environment that provides scalable computing resources as a service over a network.

[0057] In various embodiments, the computer system 700 includes, but is not limited to, a central processing unit (CPU) 702 and a system memory 704 coupled to a parallel processing subsystem 712 via a memory bridge 705 and a communication path 713. The memory bridge 705 is also coupled to an I / O (input / output) bridge 707 via a communication path 706, and the I / O bridge 707 is in turn coupled to a switch 716.

[0058] In one embodiment, I / O bridge 707 is configured to receive user input from an optional input device 708 (such as a keyboard or mouse) and forward the input to CPU 702 for processing via communication path 706 and memory bridge 705. In some embodiments, computer system 700 may be a server in a cloud computing environment. In such embodiments, computer system 700 may not have input device 708. Instead, computer system 700 may receive equivalent input by receiving commands in the form of messages sent over a network and received via network adapter 718. In one embodiment, switch 716 is configured to provide connectivity between I / O bridge 707 and other components of computer system 700, such as network adapter 718 and various add-on cards 720 and 721.

[0059] In one embodiment, I / O bridge 707 is coupled to system disk 714, which can be configured to store content, applications, and data for use by CPU 702 and parallel processing subsystem 712. In one embodiment, system disk 714 provides non-volatile storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROMs (optical disc read-only memory), DVD-ROMs (digital versatile discs), Blu-ray, HD-DVDs (high-definition DVDs), or other magnetic, optical, or solid-state storage devices. In various embodiments, other components (such as universal serial bus or other port connections, optical disc drives, digital versatile disc drives, movie recording devices, etc.) may also be connected to I / O bridge 707.

[0060] In various embodiments, memory bridge 705 may be a northbridge chip, and I / O bridge 707 may be a southbridge chip. Furthermore, communication paths 706 and 713, as well as other communication paths within computer system 700, may be implemented using any technically suitable protocol, including but not limited to AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0061] In some embodiments, the parallel processing subsystem 712 includes a graphics subsystem that transmits pixels to an optional display device 710, which may be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, etc. In such embodiments, the parallel processing subsystem 712 includes circuitry optimized for graphics and video processing, including, for example, video output circuitry. As combined below... Figure 8 and Figure 9 In more detail, this circuitry can be combined across one or more parallel processing units (PPUs), also referred to herein as parallel processors, which are included within the parallel processing subsystem 712. In other embodiments, the parallel processing subsystem 712 includes circuitry optimized for general-purpose and / or computational processing. Again, this circuitry can be combined across one or more PPUs included within the parallel processing subsystem 712, which are configured to perform such general-purpose and / or computational operations. In other embodiments, the one or more PPUs included in the parallel processing subsystem 712 can be configured to perform graphics processing, general-purpose processing, and computational processing operations. The system memory 704 includes at least one device driver configured to manage the processing operations of one or more PPUs within the parallel processing subsystem 712.

[0062] In various embodiments, the parallel processing subsystem 712 can be coupled with... Figure 7 One or more other components are integrated together to form a single system. For example, the parallel processing subsystem 712 can be integrated with the CPU 702 and other interconnect circuitry on a single chip to form a system-on-a-chip (SoC).

[0063] In one embodiment, CPU 702 is the main processor of computer system 700, which controls and coordinates the operation of other system components. In one embodiment, CPU 702 issues commands to control the operation of PPU. In some embodiments, communication path 713 is a PCI Express link, in which a dedicated channel is allocated to each PPU, as is known in the art. Other communication paths may also be used. The PPU advantageously implements a highly parallel processing architecture. The PPU can be equipped with any number of local parallel processing memories (PP memories).

[0064] It should be understood that the system illustrated herein is illustrative and can be varied and modified. The connection topology, including the number and arrangement of bridges, the number of CPUs 702, and the number of parallel processing subsystems 712, can be modified as needed. For example, in some embodiments, system memory 704 may be connected directly to CPU 702 instead of via memory bridge 705, and other devices will communicate with system memory 704 via memory bridge 705 and CPU 702. In other embodiments, parallel processing subsystem 712 may be connected to I / O bridge 707 or directly to CPU 702 instead of via memory bridge 705. In other embodiments, I / O bridge 707 and memory bridge 705 may be integrated into a single chip rather than existing as one or more discrete devices. Finally, in some embodiments, there may be no... Figure 7 One or more components are shown. For example, switch 716 can be omitted, and network adapter 718 and add-on cards 720, 721 can be directly connected to I / O bridge 707.

[0065] Figure 8 Included according to various embodiments Figure 7 A block diagram of the parallel processing unit (PPU) 802 in the parallel processing subsystem 712. Although Figure 8 A PPU 802 is depicted, but as mentioned above, the parallel processing subsystem 712 may include any number of PPUs 802. As shown, the PPU 802 is coupled to a local parallel processing (PP) memory 804. The PPU 802 and PP memory 804 may be implemented using one or more integrated circuit devices (e.g., programmable processors, application-specific integrated circuits (ASICs), or storage devices) or in any other technically feasible manner.

[0066] In some embodiments, PPU 802 includes a graphics processing unit (GPU) configured to implement a graphics rendering pipeline to perform various operations related to generating pixel data based on graphics data provided by CPU 702 and / or system memory 704. When processing graphics data, PPU 804 can be used as graphics memory to store one or more regular frame buffers and (if needed) one or more other rendering targets. Furthermore, PPU 804 can be used to store and update pixel data and transmit the final pixel data or display frame to an optional display device 710 for display. In some embodiments, PPU 802 can also be configured for general processing and computational operations. In some embodiments, computer system 700 can be a server in a cloud computing environment. In such embodiments, computer system 700 may not have a display device 710. Instead, computer system 700 can generate equivalent output information by sending commands in the form of messages over a network via network adapter 718.

[0067] In some embodiments, CPU 702 is the main processor of computer system 700, which controls and coordinates the operation of other system components. In one embodiment, CPU 702 issues commands to control the operation of PPU 802. In some embodiments, CPU 702 writes the command stream for PPU 802 into a data structure that can be located in system memory 704, PP memory 804, or another storage location accessible to both CPU 702 and PPU 802. Figure 7 or Figure 8 (Not explicitly shown in the document). A pointer to a data structure is written to a command queue (also referred to herein as a push buffer) to initiate processing of the command stream in the data structure. In one embodiment, PPU 802 reads the command stream from the command queue and then executes the commands asynchronously relative to the operation of CPU 702. In embodiments that generate multiple push buffers, the application can control the scheduling of different push buffers by specifying an execution priority for each push buffer through a device driver.

[0068] In one embodiment, PPU 802 includes an I / O (input / output) unit 805 that communicates with the remainder of computer system 700 via communication path 713 and memory bridge 705. In one embodiment, I / O unit 805 generates packets (or other signals) for transmission on communication path 713 and also receives all incoming packets (or other signals) from communication path 713, directing the incoming packets to the appropriate components of PPU 802. For example, commands related to processing tasks may be directed to host interface 806, while commands related to memory operations (e.g., reading from or writing to PP memory 804) may be directed to crossbar switch unit 810. In one embodiment, host interface 806 reads each command queue and sends the command stream stored in the command queue to front end 812.

[0069] As described above Figure 7 The connection between the PPU 802 and the rest of the computer system 700 can be varied. In some embodiments, the parallel processing subsystem 712, including at least one PPU 802, is implemented as an add-in card that can be inserted into an expansion slot of the computer system 700. In other embodiments, the PPU 802 can be integrated on a single chip with a bus bridge, such as a memory bridge 705 or an I / O bridge 707. Again, in other embodiments, some or all of the components of the PPU 802 can be included together with the CPU 702 in a single integrated circuit or system-on-a-chip (SoC).

[0070] In one embodiment, the front end 812 sends the processing tasks received from the host interface 806 to a work distribution unit (not shown) within the task / work unit 807. In one embodiment, the work distribution unit receives a pointer to the processing task, which is encoded as task metadata (TMD) and stored in the memory. The pointer to the TMD is included in a command stream, which is stored as a command queue and received by the front end unit 812 from the host interface 806. Processing tasks that can be encoded as TMDs include an index associated with the data to be processed and status parameters and commands that define how the data is to be processed. For example, the status parameters and commands can define a program to be executed on the data. Also for example, the TMD can specify the number and configuration of a set of CTAs. Generally, each TMD corresponds to one task. The task / work unit 807 receives tasks from the front end 812 and ensures that the GPC 808 is configured in a valid state before starting the processing tasks specified by each TMD. A priority can be specified for each TMD, which is used to schedule the execution of the processing tasks. Processing tasks can also be received from the processing cluster array 830. Optionally, the TMD can include a parameter that controls whether the TMD is added to the head or tail of the processing task list (or added to a list of pointers to processing tasks), thereby providing another level of control over the execution priority.

[0071] In one embodiment, the PPU 802 implements a highly parallel processing architecture based on the processing cluster array 830, which includes a set of C general processing clusters (GPCs) 808, where C ≥ 1. Each GPC 808 is capable of concurrently executing a large number (e.g., hundreds or thousands) of threads, where each thread is an instance of a program. In various applications, different GPCs 808 can be assigned to process different types of programs or to perform different types of computations. The assignment of GPCs 808 can vary depending on the workload generated by each type of program or computation.

[0072] In one embodiment, the memory interface 814 includes a set of D partition units 815, where D ≥ 1. Each partition unit 815 is coupled to one or more dynamic random access memories (DRAMs) 820 residing in the PPM memory 804. In some embodiments, the number of partition units 815 is equal to the number of DRAMs 820, and each partition unit 815 is coupled to a different DRAM 820. In other embodiments, the number of partition units 815 may differ from the number of DRAMs 820. Those skilled in the art will understand that the DRAMs 820 can be replaced by any other technically suitable storage device. In operation, various rendering targets (e.g., texture maps and framebuffers) can be stored across the DRAMs 820, allowing the partition units 815 to write portions of each rendering target in parallel to efficiently utilize the available bandwidth of the PPM memory 804.

[0073] In one embodiment, a given GPC 808 can process data to be written to any DRAM 820 in the PP memory 804. In one embodiment, a crossbar switch unit 810 is configured to route the output of each GPC 808 to the input of any partition unit 815 or any other GPC 808 for further processing. The GPC 808 communicates with the memory interface 814 via the crossbar switch unit 810 to read from or write to various DRAMs 820. In some embodiments, the crossbar switch unit 810, in addition to its connection to the PP memory 804 via the memory interface 814, also has a connection to the I / O unit 805, thereby enabling the processing cores in different GPCs 808 to communicate with the system memory 704 or other memories not native to the PPU 802. Figure 8 In some embodiments, the crossbar switch unit 810 is directly connected to the I / O unit 805. In various embodiments, the crossbar switch unit 810 may use a virtual channel to separate the traffic flow between the GPC 808 and the partition unit 815.

[0074] In one embodiment, GPC 808 can be programmed to perform processing tasks related to a wide variety of applications, including but not limited to linear and nonlinear data transformations, filtering of video and / or audio data, modeling operations (e.g., applying physical laws to determine the position, velocity, and other properties of an object), image rendering operations (e.g., tessellation shaders, vertex shaders, geometry shaders, and / or pixel / fragment shaders), general computing operations, etc. In operation, PPU 802 is configured to transfer data from system memory 704 and / or PP memory 804 to one or more on-chip memory units, process the data, and write the resulting data back to system memory 704 and / or PP memory 804. The resulting data can then be accessed by other system components, including CPU 702, another PPU 802 within parallel processing subsystem 712, or another parallel processing subsystem 712 within computer system 700.

[0075] In one embodiment, the parallel processing subsystem 712 may include any number of PPUs 802. For example, multiple PPUs 802 may be provided on a single add-on card, or multiple add-on cards may be connected to the communication path 713, or one or more PPUs 802 may be integrated into a bridge chip. The PPUs 802 in a multi-PPU system may be the same or different from each other. For example, different PPUs 802 may have different numbers of processing cores and / or different numbers of PP memories 804. In implementations with multiple PPUs 802, these PPUs can operate in parallel to process data at a higher throughput than possible with a single PPU 802. Systems containing one or more PPUs 802 can be implemented in various configurations and form factors, including but not limited to desktops, laptops, handheld PCs or other handheld devices, servers, workstations, game consoles, embedded systems, and so on.

[0076] Figure 9 Included according to various embodiments Figure 8 A block diagram of the General Processing Cluster (GPC) 808 in the Parallel Processing Unit (PPU) 802. As shown, the GPC 808 includes, but is not limited to, a pipeline manager 905, one or more texture units 915, a preROP unit 925, a work assignment crossbar switch 930, and an L1.5 cache 935.

[0077] In one embodiment, the GPC 808 can be configured to execute a large number of threads in parallel to perform graphics, general processing, and / or computational operations. As used herein, a “thread” refers to an instance of a specific program executed on a particular set of input data. In some embodiments, Single Instruction, Multiple Data (SIMD) instruction issuing techniques are used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In other embodiments, Single Instruction, Multiple Thread (SIMT) techniques are used to support the parallel execution of a large number of typically synchronous threads using a general instruction unit configured to issue instructions to a set of processing engines in the GPC 808. Unlike SIMD execution schemes, where all processing engines typically execute the same instructions, SIMT execution allows different threads to more easily follow different execution paths through a given program. Those skilled in the art will understand that SIMD processing schemes represent a subset of the functionality of SIMT processing schemes.

[0078] In one embodiment, the operation of GPC 808 is controlled via pipeline manager 905, which distributes processing tasks received from a work assignment unit (not shown) within task / work unit 807 to one or more streaming multiprocessors (SMs) 910. Pipeline manager 905 can also be configured to control work assignment crossbar switch 930 by specifying the destination of the processing data output from SM 910.

[0079] In various embodiments, the GPC 808 includes a set of M SM 910s, where M ≥ 1. Furthermore, each SM 910 includes a set of functional execution units (not shown), such as execution units and load-memory units. Processing operations specific to any functional execution unit can be piped, enabling the issuance of new instructions for execution before the previous instruction has completed. Any combination of functional execution units within a given SM 910 can be provided. In various embodiments, the functional execution units can be configured to support a variety of different operations, including integer and floating-point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, XOR), shifting, and computation of various algebraic functions (e.g., plane interpolation and trigonometric functions, exponential functions, logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations.

[0080] In various embodiments, each SM 910 includes multiple processing cores. In one embodiment, the SM 910 includes a large number (e.g., 128, etc.) of different processing cores. Each core may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, which includes floating-point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0081] In one embodiment, a tensor core is configured to perform matrix operations, and in another embodiment, one or more tensor cores are included within these cores. Specifically, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations used for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs matrix multiplication and accumulation operations D = A×B + C, where A, B, C, and D are 4×4 matrices.

[0082] In one embodiment, matrix multiplication inputs A and B are 16-bit floating-point matrices, while accumulation matrices C and D can be either 16-bit or 32-bit floating-point matrices. The Tensor Core performs 32-bit floating-point accumulation on the 16-bit floating-point input data. The 16-bit floating-point multiplication requires 64 operations and produces a full-precision product, which is then accumulated with other intermediate products using 32-bit floating-point addition for a 4×4×4 matrix multiplication. In practice, the Tensor Core is used to perform larger two-dimensional or higher-dimensional matrix operations, which are constructed from these smaller elements. APIs (such as the CUDA9 C++ API) expose specialized matrix loading, matrix multiplication and accumulation, and matrix storage operations to efficiently utilize the Tensor Core from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes 16×16 matrices across all 32 threads of the warp.

[0083] Neural networks rely heavily on matrix mathematical operations, and complex multi-layered networks require significant floating-point performance and bandwidth for both efficiency and speed. Employing thousands of processing cores optimized for matrix mathematical operations and delivering tens to hundreds of TFLOPS of performance in various embodiments, the SM 910 provides a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0084] In various embodiments, each SM 910 may further include multiple Special Function Units (SFUs) that perform special functions (e.g., attribute evaluation, inverse square root, etc.). In one embodiment, an SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, an SFU may include a texture unit configured to perform texture map filtering operations. In one embodiment, a texture unit is configured to load a texture map (e.g., a 2D array of texture pixels) from memory and sample the texture map to produce sampled texture values ​​for use in a shading procedure executed by the SM. In various embodiments, each SM 910 also includes multiple Load / Store Units (LSUs) that implement load and store operations between shared memory / L1 cache and a register file within the SM 910.

[0085] In one embodiment, each SM 910 is configured to process one or more thread groups. As used herein, a "thread group" or "thread bundle" refers to a group of threads that execute the same program simultaneously on different input data, with one thread in the group assigned to a different execution unit within the SM 910. A thread group may include fewer threads than the number of execution units in the SM 910, in which case some executions may be idle during the cycle in which the thread group is processed. A thread group may also include more threads than the number of execution units in the SM 910, in which case processing may occur in consecutive clock cycles. Since each SM 910 can support up to G thread groups simultaneously, up to G*M thread groups can be executed in the GPC 808 at any given time.

[0086] Additionally, in one embodiment, multiple related thread groups can be active simultaneously in the SM 910 (at different execution phases). This collection of thread groups is referred to herein as a “cooperative thread array” (“CTA”) or “thread array”. The size of a particular CTA is equal to m*k, where k is the number of threads executing concurrently in the thread group, which is typically an integer multiple of the number of execution units in the SM 910, and m is the number of thread groups active concurrently within the SM 910. In some embodiments, a single SM 910 can support multiple CTAs simultaneously, where these CTAs are at the granularity of assigning work to the SM 910.

[0087] In one embodiment, each SM 910 includes a Level 1 (L1) cache or uses space in a corresponding L1 cache external to the SM 910 to support load and store operations, etc., performed by the execution unit. Each SM 910 can also access a Level 2 (L2) cache (not shown) shared among all GPCs 808 in the PPU 802. The L2 cache can be used to transfer data between threads. Finally, the SM 910 can also access off-chip “global” memory, which may include PP memory 804 and / or system memory 704. It should be understood that any memory external to the PPU 802 can be used as global memory. Furthermore, as Figure 9 As shown, a Level 1.5 (L1.5) cache 935 may be included in the GPC 808 and configured to receive and store data requested from memory by the SM 910 via the memory interface 814. This data may include, but is not limited to, instructions, uniform data, and constant data. In embodiments where multiple SMs 910 are present within the GPC 808, the SMs 910 may advantageously share common instructions and data cached in the L1.5 cache 935.

[0088] In one embodiment, each GPC 808 may have an associated memory management unit (MMU) 920 configured to map virtual addresses to physical addresses. In various embodiments, the MMU 920 may reside within the GPC 808 or within the memory interface 814. The MMU 920 includes a set of page table entries (PTEs) for mapping virtual addresses to physical addresses of tiles or memory pages, and optionally, to cache line indices. The MMU 920 may include an address translation lookahead buffer (TLB) or cache that may reside within the SM 910, one or more L1 caches, or within the GPC 808.

[0089] In one embodiment, in a graphics and computing application, the GPC 808 can be configured such that each SM 910 is coupled to a texture unit 915 to perform texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data.

[0090] In one embodiment, each SM 910 sends the processed task to the work assignment crossbar switch 930 so that the processed task can be provided to another GPC 808 for further processing or stored in an L2 cache (not shown), parallel processing memory 804, or system memory 704 via the crossbar switch unit 810. Furthermore, the pre-raster operation (preROP) unit 925 is configured to receive data from the SM 910, direct the data to one or more raster operation (ROP) units within the partition unit 815, perform color blending optimization, organize pixel color data, and perform address translation.

[0091] It will be understood that the architecture described herein is illustrative and is subject to change and modification. Among other things, the GPC 808 may include any number of processing units, such as SM 910, texture units 915, or preROP units 925. Furthermore, as combined with the above... Figure 8 The PPU 802 may include any number of GPCs 808 configured to be functionally similar to each other, such that execution behavior is independent of which GPC 808 receives a specific processing task. Furthermore, each GPC 808 operates independently of the other GPCs 808 in the PPU 802 to perform tasks for one or more applications.

[0092] In summary, the neural network is trained on weakly labeled training data, which includes a first set of images with unique features and a second set of images without unique features. The weakly labeled training data includes a label for each image indicating the presence or absence of unique features. Both sets of images also include various other features common to both sets. The neural network comprises an encoder, a common decoder, and a residual decoder. The encoder encodes the input images into a latent space. The latent space separates the unique features of the input images from their common features. The common decoder decodes the common features residing in the latent space to generate a translated image lacking unique features. The residual decoder decodes the unique features residing in the latent space to generate an image increment corresponding to the unique features. The neural network combines the translated image with the image increment to generate a combined image that can contain both common and unique features. This combined image can be used to drive autoencoding. Once training is complete, the residual decoder can be modified to generate a segmentation mask instead of image increments. A given segmentation mask indicates any region of a given input image where unique features reside.

[0093] At least one technical advantage of the disclosed technique compared to existing technologies is that it can accurately segment images based solely on weakly labeled training data. Therefore, features of interest within an image can be easily identified without performing complex manual processes to generate training data. Another technical advantage of the disclosed technique compared to existing technologies is that it requires fewer reference segments to train the neural network compared to conventional methods, thereby simplifying and accelerating the training process. These technical advantages represent several technological advancements compared to existing methods.

[0094] 1. Some embodiments include a computer-implemented method comprising: generating a feature representation of an input image, wherein the input image includes a first feature associated with a first feature type and one or more additional features associated with a second feature type; decoding the feature representation using a first decoder associated with the first feature type to generate a first image, the first image including the first feature but not including one or more additional features; decoding the feature representation using a second decoder associated with the second feature type to generate a second image, the second image including one or more additional features but not including the first feature; combining the first image and the second image to generate a third image; comparing the input image and the third image to determine a difference between the input image and the third image; and updating the first decoder based on the difference between the input image and the third image to enable the first decoder to generate a segmentation mask that identifies the location of a given feature associated with the first feature type in a given image.

[0095] 2. The computer-implemented method according to Clause 1, wherein generating a feature representation of an input image comprises: encoding the input image into a latent space using a first encoder.

[0096] 3. A computer-implemented method according to any one of clauses 1-2, wherein a first feature type corresponds to a first region of the latent space and a second feature type corresponds to a second region of the latent space.

[0097] 4. The computer-implemented method according to any one of clauses 1-3 further includes: updating the first encoder based on the difference between the input image and the third image, so that the first encoder is able to encode a first feature into a first region of the latent space and encode one or more additional features into a second region of the latent space.

[0098] 5. A computer-implemented method according to any one of clauses 1-4, wherein a first encoder is coupled to a first decoder via a set of long jump connections, the long jump connections compressing and / or concatenating at least a portion of the feature representation as input to the first decoder.

[0099] 6. A computer-implemented method according to any one of clauses 1-5, wherein a first encoder is coupled to a second decoder via a set of long jump connections, the long jump connections compressing and / or concatenating at least a portion of the feature representation as input to the second decoder.

[0100] 7. The computer-implemented method according to any one of clauses 1-6 further comprises: updating the feature representation based on a translated image, wherein the translated image has been modified to remove the first feature; decoding the feature representation using a second decoder to generate a fourth image, the fourth image including one or more additional features but not including the first feature; and updating the second decoder based on the difference between the translated image and the fourth image.

[0101] 8. A computer-implemented method according to any one of clauses 1-7 further includes generating a segmentation mask based at least in part on one or more scaling and shifting parameters.

[0102] 9. A computer-implemented method according to any one of clauses 1-8, wherein updating the first decoder based on the difference between the input image and the third image comprises: evaluating a first objective function, the first objective function indicating a first loss value associated with the input image and the third image.

[0103] 10. A computer-implemented method according to any one of clauses 1-9, wherein updating the first decoder based on the difference between the input image and the third image comprises: evaluating a second objective function, the second objective function indicating a first loss value associated with the first feature and a version of the first feature translated to remove the first feature.

[0104] 11. Some embodiments include a non-transitory computer-readable medium storing program instructions that, when executed by at least one processor, cause at least one processor to at least: generate a feature representation of an input image, wherein the input image includes a first feature associated with a first feature type and one or more additional features associated with a second feature type; decode the feature representation using a first decoder associated with the first feature type to generate a first image, the first image including the first feature but not including one or more additional features; decode the feature representation using a second decoder associated with the second feature type to generate a second image, the second image including one or more additional features but not including the first feature; combine the first image and the second image to generate a third image; compare the input image and the third image to determine a difference between the input image and the third image; and update the first decoder based on the difference between the input image and the third image to enable the first decoder to generate a segmentation mask that identifies the location of a given feature associated with the first feature type in a given image.

[0105] 12. A non-transitory computer-readable medium pursuant to Clause 11, wherein at least one processor generates a feature representation of an input image by encoding the input image into a latent space using a first encoder.

[0106] 13. A non-transitory computer-readable medium according to any one of clauses 11-12, wherein a first feature type corresponds to a first region of a latent space, a second feature type corresponds to a second region of a latent space, and wherein at least one processor updates a first encoder based on a difference between an input image and a third image, such that the first encoder is able to encode a first feature into the first region of the latent space and encode one or more additional features into the second region of the latent space.

[0107] 14. A non-transitory computer-readable medium according to any one of clauses 11-13, wherein a first encoder is coupled to a first decoder via a set of long jump connections, the long jump connections compressing and concatenating one or more portions of a feature representation for input to the first decoder.

[0108] 15. A non-transitory computer-readable medium according to any one of clauses 11-14, wherein a first encoder is coupled to a second decoder via a set of long jump connections, the long jump connections compressing and concatenating one or more portions of a feature representation for input to the second decoder.

[0109] 16. A non-transitory computer-readable medium according to any one of clauses 11-15, wherein at least one processor trains a decoder by the following steps: updating a feature representation based on a translated image excluding the first feature, wherein the translated image includes a modified version of a training image that includes the first feature; decoding the feature representation using a second decoder to generate a fourth image that includes one or more additional features but excludes the first feature; and updating the second decoder based on the difference between the translated image and the fourth image.

[0110] 17. The non-transitory computer-readable medium pursuant to any one of clauses 11-16 also includes generating the segmentation mask based at least in part on one or more scaling and shifting parameters.

[0111] 18. Some embodiments include a system comprising: a memory storing one or more instructions; and a processor executing the instructions to at least: generate a feature representation of an input image, wherein the input image includes a first feature associated with a first feature type and one or more additional features associated with a second feature type; decode the feature representation using a first decoder associated with the first feature type to generate a first image, the first image including the first feature but not including one or more additional features; decode the feature representation using a second decoder associated with the second feature type to generate a second image, the second image including one or more additional features but not including the first feature; combine the first image and the second image to generate a third image; compare the input image and the third image to determine a difference between the input image and the third image; and update the first decoder based on the difference between the input image and the third image to enable the first decoder to generate a segmentation mask that identifies the location of a given feature associated with the first feature type in a given image.

[0112] 18. Some embodiments include a system comprising: a memory storing one or more instructions; and a processor executing the instructions to at least: generate a feature representation of an input image, wherein the input image includes a first feature associated with a first feature type and one or more additional features associated with a second feature type; decode the feature representation using a first decoder associated with the first feature type to generate a first image, the first image including the first feature but not including one or more additional features; decode the feature representation using a second decoder associated with the second feature type to generate a second image, the second image including one or more additional features but not including the first feature; combine the first image and the second image to generate a third image; compare the input image and the third image to determine a difference between the input image and the third image; and update the first decoder based on the difference between the input image and the third image to enable the first decoder to generate a segmentation mask that identifies the location of a given feature associated with the first feature type in a given image.

[0113] 20. A system according to any one of clauses 18-19, wherein the processor generates a feature representation of an input image by: analyzing a label associated with the input image, the label indicating the presence of a first feature in the input image; and updating the encoder that generates the feature representation based on the label.

[0114] In any way, any claim element recited in any claim and / or any combination of any element described in this application falls within the scope of this disclosure and protection.

[0115] Various embodiments have been described for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0116] Aspects of this embodiment may be embodied as a system, method, or computer program product. Therefore, aspects of this disclosure may take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects generally referred to herein as a “module” or “system.” Furthermore, aspects of this disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0117] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include the following: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any other suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0118] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. When executed by the processor of the computer or other programmable data processing apparatus, the instructions enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor, or a field-programmable gate array (FPGA).

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a non-consecutive order. For example, depending on the functions involved, two consecutively shown blocks may be executed substantially simultaneously, or sometimes in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified functions or actions.

[0120] Although the foregoing embodiments are directed to this disclosure, other and further embodiments of this disclosure may be devised without departing from the basic scope of this disclosure, the scope of which is defined by the appended claims.

Claims

1. A computer-implemented method, comprising: An encoder is used to encode an input image into a latent space to generate a feature representation of the input image, wherein the input image includes a first feature associated with a first feature type and one or more additional features associated with a second feature type, the first feature type being a common feature type and the second feature type being a unique feature type, and the latent space separating the first feature from the one or more additional features; The feature representation is decoded using a first decoder associated with the first feature type to generate a first image, which includes the first feature but does not include the one or more additional features; The feature representation is decoded using a second decoder associated with the second feature type to generate a second image, which includes one or more additional features but does not include the first feature; The first image and the second image are combined to generate a third image; Compare the input image and the third image to determine the differences between the input image and the third image; as well as The first decoder is trained based on the difference between the input image and the third image, so that the first decoder can generate a segmentation mask that identifies the location of a given feature in a given image that is associated with the first feature type.

2. The computer-implemented method according to claim 1, wherein the first feature type corresponds to a first region of the latent space, and the second feature type corresponds to a second region of the latent space.

3. The computer-implemented method of claim 1 further includes updating the encoder based on the difference between the input image and the third image, so that the encoder can encode the first feature into a first region of the latent space and encode the one or more additional features into a second region of the latent space.

4. The computer-implemented method of claim 1, wherein the encoder is coupled to the first decoder via a set of long jump connections, the long jump connections compressing and / or concatenating at least a portion of the feature representation for input to the first decoder.

5. The computer-implemented method of claim 1, wherein the encoder is coupled to the second decoder via a set of long jump connections, the long jump connections compressing and / or concatenating at least a portion of the feature representation for input to the second decoder.

6. The computer-implemented method according to claim 1, further comprising: A translated image is generated based at least in part on a modified input image, and the feature representation is updated based at least in part on the translated image, wherein the modified input image does not include the first feature; The feature representation is decoded using the second decoder to generate a fourth image, the fourth image including one or more additional features but excluding the first feature; as well as The second decoder is updated based on the difference between the modified input image and the fourth image.

7. The computer-implemented method of claim 1, further comprising generating the segmentation mask based at least in part on one or more scaling parameters and shift parameters.

8. The computer-implemented method of claim 1, wherein updating the first decoder based on the difference between the input image and the third image comprises: The first objective function is evaluated, which indicates a first loss value associated with the input image and the third image.

9. The computer-implemented method of claim 1, wherein updating the first decoder based on the difference between the input image and the third image comprises: The second objective function is evaluated, which indicates the first loss value associated with the first feature and the image version translated to remove the first feature.

10. A non-transitory computer-readable medium storing program instructions, said program instructions, when executed by at least one processor, cause said at least one processor to at least: An encoder is used to encode an input image into a latent space to generate a feature representation of the input image, wherein the input image includes a first feature associated with a first feature type and one or more additional features associated with a second feature type, the first feature type being a common feature type and the second feature type being a unique feature type, and the latent space separating the first feature from the one or more additional features; The feature representation is decoded using a first decoder associated with the first feature type to generate a first image, which includes the first feature but does not include the one or more additional features; The feature representation is decoded using a second decoder associated with the second feature type to generate a second image, which includes one or more additional features but does not include the first feature; The first image and the second image are combined to generate a third image; Compare the input image and the third image to determine the differences between the input image and the third image; as well as The first decoder is trained based on the difference between the input image and the third image, so that the first decoder can generate a segmentation mask that identifies the location of a given feature in a given image that is associated with the first feature type.

11. The non-transitory computer-readable medium of claim 10, wherein the first feature type corresponds to a first region of the latent space, the second feature type corresponds to a second region of the latent space, and wherein the at least one processor updates the encoder based on the difference between the input image and the third image, such that the encoder is able to encode the first feature into the first region of the latent space and encode the one or more additional features into the second region of the latent space.

12. The non-transitory computer-readable medium of claim 10, wherein the encoder is coupled to the first decoder via a set of long jump connections, the long jump connections compressing and concatenating one or more portions of the feature representation for input to the first decoder.

13. The non-transitory computer-readable medium of claim 10, wherein the encoder is coupled to the second decoder via a set of long jump connections, the long jump connections compressing and concatenating one or more portions of the feature representation for input to the second decoder.

14. The non-transitory computer-readable medium of claim 10, wherein the at least one processor trains the decoder by means of the following steps: A translated image is generated based at least in part on a modified input image that does not include the first feature, and the feature representation is updated at least in part based on the translated image, wherein the translated image includes the first feature; The feature representation is decoded using the second decoder to generate a fourth image, which includes one or more additional features but does not include the first feature; as well as The second decoder is updated based on the difference between the modified input image and the fourth image.

15. The non-transitory computer-readable medium of claim 10, further comprising generating the segmentation mask based at least in part on one or more scaling and shifting parameters.

16. The non-transitory computer-readable medium of claim 10, wherein the feature representation for generating the input image comprises: Analyze the labels associated with the input image, the labels indicating the presence of the first feature in the input image.

17. A system comprising: A memory that stores one or more instructions; as well as Processor, the processor executes the instructions to at least: An encoder is used to encode an input image into a latent space to generate a feature representation of the input image, wherein the input image includes a first feature associated with a first feature type and one or more additional features associated with a second feature type, the first feature type being a common feature type and the second feature type being a unique feature type, and the latent space separating the first feature from the one or more additional features; The feature representation is decoded using a first decoder associated with the first feature type to generate a first image, which includes the first feature but does not include the one or more additional features; The feature representation is decoded using a second decoder associated with the second feature type to generate a second image, which includes one or more additional features but does not include the first feature; The first image and the second image are combined to generate a third image; Compare the input image and the third image to determine the differences between the input image and the third image; as well as The first decoder is trained based on the difference between the input image and the third image, so that the first decoder can generate a segmentation mask that identifies the location of a given feature in a given image that is associated with the first feature type.

18. The system of claim 17, wherein the processor generates the feature representation of the input image by means of the following steps: Analyze the labels associated with the input image, the labels indicating the presence of the first feature in the input image; and Update the encoder that generates the feature representation based on the label.

Images