Multi-exit visual synthesis network based on dynamic plaque calculation
By adopting dynamic plaque calculation and plaque-level sparse convolution in the visual synthesis network, the limited hardware acceleration problem caused by pixel-level sparse convolution in the prior art is solved, and efficient image super-resolution processing is achieved, which is suitable for different Capacity platform.
Patent Information
- Application Number
- CN202280094499.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-06
- Publication Date
- 2024-11-26
AI Technical Summary
Existing scalable visual synthesis networks rely on pixel-level sparse convolution, resulting in limited acceleration on hardware and difficulty in realizing practical scalable solutions, especially in image super-resolution processing.
Using a multi-exit visual synthesis network based on dynamic plaque calculation, the number of layers is adaptively adjusted by segmenting the input image into multiple plaques and performing the synthesis process on each plaque. Accelerate and optimize processing flow through early exit mechanisms.
It significantly reduces the calculation cost, achieves the actual acceleration effect close to theoretical acceleration, improves the efficiency and accuracy of image super-resolution processing, and is suitable for platforms with different capacity.
Smart Images

Figure CN119032382A_ABST
Abstract
Description
Technical Field
[0001] Embodiments described herein relate generally to vision processing, and more specifically to a multi-outlet vision synthesis network based on dynamic patch computing (DPC). Background Art
[0002] As the future of computing is heterogeneous, scalability is a very important issue for visual synthesis, such as image super-resolution (SR) on general-purpose processors like Graphic Processing Units (GPUs). Recent works attempt to train scalable networks that can be deployed on platforms with different capacities. However, scalable networks may rely on pixel-wise sparse convolutions, which are hardware-unfriendly and achieve limited practical speedups. Thus, designing practical scalable solutions for visual synthesis remains to be explored. BRIEF DESCRIPTION OF THE DRAWINGS
[0003] Various advantages of the embodiments will become apparent to those skilled in the art by reading the following description and appended claims, and by referring to the following drawings, in which:
[0004] Figure 1 The actual speedup comparison of image SR networks based on unstructured pixel-wise sparsity and patch-wise sparsity with the same sparsity ratio is shown.
[0005] Figure 2 An example pipeline of a traditional image SR network is shown;
[0006] Figure 3 An example pipeline of an image SR network based on dynamic patch computation (DPC) according to some embodiments of the present disclosure is illustrated;
[0007] Figure 4 illustrates visualization of early exit patches of an image during processing by a DPC-based image SR network according to some embodiments of the present disclosure;
[0008] Figure 5a and Figure 5b illustrates quantitative results of the accuracy-efficiency tradeoff obtained by an example DPC-based multi-exit image SR network according to some embodiments of the present disclosure;
[0009] Figure 6 The figure illustrates a performance comparison between a traditional image SR network, an existing scalable image SR network, and a DPC-based image SR network according to some embodiments of the present disclosure;
[0010] Figure 7 An example process of visual synthesis of a multi-exit visual synthesis network (VSN) based on DPC according to some embodiments of the present disclosure is illustrated;
[0011] Figure 8 The block diagram illustrates components capable of reading instructions from a machine-readable or computer-readable medium and performing any one or more of the methods discussed herein, according to some example embodiments;
[0012] Fig. 9 is a block diagram of an example processor platform according to some embodiments of the present disclosure. DETAILED DESCRIPTION
[0013] The various aspects of the illustrative embodiments will be described using terms that are commonly used by those skilled in the art to convey the essence of the present disclosure to other persons skilled in the art. However, it will be appreciated by those skilled in the art that many alternative embodiments may also be implemented using some parts of the described aspects. For illustration, specific numbers, materials, and configurations are recorded to provide a thorough understanding of the illustrative embodiments. However, it will be clear to those skilled in the art that alternative embodiments may also be implemented without these specific details. In other cases, known features may be omitted or simplified to avoid blurring the illustrative embodiments.
[0014] In addition, various operations will be described as multiple discrete operations in turn in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as meaning that these operations are necessarily order-dependent. In particular, these operations do not need to be performed in the order presented.
[0015] As the future of computing is heterogeneous, scalability is a very important issue for visual synthesis, such as image SR on general-purpose processors like GPUs. Recent work attempts to train scalable networks that can be deployed on platforms with different capacities. However, scalable networks may rely on pixel-level sparse convolutions, which are hardware-unfriendly and achieve limited practical speedups. Thus, designing practical scalable solutions for visual synthesis remains to be explored.
[0016] In the present disclosure, a practical scalable multi-exit visual synthesis network (VSN) based on image patches is proposed to solve the scalability problem while considering the trade-off between accuracy and efficiency. Using the proposed multi-exit VSN, the input image can be first segmented into multiple patches, and then for each patch, a synthesis process based on a dynamic patch computation (DPC) scheme can be performed on the patch to obtain a processed patch (also called a final synthesized patch), and finally all processed patches can be merged to generate an output image. The DPC scheme can be based on the classical concept of early exit of deep learning neural networks. However, it is noted that although there are many solutions to apply early exit to visual understanding tasks, the DPC scheme is completely different from early exit in visual understanding tasks such as image classification. For visual understanding tasks, the input image is processed uniformly, while the proposed multi-exit VSN adaptively handles different patches in the input image.
[0017] In addition, according to the proposed multi-outlet VSN, patch-level sparse convolution can be applied to improve efficiency during inference. In other words, the visual synthesis network can be trained based on the patch-level sparse pattern, and then the synthesis process can be performed by the visual synthesis network based on the patch-level sparse convolution corresponding to the patch-level sparse pattern.
[0018] Figure 1 The figure shows the actual speedup comparison of image SR networks based on unstructured pixel-level sparsity and patch-level sparsity with the same sparsity rate (also referred to as sparsity in this article). Figure 1 The comparison results in can be obtained based on the experimental data of the same image SR network based on unstructured pixel-level sparsity and patch-level sparsity. Figure 1 As shown, by using patch-level sparsity instead of pixel-level sparsity, image SR networks can achieve actual speedup that is closer to the theoretical speedup.
[0019] According to the present disclosure, during inference using the proposed DPC-based multi-exit VSN, the number of executed layers can be adaptively adjusted for each patch of the input image instead of the entire input image. Therefore, the proposed DPC-based multi-exit VSN can achieve actual speedup close to the theoretical speedup.
[0020] In the present disclosure, a multi-exit VSN may include an image SR network, an image denoising network, an image deblurring network, etc. Since there is no downsampling in a multi-exit VSN, a shared upsampler may be used in the VSN to obtain processed patches at different exits. For illustration purposes only, the pipeline of the VSN is described in detail using an image SR network as an example of the VSN.
[0021] Figure 2 The figure shows an example pipeline of a traditional image SR network. Figure 2 As shown in Figure 1, traditional image SR networks, such as the Enhanced Deep Residual Network for Single Image Super-Resolution (EDSR) or the Residual Channel Attention Network (RCAN), have a concise topological structure consisting of three stages: head, body, and tail. The head stage can convert the input low-resolution (LR) image into LR features, and the body stage can learn the end-to-end mapping of LR features to high-resolution (HR) features. Finally, the tail stage can convert the HR features into the output SR image. Among the three stages, the body stage is the most time-consuming stage because it consists of several cascaded layers.
[0022] As described above, according to the proposed DPC-based multi-outlet VSN, the number of layers executed during inference can be adaptively adjusted for each patch of the input image. Figure 3 An example pipeline of a DPC-based image SR network according to some embodiments of the present disclosure is illustrated. The image SR network may be a multi-exit SR network, which means that the SR network may include several exit layers, at which the reasoning process may exit, and the reasoning results obtained by using the executed layers may be output as the final reasoning results.
[0023] like Figure 3 As shown, the input LR image can be first segmented into multiple LR patches. For each LR patch, the SR processing can be performed on the LR patch using the first layer to the i-th outlet layer of the multi-outlet SR network to obtain the i-th intermediate patch with improved features relative to the LR patch, where i is the outlet index, which is between 1 and the number of outlets in the multi-outlet SR network.
[0024] According to some embodiments of the present disclosure, a regressor may be applied to predict the incremental improvement of the (i+1)th intermediate patch relative to the i-th intermediate patch based on the features in the i-th intermediate patch. In the present disclosure, the (i+1)th intermediate patch may indicate an intermediate patch that may be obtained by performing an SR process on the LR patch using the first layer to the (i+1)th exit layer of a multi-exit SR network, and the incremental improvement of the (i+1)th intermediate patch relative to the i-th intermediate patch may indicate the improvement of the HR features in the (i+1)th intermediate patch relative to the HR features in the i-th intermediate patch.
[0025] When the incremental improvement predicted at the i-th exit layer is lower than a predetermined threshold, the SR process of the LR patch can exit from the i-th exit layer, the i-th exit can be determined as the final exit of the LR patch, and the i-th intermediate patch can be determined as the final SR patch of the LR patch; otherwise, i can be incremented by 1, and the SR process can continue to the (i+1)th exit layer, and the incremental improvement can be further predicted at the (i+1)th exit layer until the incremental improvement is lower than the predetermined threshold or the SR process has traversed all layers of the VSN. It is easy to understand that the predetermined threshold can be adjusted based on the trade-off between accuracy and efficiency of the multi-exit SR network. After obtaining the corresponding SR patches of all LR patches, the SR patches can be merged to generate an output SR image.
[0026] According to an embodiment, the regression operator may be defined as R i =σ(W*g(F i )+b), where F i represents a set of features in the ith middle patch, R i represents the incremental improvement of the prediction of the (i+1)th intermediate patch relative to the i-th intermediate patch, σ is the hyperbolic tangent (tanh) function, g is the global average pooling operation, and W and b are the weight and bias of the multi-export SR network, respectively.
[0027] Since patches in the input image may have various restoration difficulties, the exit layers for individual patches may be different. Figure 4 FIG. 4 illustrates visualization of early exit patches of an image during processing of a DPC-based image SR network according to some embodiments of the present disclosure. Figure 4 As shown in Figure 2, for patches in the smooth region of the input image, the SR process may exit at an early exit layer, for example, the number of layers performed may be 2 or 3, because these patches are easy to recover; but for patches in the complex region of the input image, the SR process may exit at a later exit layer, for example, the number of layers performed may be 4 or 5, because these patches are difficult to recover. This result is consistent with the motivation of applying appropriate networks for various recovery difficulties.
[0028] As described above, a multi-exit SR network may include multiple exit layers, and the SR process with the multi-exit SR network may be completed at any exit layer. Therefore, the training process of the multi-exit SR network may be different from the training process of the traditional SR network.
[0029] Specifically, the multi-exit SR network can be represented as f i , where i is the outlet index, which is between 1 and the number of outlets in the multi-outlet SR network. For an LR patch x, the SR patch y obtained by the SR process using the first layer to the i-th outlet layer of the multi-outlet SR network iIt can be expressed as y i =f i (x). Therefore, the reconstruction loss L of each output layer in the multi-output SR network can be i =|y i -y gt | to train the multi-export SR network, where y gt represents the ground-truth SR patch of the LR patch. In other words, the training process of the multi-outlet SR network can be expressed as follows using formulas (1) and (2).
[0030] L i =|y i -y gt | (1)
[0031]
[0032] In addition, the regression operator R i It can be based on the regression loss J i To train, the regression loss is defined as R of the i-th output layer i Incremental improvement with ground truth I i The L2 loss between is expressed as follows using formula (3).
[0033]
[0034] Therefore, the reconstruction loss L for each output layer in the multi-output SR network can be i And the regression loss J of the regression operator i The multi-outlet SR network is trained by summing up the total loss of . In this case, the training process of the multi-outlet SR network can be expressed by formula (4) as follows.
[0035]
[0036] In formula (4), λ is used to balance the reconstruction loss L i and regression loss J i Hyperparameters of .
[0037] In the above description, the architecture of the DPC-based multi-export SR network and the training process of the multi-export SR network have been described. The DPC-based multi-export SR network is a practical scalable network that can be deployed on platforms with different capacities. In addition, by adjusting the threshold for incremental improvement of each export layer, a trade-off between accuracy and efficiency can be achieved.
[0038] To demonstrate the advantages of the solution proposed in this disclosure, extensive experiments are conducted on various SR backbones, datasets, and scaling factors. Figure 5aand Figure 5b The diagram illustrates quantitative results of the accuracy-efficiency trade-off obtained by an example multi-outlet image SR network based on DPC according to some embodiments of the present disclosure. In order to evaluate the effectiveness of the proposed solution, EDSR and RCAN are used as the backbone of the multi-outlet SR network, and the DPC scheme is applied to EDSR and RCAN, respectively. EDSR based on the DPC scheme may be referred to as EDSR-DPC, and RCAN based on the DPC scheme may be referred to as RCAN-DPC. One outlet may be set for every 4 blocks for EDSR-DPC, so that EDSR-DPC may include 8 outlets. Similarly, one outlet may be set for each residual group for RCAN-DPC, so that RCAN-DPC may include 10 outlets. In Figure 5a The experimental results obtained using the DIV2K dataset for scaling factors x2, x3, x4 are shown in Figure 5b The experimental results obtained by using the DIV8K dataset for scaling factors x2, x3, x4 are shown in Fig. Figure 5a and Figure 5b In the figure, EDSR-original represents traditional EDSR, RCAN-original represents traditional RCAN, GFLOPs is the acronym for Giga Floating-point Operations, which represents the average FLOPs of all 32×32 LR patches, and PSNR is the acronym for Peak Signal to Noise Ratio, which is calculated on the full image.
[0039] from Figure 5a and Figure 5b As can be seen from the figure, the DPC scheme can significantly reduce the computational cost of EDSR and RCAN under different scaling factors. For example, on the DIV2K dataset, for scaling factors x2, x3, and x4, RCAN-DPC only requires 40%, 42%, and 44% of the original computational cost.
[0040] also, Figure 6The figure illustrates the performance comparison between the traditional image SR network (EDSR-O), the existing scalable image SR network (EDSR-AdaDSR) and the DPC-based image SR network (EDSR-DPC) according to some embodiments of the present disclosure. EDSR-AdaDSR is also a scalable image SR network that uses an adaptive inference network for deep SISR (AdaDSR). The details of AdaDSR are described in "Deep adaptive inference networks for single image super-resolution", Liu, M., Zhang, Z., Hou, L., Zuo, W., & Zhang, L., 2020, August, European Conference on Computer Vision (pp. 131-148), Springer, Cham. AdaDSR is based on pixel-level sparse convolution to achieve acceleration. However, pixel-level sparse convolution is not hardware-friendly on modern GPUs, so Figure 1 As shown in Figure 2, there is a gap between theoretical and actual speedup. Using EDSR as the backbone, the traditional SR network EDSR-O, AdaDSR-based EDSR, and DPC-based EDSR are compared at different scaling factors with the same accuracy as the baseline. Figure 6 It can be seen that with similar parameters, in practice, EDSR-DPC is faster than EDSR-AdaDSR when tested on NVIDIA 2080Ti.
[0041] Figure 7 An example process of visual synthesis of a multi-outlet visual synthesis network (VSN) based on DPC according to some embodiments of the present disclosure is illustrated. As described above, the proposed DPC scheme can be applied to a multi-outlet VSN, such as an image SR network, an image denoising network, or an image deblurring network. In summary, the process of visual synthesis of a multi-outlet VSN based on DPC may include operations 710 to 750 and may be implemented by a processor circuit.
[0042] At operation 710 , a processor circuit may segment an input image into a plurality of input patches.
[0043] At operation 720, the processor circuit may perform a synthesis process on each input patch using the first layer to the i-th exit layer of the multi-exit VSN to obtain an i-th intermediate synthesized patch having improved characteristics relative to the input patch. Here, i is an index of an intermediate exit of the VSN and is predetermined to be an integer greater than or equal to 1.
[0044] At operation 730 , the processor circuit may predict an incremental improvement of the (i+1)th intermediate synthetic patch relative to the i th synthetic patch based on features in the i th intermediate synthetic patch.
[0045] According to some embodiments, the processor circuit may predict the incremental improvement using a regression operator, where the regression operator is defined as R i =σ(W*g(F i )+b), where F i represents a set of features in the ith intermediate synthetic patch, R i represents the predicted incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch, σ is the hyperbolic tangent function, g is the global average pooling operation, and W and b are the weight and bias of the multi-exit VSN, respectively.
[0046] At operation 740 , the processor circuit may determine a final exit of the VSN and a final composite patch of the input patches based on the predicted incremental improvement.
[0047] According to some embodiments, the processor circuit may determine the i-th exit as the final exit and determine the i-th intermediate synthetic patch as the final synthetic patch of the input patch when the incremental improvement is below a predetermined threshold, otherwise, increment i and continue to execute the synthesis process and predict the incremental improvement until the incremental improvement is below a predetermined threshold or all layers in the VSN have been traversed by the synthesis process.
[0048] According to some embodiments, the processor circuit may adjust the predetermined threshold based on a trade-off between accuracy and efficiency of the multi-exit VSN.
[0049] At operation 750 , the processor circuit may merge respective final synthesized patches of the plurality of input patches to generate an output image.
[0050] According to some embodiments, the regression loss J of the regression operator i can be defined as R of the i-th exit layer i and ground truth incremental improvement I i The L2 loss between .
[0051] According to some embodiments, the reconstruction loss L of each egress layer in a multi-egress VSN may be calculated based on i =|y i -y gt| to train the multi-exit VSN, where y i represents the ith intermediate synthetic patch, and y gt A synthetic patch representing the ground truth of the input patch.
[0052] According to some embodiments, the reconstruction loss L for each outlet layer in the multi-outlet VSN may be based on i And the regression loss J of the regression operator i The total loss is the sum of the total losses to train the multi-exit VSN. The total loss can be defined as L i +λJ i , where λ is used to balance the reconstruction loss L i and regression loss J i Hyperparameters of .
[0053] According to some embodiments, a multi-outlet VSN may be trained based on a patch-level sparsity pattern, and then the synthesis process may be performed based on a patch-level sparse convolution corresponding to the patch-level sparsity pattern.
[0054] Figure 8 The block diagram of illustrates components capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and executing any one or more of the methods discussed herein according to some example embodiments. Specifically, Figure 8 A diagrammatic representation of hardware resources 800 is shown, including one or more processors (or processor cores) 810, one or more memory / storage devices 820, and one or more communication resources 830, each of which may be communicatively coupled via a bus 840. For embodiments utilizing node virtualization (e.g., NFV), a hypervisor 802 may be executed to provide an execution environment for one or more network slices / sub-slices utilizing the hardware resources 800.
[0055] Processor 810 may include, for example, processor 812 and processor 814, and processor 512 and processor 514 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a visual processing unit (VPU), a field programmable gate array (FPGA), or any appropriate combination of these.
[0056] The memory / storage device 820 may include main memory, disk storage, or any suitable combination of these. The memory / storage device 820 may include, but is not limited to, any type of volatile or non-volatile memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state storage, etc.
[0057] The communication resources 830 may include an interconnect or network interface component or other appropriate device to communicate with one or more peripheral devices 804 or one or more databases 806 via the network 808. For example, the communication resources 830 may include a wired communication component (e.g., for coupling via a Universal Serial Bus (USB)), a cellular communication component, an NFC component, Components (e.g., low energy ), components, and other communication components.
[0058] The instructions 850 may include software, programs, applications, applet programs, apps, or other executable code for causing at least any one of the processors 810 to perform any one or more of the methods discussed herein. The instructions 850 may reside in whole or in part within at least one of the processors 810 (e.g., within a cache memory of the processor), within the memory / storage device 820, or any suitable combination of these. In addition, any portion of the instructions 850 may be transferred to the hardware resources 800 from any combination of the peripheral devices 804 or the database 806. Thus, the memory of the processor 810, the memory / storage device 820, the peripheral devices 804, and the database 806 are examples of computer-readable and machine-readable media.
[0059] Fig. 9 900 is a block diagram of an example processor platform according to some embodiments of the present disclosure. The processor platform 900 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, an iPad, etc.), or a processor platform. TM), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, headphones or other wearable device, or any other type of computing device.
[0060] The processor platform 900 of the illustrated example includes a processor 912. The processor 912 of the illustrated example is hardware. For example, the processor 912 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In some embodiments, the processor implements one or more of the methods or processes described above.
[0061] The processor 912 of the illustrated example includes a local memory 913 (e.g., cache). The processor 912 of the illustrated example communicates with a main memory including a volatile memory 914 and a non-volatile memory 916 via a bus 918. The volatile memory 914 may be a synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), Dynamic Random Access Memory (DRAM) Dynamic Random Access Memory, ) and / or any other type of random access memory device. The non-volatile memory 916 can be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 914, 916 is controlled by a memory controller.
[0062] The processor platform 900 of the illustrated example further includes an interface circuit 920. The interface circuit 920 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.
[0063] In the illustrated example, one or more input devices 922 are connected to the interface circuit 920. The input device(s) 922 allow a user to enter data and / or commands into the processor 912. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, and / or a voice recognition system.
[0064] One or more output devices 924 are also connected to the interface circuit 920 of the illustrated example. The output device 924 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. The interface circuit 920 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.
[0065] The interface circuitry 920 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate data exchange with external machines (e.g., any kind of computing device) via a network 926. Communications may be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, and the like.
[0066] For example, interface circuit 920 may include a training data set input through input device(s) 922 or retrieved from network 926 .
[0067] The processor platform 900 of the illustrated example also includes one or more mass storage devices 928 for storing software and / or data. Examples of such mass storage devices 928 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.
[0068] The machine-executable instructions 932 may be stored in the mass storage device 928, in the volatile memory 914, in the non-volatile memory 916, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.
[0069] Additional notes and examples:
[0070] Example 1 includes a device for visual synthesis, which includes: an interface circuit; and a processor circuit, which is coupled to the interface circuit and configured to: segment an input image received via the interface circuit into multiple input patches; perform a synthesis process on each input patch using the first layer to the i-th outlet layer of a multi-outlet visual synthesis network (VSN) to obtain an i-th intermediate synthetic patch, where i is the index of the intermediate outlet of the VSN and is predetermined to be an integer greater than or equal to 1; based on the features in the i-th intermediate synthetic patch, predict an incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch; determine the final outlet of the VSN and the final synthetic patch of the input patches based on the predicted incremental improvement; and merge the corresponding final synthetic patches of the multiple input patches to generate an output image.
[0071] Example 2 includes an apparatus as described in Example 1, wherein the processor circuit is configured to determine the final exit of the VSN and the final synthesized patch of the input patch by the following steps: when the incremental improvement is lower than a predetermined threshold, determining the i-th exit as the final exit and determining the i-th intermediate synthesized patch as the final synthesized patch of the input patch, otherwise, incrementing i and continuing to execute the synthesis process and predict the incremental improvement until the incremental improvement is lower than the predetermined threshold or all layers in the VSN have been traversed by the synthesis process.
[0072] Example 3 includes the apparatus of Example 2, wherein the processor circuit is further configured to adjust the predetermined threshold based on a tradeoff between accuracy and efficiency of the multi-exit VSN.
[0073] Example 4 includes the apparatus of any of Examples 1 to 3, wherein the processor circuit is configured to predict the incremental improvement using a regression operator defined as R i =σ(W*g(F i )+b), where F i represents a set of features in the i-th intermediate synthetic patch, R i represents the predicted incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch, σ is the hyperbolic tangent function, g is the global average pooling operation, W and b are the weight and bias of the multi-exit VSN, respectively.
[0074] Example 5 includes the apparatus of Example 4, wherein the regression loss J of the regression operator is i is defined as R of the i-th exit layer i and ground truth incremental improvement I i The L2 loss between .
[0075] Example 6 includes the apparatus of any one of Examples 1 to 5, wherein the multi-egress VSN is based on a reconstruction loss L for each egress layer in the multi-egress VSN. i =|y i -y gt | for training, where y i Indicates the i-th th The intermediate synthetic patch, and y gt A synthetic patch representing the ground truth of the input patch.
[0076] Example 7 includes the apparatus of Example 6, wherein the multi-exit VSN is trained based on a sum of total losses including the following losses: a reconstruction loss L for each exit layer in the multi-exit VSN i and the regression loss J of the regression operator i .
[0077] Example 8 includes the apparatus of Example 7, wherein the total loss is defined as L i +λJ i , where λ is used to balance the reconstruction loss L i and the regression loss J i Hyperparameters of .
[0078] Example 9 includes an apparatus as described in any one of Examples 1 to 8, wherein the multi-exit VSN is trained based on a patch-level sparse pattern, and the processor circuit is configured to perform the synthesis process based on a patch-level sparse convolution corresponding to the patch-level sparse pattern.
[0079] Example 10 includes the apparatus of any one of Examples 1 to 8, wherein the multi-exit VSN comprises an image super-resolution network, an image denoising network, or an image deblurring network.
[0080] Example 11 includes a method for visual synthesis, comprising: segmenting an input image into multiple input patches; performing a synthesis process on each input patch using a first layer to an i-th outlet layer of a multi-outlet visual synthesis network (VSN) to obtain an i-th intermediate synthetic patch, where i is the index of the intermediate outlet of the VSN and is predetermined to be an integer greater than or equal to 1; based on features in the i-th intermediate synthetic patch, predicting an incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch; determining a final outlet of the VSN and a final synthetic patch of the input patches based on the predicted incremental improvement; and merging the corresponding final synthetic patches of the multiple input patches to generate an output image.
[0081] Example 12 includes a method as described in Example 11, wherein determining the final exit of the VSN and the final synthesized patch of the input patch includes: when the incremental improvement is lower than a predetermined threshold, determining the i-th exit as the final exit and determining the i-th intermediate synthesized patch as the final synthesized patch of the input patch, otherwise, incrementing i and continuing to execute the synthesis process and predict the incremental improvement until the incremental improvement is lower than the predetermined threshold or all layers in the VSN have been traversed by the synthesis process.
[0082] Example 13 includes the method of Example 12, further comprising adjusting the predetermined threshold based on a tradeoff between accuracy and efficiency of the multi-exit VSN.
[0083] Example 14 includes the method of any one of Examples 11 to 13, wherein predicting the incremental improvement comprises predicting the incremental improvement using a regression operator, the regression operator being defined as R i =σ(W*g(F i )+b), where F i represents a set of features in the i-th intermediate synthetic patch, R i represents the predicted incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch, σ is the hyperbolic tangent function, g is the global average pooling operation, W and b are the weight and bias of the multi-exit VSN, respectively.
[0084] Example 15 includes the method of Example 14, wherein the regression loss J of the regression operator is i is defined as R of the i-th exit layer i and ground truth incremental improvement I i The L2 loss between .
[0085] Example 16 includes the method of any one of Examples 11 to 15, wherein the multi-egress VSN is based on a reconstruction loss L for each egress layer in the multi-egress VSN.i =|y i -y gt | for training, where y i Indicates the i-th th The intermediate synthetic patch, and y gt A synthetic patch representing the ground truth of the input patch.
[0086] Example 17 includes the method of Example 16, wherein the multi-exit VSN is trained based on a sum of total losses including the following losses: a reconstruction loss L for each exit layer in the multi-exit VSN i and the regression loss J of the regression operator i .
[0087] Example 18 includes the method of Example 17, wherein the total loss is defined as L i +λJ i , where λ is used to balance the reconstruction loss L i and the regression loss J i Hyperparameters of .
[0088] Example 19 includes a method as described in any one of Examples 11 to 18, wherein the multi-exit VSN is trained based on a patch-level sparsity pattern, and the synthesis process is performed based on patch-level sparse convolution corresponding to the patch-level sparsity pattern.
[0089] Example 20 includes the method of any one of Examples 11 to 18, wherein the multi-exit VSN comprises an image super-resolution network, an image denoising network, or an image deblurring network.
[0090] Example 21 includes a computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor circuit, cause the processor circuit to perform the method of any of Examples 11 to 20.
[0091] Example 22 includes an apparatus for visual synthesis, the apparatus comprising means for performing the method as described in any of Examples 11 to 20.
[0092] Various technologies or some aspects or parts thereof may take the form of program codes (i.e., instructions) embodied in tangible media, such as floppy disks, CD-ROMs, hard disk drives, non-transient computer-readable storage media, or any other machine-readable storage media, wherein when the program code is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for implementing various technologies. Non-transient computer-readable storage media may be computer-readable storage media that do not include signals. In the case of program code execution on a programmable computer, a computing system may include a processor, a storage medium (including volatile and non-volatile memory and / or storage elements) that can be read by the processor, at least one input device, and at least one output device. Volatile and non-volatile memory and / or storage elements may be RAM, EPROM, flash drives, optical drives, magnetic hard disk drives, solid-state drives, or other media for storing electronic data. One or more programs that may implement or utilize the various technologies described herein may use application programming interfaces (application programming interfaces, APIs), reusable controls, and the like. Such programs may be implemented in high-level procedural or object-oriented programming languages to communicate with a computer system. However, if desired, the program(s) may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language, and combined with a hardware implementation. Exemplary systems or devices may include, but are not limited to, laptop computers, tablet computers, desktop computers, smart phones, computer terminals and servers, storage databases, and other electronic devices utilizing circuits and programmable memory, such as household appliances, smart televisions, digital video disc (DVD) players, heating, ventilating, and air conditioning (HVAC) controllers, light switches, and the like.
[0093] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings illustrate specific embodiments that can be implemented. These embodiments are also referred to as "examples" in this article. Such examples may include additional elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. In addition, the inventors also contemplate examples (or one or more aspects thereof) using any combination or arrangement of those elements shown or described, whether for a specific example (or one or more aspects thereof), or for other examples (or one or more aspects thereof) shown or described herein.
[0094] All publications, patents, and patent documents cited in this document are hereby incorporated by reference in their entirety, as if individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.
[0095] In this document, the term "a" is used to include one or more than one, independent of any other instance or use of "at least one" or "one or more," as is common in patent documents. In this document, the term "or" is used to refer to a non-exclusive or, so that "A or B" includes "A but not B," "B but not A," and "A and B," unless otherwise indicated. In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the respective terms "comprising" and "wherein." In addition, in the appended claims, the terms "including" and "comprising" are open-ended, that is, systems, devices, articles, or processes that include other elements in addition to those listed after such terms in the claim are still considered to fall within the scope of the claim. In addition, in the appended claims, the terms "first," "second," and "third," etc. are used merely as labels and are not intended to impose numerical requirements on their objects.
[0096] The above description is intended to be illustrative, not restrictive. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, for example, by a person of ordinary skill in the art after viewing the above description. The abstract is intended to allow the reader to quickly determine the nature of the technical disclosure and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above detailed description, various features may be grouped together to simplify the disclosure. This should not be interpreted as a desire to make the disclosed features that are not claimed necessary for any claim. More precisely, the subject matter of the invention may be present in less than all the features of a particular disclosed embodiment. Thus, the attached claims are incorporated herein into the "Specific Implementation" section, with each claim acting independently as a separate embodiment. The scope of the embodiment should be determined with reference to the attached claims and the full scope of equivalents to which such claims are entitled.
Claims
1. A device for visual synthesis, comprising: Interface circuit; and a processor circuit coupled to the interface circuit and configured to: segmenting an input image received via the interface circuit into a plurality of input patches; Performing a synthesis process on each input patch using a first layer to an i-th outlet layer of a multi-outlet visual synthesis network (VSN) to obtain an i-th intermediate synthesized patch, where i is an index of an intermediate outlet of the VSN and is predetermined to be an integer greater than or equal to 1; Based on the features in the i-th intermediate synthetic patch, predicting an incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch; determining a final exit of the VSN and a final composite patch of the input patches based on the predicted incremental improvements; and The respective final synthesized patches of the plurality of input patches are merged to generate an output image.
2. The device according to claim 1, wherein: The processor circuit is configured to determine the final exit of the VSN and the final synthesized patch of the input patch by the following steps: when the incremental improvement is lower than a predetermined threshold, determine the i-th exit as the final exit and determine the i-th intermediate synthesized patch as the final synthesized patch of the input patch, otherwise, increment i and continue to execute the synthesis process and predict the incremental improvement until the incremental improvement is lower than the predetermined threshold or all layers in the VSN have been traversed by the synthesis process.
3. The device as claimed in claim 2, wherein: The processor circuit is further configured to adjust the predetermined threshold based on a tradeoff between accuracy and efficiency of the multi-exit VSN.
4. The device according to claim 1, wherein: The processor circuit is configured to predict the incremental improvement using a regression operator defined as R i =σ(W*g(F i )+b), where F i represents a set of features in the i-th intermediate synthetic patch, R i represents the predicted incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch, σ is the hyperbolic tangent function, g is the global average pooling operation, W and b are the weight and bias of the multi-exit VSN, respectively.
5. The device according to claim 4, wherein: The regression loss J of the regression operator i is defined as R of the i-th outlet layer i and ground truth incremental improvement I i The L2 loss between .
6. The device according to claim 1, wherein: The multi-outlet VSN is constructed based on the reconstruction loss L of each outlet layer in the multi-outlet VSN. i =|y i -y gt | for training, where y i Indicates the i-th th The intermediate synthetic patch, and y gt A synthetic patch representing the ground truth of the input patch.
7. The device according to claim 6, wherein: The multi-outlet VSN is trained based on the sum of the total losses including the following losses: i and the regression loss J of the regression operator i .
8. The device according to claim 7, wherein: The total loss is defined as L i +λJ i , where λ is used to balance the reconstruction loss L i and the regression loss J i Hyperparameters of .
9. The device according to any one of claims 1 to 8, wherein: The multi-exit VSN is trained based on a patch-level sparsity pattern, and the processor circuit is configured to perform the synthesis process based on a patch-level sparse convolution corresponding to the patch-level sparsity pattern.
10. The device according to any one of claims 1 to 8, wherein: The multi-exit VSN includes an image super-resolution network, an image denoising network, or an image deblurring network.
11. A method for visual synthesis, comprising: Split the input image into multiple input patches; Performing a synthesis process on each input patch using a first layer to an i-th outlet layer of a multi-outlet visual synthesis network (VSN) to obtain an i-th intermediate synthesized patch, where i is an index of an intermediate outlet of the VSN and is predetermined to be an integer greater than or equal to 1; Based on the features in the i-th intermediate synthetic patch, predicting an incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch; determining a final exit of the VSN and a final composite patch of the input patches based on the predicted incremental improvements; as well as The respective final synthesized patches of the plurality of input patches are merged to generate an output image.
12. The method of claim 11, wherein: Determining the final exit of the VSN and the final synthesized patch of the input patch includes: when the incremental improvement is lower than a predetermined threshold, determining the i-th exit as the final exit and determining the i-th intermediate synthesized patch as the final synthesized patch of the input patch; otherwise, incrementing i and continuing to execute the synthesis process and predict the incremental improvement until the incremental improvement is lower than the predetermined threshold or all layers in the VSN have been traversed by the synthesis process.
13. The method of claim 12, further comprising: The predetermined threshold is adjusted based on a trade-off between accuracy and efficiency of the multi-exit VSN.
14. The method of claim 11, wherein: Predicting the incremental improvement includes predicting the incremental improvement using a regression operator, wherein the regression operator is defined as R i =σ(W*g(F i )+b), where F i represents a set of features in the i-th intermediate synthetic patch, R i represents the predicted incremental improvement of the (i+1)th intermediate synthetic patch relative to the i-th intermediate synthetic patch, σ is the hyperbolic tangent function, g is the global average pooling operation, W and b are the weight and bias of the multi-exit VSN, respectively.
15. The method of claim 14, wherein: The regression loss J of the regression operator i is defined as R of the i-th outlet layer i and ground truth incremental improvement I i The L2 loss between .
16. The method of claim 11, wherein: The multi-outlet VSN is constructed based on the reconstruction loss L of each outlet layer in the multi-outlet VSN. i =|y i -y gt | for training, where y i Indicates the i-th th The intermediate synthetic patch, and y gt A synthetic patch representing the ground truth of the input patch.
17. The method of claim 16, wherein: The multi-outlet VSN is trained based on the sum of the total losses including the following losses: i and the regression loss J of the regression operator i .
18. The method of claim 17, wherein: The total loss is defined as L i +λJ i , where λ is used to balance the reconstruction loss L i and the regression loss J i Hyperparameters of .
19. The method according to any one of claims 11 to 18, wherein: The multi-exit VSN is trained based on a patch-level sparse pattern, and the synthesis process is performed based on a patch-level sparse convolution corresponding to the patch-level sparse pattern.
20. The method of any one of claims 11 to 18, wherein: The multi-exit VSN includes an image super-resolution network, an image denoising network, or an image deblurring network.
21. A computer readable medium having instructions stored thereon, wherein: When the instructions are executed by a processor circuit, the processor circuit is caused to perform the method according to any one of claims 11 to 20.
22. An apparatus for visual synthesis, the apparatus comprising means for performing the method of any one of claims 11 to 20.
