Multi-receptive field in vision transducer

Abstract

Methods and apparatus for neural network-based image compression are provided. The method may include generating a feature map including a plurality of channels for a compressed image; segmenting the generated feature map into a first number of segmented feature maps in a vision converter; and reconstructing the compressed image using one or more remote attention models having one or more variable receptive fields associated with the respective segmented feature maps.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based upon and claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 435,510 filed on December 27, 2022 and U.S. Patent Application No. 18 / 455,982 filed on August 25, 2023, the disclosures of which are incorporated herein by reference in their entirety. Background Art

[0002] Traditional hybrid video codecs are difficult to optimize as a whole. Improvements in a single module may not result in coding gains in overall performance. Recently, standard groups and companies have been actively looking for potential needs for standardization of future video codec technologies. These standard groups and companies have established the JPEG-AI group that focuses on AI-based end-to-end neural image compression using deep neural networks (DNNs). The Chinese AVS standard has also formed an AVS-AI special group dedicated to neural image and video compression technology. The success of recent approaches has attracted increasing industrial interest in advanced neural image and video compression methods.

[0003] In the learned image compression framework, context models play a key role in capturing the correlations in the latent representation. In the related art, autoregressive context models are used to reduce the decoding time generated by serial autoregressive contexts. However, due to the continuous prediction of the latent variables, the corresponding computational complexity of the autoregressive model is O(n 2 ) order (n is the height or width of the latent variable). Such high cost stems from the basic scheme of the context model, which requires all previously decoded latent variables to predict new latent variables. Therefore, a more efficient and parallelizable context model is needed to reduce the complexity of the context model and improve the efficiency of the context model.

[0004] In addition, the related art autoregressive context model only examines a locally limited receptive field (constrained by the kernel size (such as 3×3, 5×5)) when decoding the current latent variable. Therefore, a context model that more effectively considers global latent features is needed. Summary of the invention

[0005] According to an embodiment, a method and apparatus for implementing a long-range context model in neural image compression are provided. The embodiments disclosed herein can be applied to encoding and decoding processes in neural image compression.

[0006] According to one aspect of the present disclosure, a method for neural network-based image compression may be provided. The method may be performed by at least one processor and may include generating a multi-channel feature map for a compressed image; segmenting the multi-channel feature map into a first number of segmented feature maps; and reconstructing the compressed image using one or more remote attention models having one or more variable attention windows associated with the corresponding segmented feature maps.

[0007] According to another aspect of the present disclosure, a device for neural network-based image compression may be provided. The device may include: at least one memory configured to store computer program code; and at least one processor configured to read the computer program code and operate according to the instructions of the computer program code. The program code may include a generation code configured to cause at least one processor to generate a multi-channel feature map for a compressed image; a segmentation code configured to cause at least one processor to segment the multi-channel feature map into a first number of segmentation feature maps; and a reconstruction code configured to cause at least one processor to use one or more remote attention models having one or more variable attention windows associated with corresponding segmentation feature maps to reconstruct the compressed image.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable medium storing instructions executed by at least one processor may be provided, the instructions enabling the at least one processor to generate a multi-channel feature map for a compressed image; segment the multi-channel feature map into a first number of segmented feature maps; and reconstruct the compressed image using one or more long-range attention models having one or more variable attention windows associated with corresponding segmented feature maps.

[0009] Additional embodiments will be set forth in the description which follows and, in part, will be apparent from the description, and / or may be accomplished by practice of the presented embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram of an environment in which the methods, apparatus, and systems described herein may be implemented, according to an embodiment.

[0011] Figure 2 yes Figure 1 A block diagram of example components of one or more devices.

[0012] Figure 3A An example of a framework for a variational autoencoder (VAE) based neural image compression network is illustrated.

[0013] FIG. 3B to FIG. 3C Pictured Figure 3AExamples of encoder and decoder structures of one or more encoders or decoders of a VAE-based neural image compression network.

[0014] FIG. 4A to FIG. 4B is an example of the structure of a visual transformer in a neural image compression network.

[0015] Figure 5 Illustrated are examples of multiple receptive fields and / or attention windows in a transformer according to an embodiment.

[0016] Figure 6 is a flow chart illustrating a method for neural image compression (NIC) using a neural network according to an embodiment. DETAILED DESCRIPTION

[0017] The following detailed description of example embodiments refers to the accompanying drawings.The same reference numbers in different drawings may identify the same or similar elements.

[0018] The above disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise form disclosed. According to the above disclosure, modifications and variations are possible, or modifications and variations can be obtained from the practice of the embodiments. In addition, one or more features or components of an embodiment may be incorporated into another embodiment (or one or more features of another embodiment) or combined with another embodiment (or one or more features of another embodiment). Additionally, in the flowcharts and operational descriptions provided below, it should be understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least partially), and the order of one or more operations may be switched.

[0019] Obviously, the systems and / or methods described herein can be implemented in different forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods described herein are not referenced to specific software codes. It should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0020] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim group.

[0021] The proposed features discussed below can be used alone or in any order. In addition, these embodiments can be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer readable medium.

[0022] Unless so clearly described, any element, behavior or instruction used herein should not be interpreted as critical or necessary. In addition, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more". When only one item is intended to be referred to, the term "one" or similar language is used. In addition, as used herein, the terms "has", "have", "having", "include", "including", etc. are intended to be open terms. In addition, unless otherwise clearly stated, the phrase "based on" is intended to mean "based at least in part". In addition, expressions such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include only A, only B, or both A and B.

[0023] Embodiments of the present disclosure relate to a Corner-to-Center transformer-based Context Model (C3M) or an Edge-to-Center transformer-based Context Model, which is designed to enhance context and potential prediction and improve rate-distortion performance.

[0024] The VAE-based framework (e.g., Figure 3) utilizes a DNN-based transform as the primary encoder to project the image into a low-dimensional latent space. After quantization, the entropy estimation model predicts the distribution of the latent variables, which is then compressed into a bitstream using an arithmetic encoder assisted by the estimated distribution. At the decoding end, the same entropy estimation model is applied to the arithmetic decoder to recover the latency information. This information is then fed to the DNN-based primary decoder to reconstruct the original image.

[0025] Although context models offer promising advantages in enhancing compression performance, they come with significant deployment costs. Typically, due to the continuous prediction of latent variables, the corresponding computational complexity of autoregressive models is O(n 2 ) (n is the height or width of the latent variable). Such high cost stems from the base scheme of the context model, which requires all previously decoded latent variables to predict new latent variables. Therefore, the autoregressive context model cannot perform concurrently.

[0026] In the above learned image compression framework, the key component is the autoregressive context model, which attempts to predict unknown codes based on the availability of previously decoded codes. The disadvantage of the autoregressive context model in the related art is that the model only extracts image texture in local areas without considering global semantic information. This incomplete causal context and restricted receptive field essentially limit the performance of the context model.

[0027] Therefore, in order to address the above challenges, embodiments of the present disclosure are directed to a context model based on an efficient and high-performance transformer, referred to as a corner-to-center context model (C3M). Specifically, the C3M model predicts context features progressively as it moves from corner positions to center positions. At each prediction, the total received information is maximized as the number of context features to be predicted increases incrementally.

[0028] Figure 1 is a schematic diagram of an environment 100 in which the methods, apparatuses, and systems described herein may be implemented, according to an embodiment.

[0029] like Figure 1 As shown, environment 100 may include user device 110, platform 120, and network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0030] The user device 110 includes one or more devices that can receive, generate, store, process, and / or provide information related to the platform 120. For example, the user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some embodiments, the user device 110 can receive information from the platform 120 and / or send information to the platform 120.

[0031] The platform 120 includes one or more devices as described elsewhere herein. In some embodiments, the platform 120 may include a cloud server or a cloud server group. In some embodiments, the platform 120 may be designed to be modular so that software components can be swapped in or out. In this way, the platform 120 may be easily and / or quickly reconfigured to have different uses.

[0032] In some embodiments, as shown, the platform 120 may be hosted in a cloud computing environment 122. It is worth noting that although the embodiments described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some embodiments, the platform 120 is not cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0033] Cloud computing environment 122 includes an environment hosting platform 120. Cloud computing environment 122 may provide computing, software, data access, storage, and other services that do not require end users (e.g., user devices 110) to be aware of the physical location and configuration of the systems and / or devices hosting platform 120. As shown, cloud computing environment 122 may include a set of computing resources 124 (collectively, “computing resources 124” and individually, “computing resource 124”).

[0034] The computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some embodiments, the computing resources 124 can host the platform 120. Cloud resources can include computing instances executed in the computing resources 124, storage devices provided in the computing resources 124, data transmission devices provided by the computing resources 124, etc. In some embodiments, the computing resources 124 can communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0035] Further Figure 1 As shown, the computing resources 124 include a set of cloud resources, such as one or more application programs ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage ("VS") 124-3, one or more hypervisors ("HYP") 124-4, etc.

[0036] The applications 124-1 include one or more software applications that can be provided to or accessed by the user device 110 and / or the platform 120. The applications 124-1 do not require the software applications to be installed and executed on the user device 110. For example, the applications 124-1 may include software associated with the platform 120, and / or any other software that can be provided through the cloud computing environment 122. In some embodiments, one application 124-1 can send / receive information to or from one or more other applications 124-1 through the virtual machine 124-2.

[0037] Virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs, similar to a physical machine. Virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the use and correspondence of virtual machine 124-2 to any real machine. System virtual machines can provide a complete system platform that supports the execution of a complete operating system ("OS"). Process virtual machines can execute a single program and can support a single process. In some embodiments, virtual machine 124-2 can execute on behalf of a user (e.g., user device 110) and can manage the infrastructure of cloud computing environment 122, such as data management, synchronization, or long-term data transfer.

[0038] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of the computing resource 124. In some embodiments, within the context of the storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage so that the storage system can be accessed without considering the physical storage or heterogeneous structure. Separation may allow administrators of the storage system to flexibly manage the storage of end users. File virtualization may eliminate the dependency between data accessed at the file level and the location of the physical storage file. This may optimize the performance of storage usage, server consolidation, and / or non-disruptive file migration.

[0039] Hypervisor 124-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 may provide a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.

[0040] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0041] Figure 1 The number and arrangement of devices and networks shown are provided as examples. Figure 1 There may be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different arrangements of devices and / or networks than those shown. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100.

[0042] Figure 2 yes Figure 1 A block diagram of example components of one or more devices in FIG.

[0043] Device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0044] The bus 210 includes components that allow communication between components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory), which stores information and / or instructions for use by the processor 220.

[0045] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-volatile computer-readable medium, and a corresponding drive.

[0046] Input components 250 include components that allow device 200 to receive information, such as through user input, such as a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone. Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., global positioning system (GPS) components, accelerometers, gyroscopes, and / or actuators). Output components 260 include components that provide output information from device 200, such as a display, a speaker, and / or one or more light emitting diodes (LEDs).

[0047] The communication interface 270 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 270 can allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0048] Device 200 can perform one or more processes described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored by non-volatile computer-readable media (e.g., memory 230 and / or storage component 240). Computer-readable media is defined herein as non-volatile memory devices. Memory devices include storage space within a single physical storage device or storage space distributed on multiple physical storage devices.

[0049] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device through the communication interface 270. When executed, the software instructions stored in the memory 230 and / or storage component 240 may cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0050] Figure 2 The number and arrangement of components shown are provided as examples. Figure 2 The device 200 may include more components, fewer components, different components, or components arranged differently than the components shown. Additionally or alternatively, a set of components (e.g., one or more components) of the device 200 may perform one or more functions described as being performed by another set of components of the device 200.

[0051] In the embodiment, FIG. 3 to FIG. Figure 5 Any of the operations or processes may be performed by or using Figure 1 and implemented by any one of the elements illustrated in FIG. 3 .

[0052] Hybrid image codecs in related art are difficult to optimize. Improvements in a single module may not lead to the best coding gain in overall performance. In contrast, in an artificial neural network-based image coding framework, different modules can be jointly optimized from input to output by performing a machine learning process to improve the final goal (e.g., rate-distortion performance), resulting in an end-to-end (E2E) optimized Neural Image Compression (NIC).

[0053] Figure 3A is an illustration of an exemplary block diagram 300 of a framework of a variational autoencoder (VAE) based neural image compression network, according to an embodiment.

[0054] As shown in FIG. 3 , the NIC framework includes a main encoder 310 (e.g., Figure 3B ), the main decoder 320 (eg, Figure 3C ), super encoder 330, super decoder 340, context model 350, entropy parameter network 355 and factorized entropy model 390. The VAE-based NIC framework may include one or more such modules. The VAE-based NIC framework further includes a quantizer 360 / 361, an arithmetic encoder 370 / 371 and an arithmetic decoder 380 / 381. The same or similar modules are represented by the same reference numerals. The NIC framework may include one or more modules not shown in Figure 3.

[0055] The NIC framework can use any DNN-based image compression method, such as the scaled hyper-prior encoder-decoder framework (or Gaussian mixture likelihood framework) and its variants, RNN-based recursive compression methods and their variants.

[0056] According to an embodiment of the present disclosure, the NIC framework may utilize block diagram 300 as follows. Given an input image or video sequence x, the primary encoder 310 may compute a compressed representation or y, when compared with the input image x, the compressed representation or y is compact for storage and transmission purposes. The compressed representation can be converted using quantizer 360 Quantization into discrete value quantization representation The discrete value can then be quantized using arithmetic coding (lossless or lossy) using an arithmetic encoder 370 At the decoder side, the code stream can be restored to the discrete value quantized representation by lossless or lossy entropy decoding using the arithmetic decoder 380. Then the discrete value is quantized to represent may be input to the primary decoder 320 to recover and / or reconstruct the input image or video sequence The primary encoder 310 and the primary decoder 320 may be a neural network based encoder and decoder (eg, a DNN based encoder).

[0057] In some embodiments, previous NIC methods adopt a variational autoencoder (VAE) structure, where the DNN encoder directly uses the entire image x as its input, which computes the output representation x through a set of network layers that work like a black box. Correspondingly, the DNN decoder uses the entire representation As its input, it passes through another set of network layers that works like another black box to compute the reconstructed

[0058] The super encoder 330 can use a series of convolutional layers and long-range crossing attention modules (LCAM) to compress the representation 361 and the arithmetic encoder 371 can then be used to generate a super-compressed representation of the super-encoded compressed representation. The arithmetic decoder 381 can decode the super-compressed representation. The super-decoder 340 can then be used to generate a super-reconstructed image x ′ The neural network based context model 350 may be trained using the super-reconstructed image and the quantized representation from the quantizer 360. The context model 350 may be used by the arithmetic encoder 370 and the arithmetic decoder 380 for encoding and decoding, respectively.

[0059] The VAE-based neural image compression architecture can further incorporate super priors to effectively capture spatial correlations in the latent representation. A context model inspired by the concept of context from traditional codecs can be used to predict the probability of an unknown code based on the already decoded latent variables. The latent variables can be generated by the main encoder 310 in the VAE structure. Super latent variables and context can be used jointly to predict the location (e.g., mean) and scale parameters of the entropy model.

[0060] Using the following objective loss function L, the rate-distortion (RD) loss is optimized with the trade-off hyperparameter λ to achieve the reconstructed image Distortion loss and compressed representation The trade-off between bit consumption R:

[0061]

[0062] Embodiments of the proposed long-range context model predict latent variables by focusing on long-range global regions and covering the entire latent features more effectively. Unlike autoregressive-based context models (and their variants) and parallel-based context models (and their variants), known convolutional layer-based methods only examine local restricted receptive fields (constrained by kernel size (such as 3×3, 5×5)) when decoding the current latent variable, the embodiments disclosed herein focus on long-range global regions (which can learn correlations in the entire latent variable and are not constrained by kernel size), which covers the entire latent feature more effectively.

[0063] Figure 3A An example legend can be seen in Table 1 below. Table 1 - Exemplary legend for Figure 3

[0064] FIG. 4A to FIG. 4B 4 and 50 according to an embodiment. Instead of using convolutional layers to model correlations or extract features / in addition to using convolutional layers to model correlations or extract features, the video transformer can also use self-attention. Figure 4A As shown, the image can be split into small patches of fixed size and then linearly embedded. By adding the position embedding and feeding the resulting vector sequence to a standard transformer encoder (e.g., Figure 4B ), downstream applications such as classification can be added.

[0065] Figure 5 5 is a diagram 500 illustrating an example of multiple receptive fields and / or attention windows in a visual transformer. Figure 5 As shown, in an embodiment, an input feature map with multiple channels is segmented into n blocks channel by channel. In an embodiment, such a feature map can be obtained by transforming an image using a neural network. For each block of the feature map, receptive fields of different shapes provide more flexibility. For example, the feature map can be segmented into 4 blocks, each corresponding to a corresponding channel. For each block of the feature map, the receptive field is different (the shaded area in Figure 500).

[0066] The receptive field is an indication of the network's ability to model long-range dependencies. As mentioned above, the related art focuses on extracting global information and ignores local information, resulting in a large loss of texture information. To this end, a shape-fused local window as an attention range is disclosed, which can simultaneously extract important global and local texture information.

[0067] In an embodiment, the feature map may be equally partitioned into n blocks. In an embodiment, the feature map may be partitioned into n blocks with different numbers of channels. In an embodiment, n may be 1, or any integer value equal to or less than the number of channels of a feature map (or a latent space transformed from an input image). In an embodiment, a feature map may be segmented by grouping channels based on channel characteristics. For example, channels with lower variance may be grouped together, and channels with higher variance may be grouped together.

[0068] In an embodiment, the feature map may be segmented according to the order of the channels. In an embodiment, the feature map may be segmented by randomly selecting channels. In an embodiment, the shape of the receptive field may be randomly initialized. In an embodiment, the shape of the receptive field may be predefined. The embodiments are not intended to constrain the size or shape of the receptive field.

[0069] Figure 6is an exemplary flow chart illustrating a method 600 of performing neural image compression using a neural network.

[0070] At operation 605, a multi-channel feature map may be generated for the compressed image.

[0071] In some embodiments, the multi-channel feature map may include one or more tensors generated during a neural image compression process. In some embodiments, the multi-channel feature map may include one or more latent spaces generated during a neural image compression process.

[0072] At operation 610, the multi-channel feature map may be segmented into a first number of segmentation feature maps. In some embodiments, the multi-channel feature map is segmented into the first number of segmentation feature maps evenly, and each segmentation feature map has the same number of channels. In some embodiments, the multi-channel feature map is segmented into the first number of segmentation feature maps, and each segmentation feature map has a different number of channels. In some embodiments, the multi-channel feature map is segmented into the first number of segmentation feature maps based on grouping the channels based on one or more channel characteristics.

[0073] In some embodiments, the one or more channel characteristics include variance or order. In some embodiments, the corresponding attention window shape of the corresponding segmentation feature map is randomly initialized. In some embodiments, the first number is less than or equal to the number of channels in the multi-channel feature map.

[0074] At operation 615, the compressed image may be reconstructed using one or more long-range attention models having one or more variable receptive fields (also referred to as attention windows) associated with the corresponding segmentation feature maps.

[0075] In some embodiments, reconstruction may include selecting a first attention window of a first shape for a first segmentation feature map among a first number of segmentation feature maps; selecting a second attention window of a second shape for a second segmentation feature map among the first number of segmentation feature maps, wherein the first shape and the second shape are different; and concatenating corresponding outputs of one or more remote attention models generated based on the first attention window and the second attention window.

[0076] It is known that the above-described method 600 can be modified to encode images using a neural image compression network.

[0077] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media, or implemented by one or more specially configured hardware processors. Figure 1An environment 100 suitable for implementing various embodiments is shown. In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0078] As used herein, the term component is intended to be broadly interpreted as hardware, software, or a combination of hardware and software.

[0079] Obviously, the systems and / or methods described herein can be implemented in various forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0080] Computer software may be encoded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or through interpretation, microcode execution, etc.

[0081] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0082] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various substitute equivalents that fall within the scope of the present disclosure. It should therefore be appreciated that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method for image compression based on neural network, characterized in that The method is executed by at least one processor, and the method includes: generating a feature map including a plurality of channels for the compressed image; Segmenting the generated feature map into a first number of segmented feature maps in a visual transformer; and The compressed image is reconstructed using one or more long-range attention models having one or more variable receptive fields associated with corresponding segmentation feature maps.

2. The method according to claim 1, characterized in that: Reconstructing the compressed image using the one or more long-range attention models having the one or more variable receptive fields comprises: Selecting a first receptive field of a first shape for a first segmentation feature map among the first number of segmentation feature maps; Selecting a second receptive field of a second shape for a second segmentation feature map among the first number of segmentation feature maps, wherein the first shape and the second shape are different; and The corresponding outputs of the one or more long-range attention models generated based on the first receptive field and the second receptive field are concatenated.

3. The method according to claim 1, characterized in that The generated feature map is evenly divided into the first number of segmentation feature maps, and each segmentation feature map has the same number of channels.

4. The method according to claim 1, characterized in that: The generated feature map is segmented into the first number of segmentation feature maps, wherein each segmentation feature map has a different number of channels.

5. The method according to claim 4, characterized in that The generated feature map is segmented into the first number of segmented feature maps according to grouping the channels based on one or more channel characteristics.

6. The method according to claim 5, characterized in that The one or more channel characteristics include variance or order.

7. The method according to claim 1, characterized in that Randomly initialize the corresponding receptive field shapes of the corresponding segmentation feature maps.

8. The method according to claim 1, characterized in that The first number is less than or equal to the number of channels in the generated feature map.

9. A device for image compression based on neural network, characterized in that The device comprises: at least one memory configured to store computer program code; and At least one processor is configured to read the computer program code and operate according to the instructions of the computer program code, the computer program code comprising: generating code configured to cause the at least one processor to generate a feature map comprising a plurality of channels for the compressed image; segmentation code configured to cause the at least one processor to segment the generated feature map into a first number of segmented feature maps in the visual transformer; and Reconstruction code configured to cause the at least one processor to reconstruct the compressed image using one or more long-range attention models having one or more variable receptive fields associated with corresponding segmentation feature maps.

10. The device according to claim 9, characterized in that The reconstruction code includes: first selection code configured to cause the at least one processor to select a first receptive field of a first shape for a first segmentation feature map among the first number of segmentation feature maps; a second selection code configured to cause the at least one processor to select a second receptive field of a second shape for a second segmentation feature map among the first number of segmentation feature maps; wherein the first shape and the second shape are different; and The connection code is configured to cause the at least one processor to concatenate corresponding outputs of the one or more long-range attention models generated based on the first receptive field and the second receptive field.

11. The device according to claim 9, characterized in that The generated feature map is evenly divided into the first number of segmentation feature maps, and each segmentation feature map has the same number of channels.

12. The device according to claim 9, characterized in that The generated feature map is segmented into the first number of segmentation feature maps, wherein each segmentation feature map has a different number of channels.

13. The device according to claim 12, characterized in that The generated feature map is segmented into the first number of segmented feature maps according to grouping the channels based on one or more channel characteristics.

14. The device according to claim 13, characterized in that The one or more channel characteristics include variance or order.

15. A non-volatile computer-readable medium storing instructions, which, when executed by at least one processor of a device for neural network-based image compression, are characterized in that: causing the at least one processor to: generating a feature map including a plurality of channels for the compressed image; Segmenting the generated feature map into a first number of segmented feature maps in a visual transformer; as well as The compressed image is reconstructed using one or more long-range attention models having one or more variable receptive fields associated with corresponding segmentation feature maps.

16. The non-volatile computer readable medium of claim 15, wherein: Reconstructing the compressed image comprises: Selecting a first receptive field of a first shape for a first segmentation feature map among the first number of segmentation feature maps; Selecting a second receptive field of a second shape for a second segmentation feature map among the first number of segmentation feature maps, wherein the first shape and the second shape are different; and The corresponding outputs of the one or more long-range attention models generated based on the first receptive field and the second receptive field are concatenated.

17. The non-volatile computer readable medium of claim 15, wherein: The generated feature map is evenly divided into the first number of segmentation feature maps, and each segmentation feature map has the same number of channels.

18. The non-volatile computer readable medium of claim 15, wherein: The generated feature map is segmented into the first number of segmentation feature maps, wherein each segmentation feature map has a different number of channels.

19. The non-volatile computer readable medium of claim 18, wherein: The generated feature map is segmented into the first number of segmented feature maps according to grouping the channels based on one or more channel characteristics.

20. The non-volatile computer readable medium of claim 19, wherein: The one or more channel characteristics include variance or order.

Images