An image and harmonic method and system based on global perception dynamic kernel
Patent Information
- Application Number
- CN202310268331.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-20
- Publication Date
- 2026-10-23
- Estimated Expiration
- 2043-03-20
AI Technical Summary
然而，基于局部调整的方法未获得和谐化中重要的全局感知；成图像的邻近先验缺少关注，忽略对于局部特征的建模，基于全局迁移的方法将背景参考进行了一致性整体看待，忽略了背景和前景的内部信息，从而使得背景参考不具有针对性
[0034]本发明将动态可预测核的方法在图像和谐化中加以应用，通过像素级别的可预测核，对当前像素进行色调、光照进行和谐化处理时更关注窗口区域的内部特征；本发明的方法在预测动态核过程中引入全局信息采用注意力机制，在光线复杂的背景参考中能够提取适应的背景参考信息；本发明设计全局感知的和谐化核方案，实现对于全局和局部信息的交互参考，将全局信息更好的与局部信息进行融合，更有利于适用于实用场景的图像和谐化。
Smart Images
Figure CN116385598B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, specifically to an image harmonization method and system based on a globally perceptive dynamic kernel. Background Technology
[0002] With the further development of the mobile internet era and the widespread adoption of 5G technology, the number of users on social media platforms is constantly rising, and everyone is becoming a content producer, leading to increased demand for post-production image editing. Today, the dimensions of information presentation are no longer limited to simple text. Images, as a crucial carrier of current needs, will see image editing and production become a major demand. However, the development of image content production tools has lagged far behind the development of image content, necessitating more targeted image editing and production tools. Existing professional image post-production editing tools have high entry barriers and cannot meet the needs of the general public. As a common requirement in image editing, image compositing aims to cut out the foreground from one image and composite it into another background. More generally, image compositing combines foregrounds from multiple different sources to form a composite image, which has numerous applications in augmented reality, artistic creation, and e-commerce. However, the composite image may lack realism, thus reducing quality and making it easily distinguishable to the naked eye. Image harmonization aims to solve the visual disharmony caused during image compositing. Existing methods can be mainly divided into two types: those based on local feature adjustment and those based on global transfer. However, methods based on local adjustments fail to achieve the crucial global awareness required for harmonization; they neglect the proximity prior in image synthesis, ignoring the modeling of local features; and methods based on global transfer treat the background reference as a uniform whole, ignoring the internal information of the background and foreground, thus rendering the background reference unspecific. Based on this, this invention aims to address the visual dissonance caused during image synthesis by fusing and interacting global and local information, proposing an image harmonization method based on a globally aware dynamic kernel. Summary of the Invention
[0003] To address the aforementioned problems, this invention proposes an image harmonization method and system based on a globally perceptive dynamic kernel. It employs a dynamically learned perceptual kernel approach, establishing local and global references based on the dynamic kernel, thus overcoming the issues encountered by existing methods in selecting background references. The neural network can be specifically divided into two parts: a globally perceptual kernel prediction module and a region harmonization modulation module. The perceptual kernel learning part further includes a global feature extraction module and a long-range information injection module. The global feature extraction module models long-range reference information in image harmonization, while the long-range information matching module injects long-range information suitable for local harmonization. The kernel modulation module utilizes the dynamic kernel obtained from the perceptual kernel learning module to achieve the application of global references for local harmonization processing.
[0004] To achieve the above objectives, this invention provides an image harmonization method based on a globally perceptive dynamic kernel, comprising the following steps:
[0005] S1. Global content learning deep feature learning dynamic perception kernel based on synthetic images;
[0006] S2. Implement local harmonic modulation at multiple levels of the decoder, and achieve multi-level local feature harmony in the learned dynamic sensing kernel domain features of the decoder.
[0007] Preferably, the construction process of the dynamic perception kernel is as follows: first, a global long-distance information extraction module is modeled, and then an attention mechanism module is designed based on deep features and long-distance information, so that when long-distance information is introduced, it interacts with local features to obtain the preferred global information, and the dynamic perception kernel is obtained based on the harmonized kernel prediction network.
[0008] Preferably, step S1 specifically includes the following steps:
[0009] S11. The deepest features obtained by the encoder are sent to the long-distance information extraction module to obtain long-distance information fused with global features based on several Transformer layer structures.
[0010] S12. The long-range information is fused with the hierarchical features in the encoder, and the global dynamic perception kernel and the next-level features in the harmonized kernel prediction network are obtained based on the multi-layer harmonized kernel prediction network module. This can be expressed by the following formula:
[0011]
[0012] Here, HKPLayer(·) is the harmonization kernel prediction network layer. For each level of the harmonization network prediction layer, we will input the features from the previous level. (For the first layer, we input the long-range feature F) LR and the l-th layer features in the encoder Output features for the next level kernel prediction and the global perception kernel K l .
[0013] Preferably, the process of designing an attention mechanism module based on deep features and long-range information is as follows: for the features in the harmonization kernel prediction network... and Firstly, vector extraction is performed based on convolutional layers and multilayer perceptrons. Then divide them into n groups to obtain Therefore, the attention matrix A is calculated as follows:
[0014]
[0015] Where ⊙ represents the inner product operation;
[0016] The channel attention vectors applied to the feature layers are calculated through convolutional layers. Finally, based on the original extracted vector α∈{α HKP ,α E}and The channel attention is calculated as follows:
[0017]
[0018] Where b is a learnable parameter, and σ represents the sigmoid function, which, when applied as the channel dimension of the input feature, can be written as follows:
[0019]
[0020] Where · represents matrix multiplication, Upsample(·) represents upsampling, and Conv(·) represents a convolutional layer.
[0021] Preferably, the process of implementing local harmonic modulation at the l-th layer in the multi-level decoder in step S2 can be expressed by the following formula:
[0022]
[0023] in, This is represented as the modulation process of the harmonization nucleus. This is represented as the l-th layer feature in the decoder. This is the corresponding harmonic modulation result. Let N be the predicted dynamic harmonicization kernel of size N applied to the l-th layer. For the pixel-level harmonicization modulation process, the formula is extended to the following form:
[0024]
[0025] Where p and q are corresponding elements in the image features. For the harmonicization kernel applied to element p, To correspond to nearby elements in the feature map, the encoder and decoder structures are implemented using U-Net, and the decoder layer of U-Net is defined as φ. D (·), thus the harmonization nuclear modulation process can be expressed by the formula:
[0026]
[0027] Where M is the input mask in image harmonicization, and I c This is a composite image.
[0028] Applying the above method, the present invention also provides an image harmonization system based on a globally perceptive dynamic kernel, including a main network structure module, a globally perceptive kernel prediction module, and a regional harmonization modulation module.
[0029] Preferably, the main network structure module consists of a decoder and an encoder, with U-Net as its main structure, wherein several attention modules are integrated in the decoder part.
[0030] Preferably, the global perception kernel prediction module includes a global feature extraction module and a long-distance information injection module.
[0031] Preferably, the global feature extraction module uses the deepest features in the encoder to model long-distance information based on the Transformer structure. The long-distance information injection module designs an attention mechanism to fuse the long-distance information obtained from the deepest features with the features in the encoder, and uses the fused features to predict the dynamic sensing kernel used in the region harmonization modulation.
[0032] Preferably, the region harmonization modulation module is used to achieve region harmonization in the image feature layer by utilizing the predicted global perception kernel, so as to achieve region harmonization processing when simultaneously focusing on long-distance information.
[0033] Compared with the prior art, the beneficial effects of the present invention are:
[0034] This invention applies the method of dynamic predictable kernels to image harmonization. By using a pixel-level predictable kernel, it focuses more on the internal features of the window region when harmonizing the tone and illumination of the current pixel. The method of this invention introduces global information and adopts an attention mechanism in the process of predicting the dynamic kernel, which can extract adaptive background reference information in complex lighting background references. This invention designs a globally aware harmonization kernel scheme to realize interactive reference of global and local information, better integrate global information with local information, and is more conducive to image harmonization applicable to practical scenarios. Attached Figure Description
[0035] Figure 1 A schematic diagram illustrating the principle of an image harmonization method based on a globally perceptive dynamic kernel provided by this invention;
[0036] Figure 2 The image shows the effect of an image harmonization method based on a globally perceptive dynamic kernel provided by this invention. Detailed Implementation
[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] Existing image harmonization methods often neglect the proximity prior of the synthesized image and ignore the modeling of local features. This invention applies a dynamically predictable kernel method to image harmonization. By using a pixel-level predictable kernel to harmonize the tone and illumination of the current pixel, it focuses more on the internal features of the window region. Existing image harmonization methods include two background reference schemes. One is a global matching method that ignores the internal information of both the background and foreground, using statistical matching to achieve harmonization. This invention's method introduces global information and employs an attention mechanism during the prediction of the dynamic kernel, enabling the extraction of adaptive background reference information even in complex lighting environments. The other scheme is limited by the field of view of convolutional networks. This invention designs a globally perceptive harmonization kernel scheme, achieving interactive reference between global and local information, better integrating global and local information, and making it more suitable for practical image harmonization scenarios.
[0039] This invention proposes an image harmonization method based on a globally perceptive dynamic kernel, such as... Figure 1 The diagram shown illustrates the principle of an image harmonization method based on a globally perceptive dynamic kernel provided by this invention, comprising the following steps:
[0040] S1. Global content learning deep feature learning dynamic perception kernel based on synthetic images;
[0041] S2. Implement local harmonic modulation at multiple levels of the decoder, and achieve multi-level local feature harmony in the learned dynamic sensing kernel domain features of the decoder.
[0042] The following is a detailed explanation of each step.
[0043] Specifically, the construction process of the dynamic perception kernel is as follows: First, a global long-distance information extraction module is modeled. Then, an attention mechanism module is designed based on deep features and long-distance information, so that it interacts with local features when introducing long-distance information to obtain optimized global information. The dynamic perception kernel is obtained based on the harmonized kernel prediction network.
[0044] Specifically, step S1 includes the following steps:
[0045] S11. The deepest features obtained by the encoder are sent to the long-distance information extraction module to obtain long-distance information fused with global features based on several Transformer layer structures.
[0046] S12. The long-range information is fused with the hierarchical features in the encoder, and the global dynamic perception kernel and the next-level features in the harmonized kernel prediction network are obtained based on the multi-layer harmonized kernel prediction network module. This can be expressed by the following formula:
[0047]
[0048] Here, HKPLayer(·) is the harmonization kernel prediction network layer. For each level of the harmonization network prediction layer, we will input the features from the previous level. (For the first layer, we input the long-range feature F) LR ) and the l-th layer features in the encoder Output features for the next level of kernel prediction and the global perception kernel K l .
[0049] Specifically, the process of designing an attention mechanism module based on deep features and long-range information is as follows: for the features in the harmonization kernel prediction network... and Firstly, vector extraction is performed based on convolutional layers and multilayer perceptrons. Then divide them into n groups to obtain Therefore, the attention matrix A is calculated as follows:
[0050]
[0051] Where ⊙ represents the inner product operation;
[0052] The channel attention vectors applied to the feature layers are calculated through convolutional layers. Finally, based on the original extracted vector α∈{α HKP ,α E}and The channel attention is calculated as follows:
[0053]
[0054] Where b is a learnable parameter, and σ represents the sigmoid function, which, when applied as the channel dimension of the input feature, can be written as follows:
[0055]
[0056] Where · represents matrix multiplication, Upsample(·) represents upsampling, and Conv(·) represents a convolutional layer.
[0057] Specifically, the process of implementing local harmonic modulation at the l-th layer in the multi-level decoder in step S2 can be expressed by the following formula:
[0058]
[0059] in, This is represented as the modulation process of the harmonic nucleus. This is represented as the l-th layer feature in the decoder. This is the corresponding harmonic modulation result. Let N be the predicted dynamic harmonicization kernel of size N applied to the l-th layer. For the pixel-level harmonicization modulation process, the formula is extended to the following form:
[0060]
[0061] Where p and q are corresponding elements in the image features. For the harmonicization kernel applied to element p, To correspond to nearby elements in the feature map, the encoder and decoder structures are implemented using U-Net, and the decoder layer of U-Net is defined as φ. D (·), thus the harmonization nuclear modulation process can be expressed by the formula:
[0062]
[0063] Where M is the input mask in image harmonicization, and I c This is a composite image.
[0064] This invention discloses an image harmonicization method based on a globally perceptive dynamic kernel, implemented using the PyTorch deep learning framework. The image harmonicization generator is trained on a supervised dataset. Training images are in the form of image pairs, where each synthesized image corresponds to its ground truth image and contains a mask for the corresponding foreground region. During training, the mask and the synthesized image are input together and used for model training. After training, the model is stored. Figure 2 The image shown is an effect diagram of an image harmonization method based on a globally perceptive dynamic kernel provided by the present invention.
[0065] This invention also provides an image harmonization system based on a globally perceptive dynamic kernel, comprising the following modules:
[0066] The main network structure module consists of a decoder and an encoder, with U-Net as its main structure. Several attention modules are integrated into the decoder part.
[0067] A global sensing kernel prediction module is included to predict the global sensing kernel used in the local harmonic modulation module. This module comprises a global feature extraction module and a long-range information injection module. The global feature extraction module uses the deepest features from the encoder to model long-range information based on a Transformer structure. The long-range information injection module employs an attention mechanism to fuse the long-range information obtained from the deepest features with the features from the encoder, making the injection of long-range information more efficient and targeted. Finally, the fused features are used to predict the dynamic sensing kernel applied in regional harmonic modulation.
[0068] The regional harmonic modulation module, located in the decoder section, aims to achieve regional harmonic processing by utilizing the predicted global perception kernel at the image feature layer, thereby enabling simultaneous attention to long-distance information while performing harmonic processing on regions.
[0069] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.
Claims
1. An image harmonization method based on a globally perceptive dynamic kernel, characterized in that, Includes the following steps: S1. Global content learning deep feature learning dynamic perception kernel based on synthetic images; S2. Implement local harmonic modulation at multiple levels of the decoder, and achieve multi-level local feature harmony in the features of the learned dynamic sensing kernel domain decoder. The construction process of the dynamic perception kernel is as follows: First, a global long-distance information extraction module is modeled. Then, an attention mechanism module is designed based on deep features and long-distance information, so that it interacts with local features when introducing long-distance information to obtain optimized global information. The dynamic perception kernel is obtained based on the harmonized kernel prediction network. Step S1 specifically includes the following steps: S11. The deepest features obtained by the encoder are sent to the long-distance information extraction module to obtain long-distance information fused with global features based on several Transformer layer structures. S12. The long-range information is fused with the hierarchical features in the encoder, and the global dynamic perception kernel and the next-level features in the harmonized kernel prediction network are obtained based on the multi-layer harmonized kernel prediction network module. This can be expressed by the following formula: ， in, It is a harmonization kernel prediction network layer. For each level of the harmonization network prediction layer, the input is the feature of the previous level. and the l-th layer features in the encoder Output features for the next level of kernel prediction. and global perception kernel .
2. The image harmonization method based on a globally perceptive dynamic kernel according to claim 1, characterized in that, The process of designing an attention mechanism module based on deep features and long-range information is as follows: for the features in the harmonization kernel prediction network... and Firstly, vector extraction is based on convolutional layers and multilayer perceptrons. Then divide them into n groups to obtain Therefore, the attention matrix is calculated. : ， in, Indicates inner product operation; The channel attention vectors applied to the feature layers are calculated through convolutional layers. Ultimately, based on the original extracted vector and The channel attention is calculated as follows: ， Where b is a learnable parameter, Represented as the sigmoid function, when applied as the channel dimension of the input features, it can be written as follows: ， in, Represented as matrix dot product, This is represented as upsampling. This is represented as a convolutional layer.
3. The image harmonization method based on a globally perceptive dynamic kernel according to claim 2, characterized in that, The process of implementing local harmonic modulation at the l-th layer in the multi-level decoder in step S2 can be expressed by the following formula: ， in, This is represented as the modulation process of the harmonization nucleus. This is represented as the l-th layer feature in the decoder. This is the corresponding harmonic modulation result. Let N be the predicted dynamic harmonicization kernel of size N applied to the l-th layer. For the pixel-level harmonicization modulation process, the formula is extended to the following form: ， in and For the corresponding elements in the image features, To be applied to elements The core of harmony, To correspond to nearby elements in the feature map, the encoder and decoder structures are implemented using U-Net, and the decoder layer of U-Net is defined as follows: Therefore, the harmonization nuclear modulation process can be expressed by the following formula: ， Where M is the input mask in image harmonization. This is a composite image.
4. An image harmonization system based on a globally perceptive dynamic kernel, applied in an image harmonization method based on a globally perceptive dynamic kernel as described in any one of claims 1-3, characterized in that, It includes the main network structure module, the global perception kernel prediction module, and the regional harmonic modulation module.
5. The image harmonization system based on a globally perceptive dynamic kernel according to claim 4, characterized in that, The main network structure module consists of a decoder and an encoder, with U-Net as its main structure, and several attention modules are integrated in the decoder part.
6. The image harmonization system based on a globally perceptive dynamic kernel according to claim 4, characterized in that, The global perception kernel prediction module includes a global feature extraction module and a long-distance information injection module.
7. The image harmonization system based on a globally perceptive dynamic kernel according to claim 6, characterized in that, The global feature extraction module uses the deepest features in the encoder to model long-distance information based on the Transformer structure. The long-distance information injection module designs an attention mechanism to fuse the long-distance information obtained from the deepest features with the features in the encoder. The fused features are then used to predict the dynamic sensing kernel used in the region harmonization modulation.
8. The image harmonization system based on a globally perceptive dynamic kernel according to claim 7, characterized in that, The region harmonization modulation module is used to achieve region harmonization in the image feature layer by utilizing the predicted global perception kernel, so as to achieve region harmonization processing when focusing on long-distance information at the same time.