Image blending method and system based on semantic understanding of foreground characters

Abstract

The present invention relates to the field of image processing technology and discloses an image blending method based on semantic understanding of foreground characters. The method comprises: employing a joint training mechanism to perform semantic segmentation of foreground characters in an image using a foreground character segmentation network while simultaneously blending the images using an image blending network to obtain semantic information about the foreground characters; transferring the semantic information about the foreground characters from the foreground character segmentation network to the image blending network; and assisting the image blending task in the image blending network using the semantic information about the foreground characters. The present invention can, to a certain extent, alleviate the problem of skin color distortion in foreground characters after blending.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image blending method and system based on semantic understanding of foreground characters. Background Art

[0002] Image blending is to adjust the color style of the foreground image so that the color style of the foreground image is consistent with the background, thereby improving the quality of the composite image.

[0003] Existing image blending methods can be categorized as traditional and deep learning-based. Traditional methods adjust the foreground's color style to align with the background by unifying the image's overall or local color distribution, adjusting the gradient domain, and using multi-scale image data statistics. These traditional methods utilize low-dimensional image features, resulting in limited blending results. Deep learning-based image blending methods primarily employ encoder-decoder algorithms or generative adversarial networks. They typically take as input an image with inconsistent foreground and background colors and a foreground mask, then pass it through a convolutional neural network to output a blended image. The foreground mask simply divides the image into foreground and background, treating the foreground image as a single entity.

[0004] However, the foreground person is composed of different parts and contains rich semantic information. Existing image blending methods do not fully utilize the semantic information of the foreground image, which will cause some problems: lack of semantic understanding of the foreground person, the skin color of the foreground person is interfered with by the background image color during image blending, making the blended skin color of the person look unnatural and not the color tone of normal skin color, which in turn reduces the authenticity of the blended image.

[0005] Based on this, there is an urgent need for a technology that can solve the problem that the existing image blending method easily produces skin color distortion after blending the foreground character. Summary of the Invention

[0006] The present invention provides an image blending method, system, electronic device and storage medium based on semantic understanding of foreground characters, the main purpose of which is to solve the problem that existing image blending methods easily cause skin color distortion of foreground characters.

[0007] To achieve the above-mentioned object, the present invention provides an image blending method based on semantic understanding of foreground characters, which is applied to an electronic device and includes:

[0008] A joint training mechanism is adopted to perform semantic segmentation of foreground people in an image using a foreground person segmentation network while performing image harmonization through an image harmonization network. The foreground people are segmented into different categories according to a preset segmentation rule, and semantic information of the foreground people is obtained based on the different categories of the segmented foreground people.

[0009] Transferring semantic information of the foreground person from the foreground person segmentation network to the image reconciliation network;

[0010] In the image reconciliation network, the semantic information of the foreground person is used to assist the image reconciliation task.

[0011] In order to solve the above problems, the present invention further provides an image blending system based on semantic understanding of foreground characters, the system comprising:

[0012] a joint training unit configured to employ a joint training mechanism to perform semantic segmentation of foreground people in an image using a foreground person segmentation network while performing image harmonization using an image harmonization network, so as to segment the foreground people into different categories according to a preset segmentation rule, and to obtain semantic information of the foreground people according to the different categories of the segmented foreground people;

[0013] a feature transfer unit, configured to transfer semantic information of the foreground person from the foreground person segmentation network to the image harmonization network;

[0014] An auxiliary reconciliation unit is used to assist the image reconciliation task in the image reconciliation network through the semantic information of the foreground person.

[0015] In order to solve the above problems, the present invention also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps in the aforementioned image blending method based on semantic understanding of foreground characters.

[0016] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the above-mentioned image blending method based on semantic understanding of foreground characters.

[0017] The above-mentioned image harmonization scheme based on semantic understanding of foreground characters provided by the present invention obtains the semantic information of foreground characters by adding a branch for foreground character segmentation in the image harmonization network, and harmonizes the foreground characters in combination with the semantic information of the foreground characters. This alleviates the problem of skin color distortion of foreground characters after harmonization to a certain extent, and also improves the objective evaluation indicators mean square error (MSE) and peak signal to noise ratio (PSNR). BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 2 is a flow chart of an image blending method based on semantic understanding of foreground characters according to an embodiment of the present invention;

[0019] Figure 2 is a network structure of an encoder according to an embodiment of the present invention;

[0020] Figure 3 is a network structure of a decoder according to an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of a method for adjusting the inconsistency between the color of a foreground character and the background using a color conversion method according to an embodiment of the present invention to synthesize an image with inconsistent foreground and background colors;

[0022] Figure 5 A feature transfer process according to an embodiment of the present invention;

[0023] Figure 6 1. A block diagram of the logical structure of an image blending system based on semantic understanding of foreground characters according to an embodiment of the present invention;

[0024] Figure 7 A schematic diagram of the internal structure of an electronic device for implementing an image blending method based on semantic understanding of a foreground person according to an embodiment of the present invention;

[0025] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0026] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] Current deep learning-based image blending methods do not distinguish between foreground types. However, when blending composite images with people as the foreground, the skin color of the foreground person is affected by the background image, becoming unnatural and not the expected color tone of normal skin, resulting in distortion. This solution adds a foreground person segmentation branch to the image blending network to obtain semantic information about the foreground person. This information is then used to blend the foreground person, alleviating the skin color distortion issue to a certain extent. It also improves the objective evaluation metrics of mean square error (MSE) and peak signal-to-noise ratio (PSNR).

[0028] The present invention provides an image blending method based on semantic understanding of foreground characters. Figure 1 FIG. 1 is a flow chart of an image blending method based on semantic understanding of foreground characters according to an embodiment of the present invention. The method can be executed by a device that can be implemented by software and / or hardware.

[0029] Figure 1 FIG. 1 is a flow chart of an image blending method based on semantic understanding of foreground characters according to an embodiment of the present invention. Figure 1 As shown, in this embodiment, the image blending method based on semantic understanding of foreground characters includes the following steps:

[0030] S110: Using a joint training mechanism, while performing image harmonization using the image harmonization network, semantically segmenting foreground people in the image using the foreground person segmentation network, so as to segment the foreground people into different categories according to a preset segmentation rule, and obtaining semantic information of the foreground people based on the different categories of the segmented foreground people;

[0031] S120: transferring the semantic information of the foreground person from the foreground person segmentation network to the image harmonization network;

[0032] S130: Assisting the image reconciliation task in the image reconciliation network by using the semantic information of the foreground person.

[0033] The following will further describe in detail the various steps in the above-mentioned image blending method based on semantic understanding of foreground characters through specific implementation examples.

[0034] To incorporate foreground semantic information into the image blending process, this paper innovatively employs a joint training mechanism. While blending, a separate foreground person segmentation network is used to semantically segment foreground characters, thereby acquiring their semantic information. Feature transfer between the two networks is then employed to transfer features from the foreground person segmentation network to the image blending network, assisting with the blending task and making the blended image more realistic.

[0035] Therefore, in step S110, a joint training mechanism is first required to jointly train the image blending network and the foreground person segmentation network. Specifically, in the joint training mechanism, the image blending network and the foreground person segmentation network share the same encoder. The pre-blend image and the foreground person mask are concatenated and encoded by the encoder. The image blending and foreground person semantic segmentation are then performed separately by two decoders.

[0036] More specifically, as an example, the encoder is composed of an eight-layer convolutional neural network; the corresponding two decoders are also composed of eight-layer deconvolutional neural networks; residual connections are used between the encoder and the two decoders to connect the corresponding convolutional layers and deconvolution layers to restore the details in the image.

[0037] Figure 2 and Figure 3 1 and 2 show the network structures of the encoder and decoder according to the embodiment of the present invention respectively. Figure 2 and Figure 3 As shown, in this embodiment, the network structures of the encoder and decoder are both eight-layer structures, wherein the encoder includes a convolution layer, an activation function layer and a normalization layer, and the decoder includes a deconvolution layer, an activation function layer, a normalization layer and an attention module.

[0038] Residual connections are used between the neural networks corresponding to each encoder and decoder layer to ensure that the generated image can restore the details of the input image. Two decoders are used in this paper to perform different tasks: image blending and foreground character segmentation. The output of the image blending network is the blended image, and its loss function is the mean absolute error (MAE) compared to the true value. The output of the foreground character segmentation network is the segmentation result, and its loss function is the cross-entropy loss compared to the true value. The final loss of the entire network is the weighted sum of these two losses.

[0039] Specifically, as an example, after performing image reconciliation through the image reconciliation network, a first image reconciliation result is output, and the image reconciliation loss L can be calculated by comparing the first image reconciliation result with the true value of the preset image reconciliation. rec :

[0040]

[0041] in, is the first image blending result, and I is the true value of the preset image blending.

[0042] Figure 4 2 is a schematic diagram of a method for using a color conversion method to adjust the color of a foreground character to be inconsistent with the background to synthesize an image with inconsistent foreground and background colors according to an embodiment of the present invention.

[0043] like Figure 4 As shown in the figure, given a real image, the foreground person A in the image is segmented out, a reference image is randomly selected, and the foreground person B in the reference image is segmented out. According to the hue of the reference person B, the color of the foreground person A is adjusted using the color conversion method. The image with uncoordinated foreground and background colors is used as the input of the network, and the original real image can be regarded as the true value of the reconciled image.

[0044] In the process of using the foreground person segmentation network to perform semantic segmentation on the foreground person in the image, the foreground person can be segmented into three different categories: clothing, skin, and hair according to the preset segmentation rules, and the semantic information of the foreground person can be obtained according to the different categories of the segmented foreground person.

[0045] Specifically, as an example, the output of the foreground person segmentation network includes three channels 0, 1, and 2, which represent the probability of the pixel being clothing, skin, and hair, respectively. The channel values ​​of channels 0, 1, and 2 are the probabilities of the pixel being clothing, skin, and hair. After using the foreground person segmentation network to perform semantic segmentation on the foreground person in the image, the output includes the three categories of clothing, skin, and hair of the foreground person in pixel units. The loss function of the output of the foreground person segmentation network is:

[0046]

[0047] Where h and w are the height and width of the output image of the foreground person segmentation network, X is the output of the foreground person segmentation network, including three channels, the channel value of the corresponding category is 1, and the other channel values ​​are 0, θ is the true value of the image segmentation; the final loss function L of the entire network is the weighted sum of the loss functions of the image reconciliation network and the foreground person segmentation network:

[0048] L=λ1L rec +λ2L cro

[0049] Among them, λ1 is the weight coefficient of the loss function of the image reconciliation network, and λ2 is the weight coefficient of the loss function of the foreground character segmentation network.

[0050] By simultaneously training the image blending network and the foreground person segmentation network through the loss function L, the network can obtain the semantic information of the foreground person while blending the image.

[0051] After obtaining the semantic information of the foreground person, the feature transfer technology can be used to transfer the semantic information of the foreground person obtained by the foreground person segmentation network to the image harmony network.

[0052] In the image harmony network decoder with a hierarchical structure composed of a series of deconvolution layers, the layers close to the output end are the upper layers of the network, and the layers far from the output end are the lower layers of the network. Among them, the features of the upper layers of the network are related to the low-dimensional features of the output image, that is, the color of each pixel; the features of the lower layers of the network are related to the high-dimensional semantic features of the output image. The image harmony decoder G(z) is expressed as:

[0053] G(z)=g n (g n-1 (…g2(g1(z))…))

[0054] Where z is the image code output by the encoder, n is the number of layers of the image reconciliation decoder, g n is the nth layer of deconvolutional neural network in the image harmony decoder; at the same time, the image harmony decoder G(z) can also be expressed as:

[0055] G(z)=G H (G L (z))

[0056] Among them, G H is the high-level network of the image reconciliation network, G L The lower layers of the image harmonization network.

[0057] In this embodiment, feature transfer is performed at the top layer of the network, that is, between the three layers near the network output. Specifically, as an example, features from the fifth, sixth, and seventh layers of the foreground person segmentation network are transferred to the fifth, sixth, and seventh layers of the image blending network, respectively. After being concatenated with the features from the fifth, sixth, and seventh layers of the image blending network, they are then input into the next layer of the corresponding image blending network.

[0058] Figure 5 FIG. 4 shows a feature transfer process according to an embodiment of the present invention. Figure 5 As shown in Figure 1, the features from the foreground person segmentation network are passed to the image harmony network, concatenated with the features from the image harmony network, and input into the next layer of the corresponding image harmony network. If the dimensions of the foreground person semantic features output by the foreground person segmentation network are represented as (hi, wi, ci), and the dimensions of the image harmony features output by the image harmony network are represented as (hi, wi, ci), then the dimensions of the features formed after the concatenation are represented as (hi, wi, 2ci), where hi and wi are the height and width of the features of the i-th layer, and ci is the number of feature channels of the i-th layer.

[0059] In the image harmonization network, an attention module can be used to process features from different networks. The deconvolution layer near the output of the image harmonization network needs to simultaneously receive three different features: the image harmonization feature from the previous layer, the residual connection feature from the encoder, and the foreground semantic feature from the foreground person segmentation network. After receiving these three features, the attention module first concatenates them together. Then, a 1×1 convolution layer is passed to obtain a feature map with unchanged feature dimension. This feature map is the learned weight value of each feature.

[0060] f m =A i (f con )

[0061] Among them, the 1×1 convolution layer is A i (), f m is the feature map, f con It is a cascade feature;

[0062] f out =f m *f co n

[0063] Because f m With f con The dimensions are the same, so they can be multiplied element-wise, f out is the output feature of the attention module.

[0064] The cascaded features are element-wise multiplied with the feature map as the output of the attention module, which is the input of the next deconvolution layer of the image harmony network for subsequent image harmony processing.

[0065] In a specific embodiment of the present invention, the output results of the existing image blending method and the image blending method after applying the present invention are compared, and the comparison results are shown in the following table:

[0066]

[0067] It can be seen that by applying the image blending method based on semantic understanding of foreground characters of the present invention, the mean square error of the output image is significantly reduced, the peak signal-to-noise ratio is improved, and the problem of skin color distortion of foreground characters after blending is effectively alleviated.

[0068] As can be seen from the above examples, the image blending method based on foreground person semantic understanding proposed by the present invention, by adding a foreground person segmentation branch to the image blending network, obtains the foreground person's semantic information, and blends the foreground person based on this semantic information. This alleviates the problem of skin color distortion after foreground blending to a certain extent, and also improves the objective evaluation indicators of mean square error (MSE) and peak signal-to-noise ratio (PSNR). This effectively solves the problem of foreground person skin color distortion caused by the background image, which is caused by existing deep learning-based image blending methods not distinguishing foreground types.

[0069] Corresponding to the above-mentioned image blending method based on semantic understanding of foreground characters, the present invention also provides an image blending system based on semantic understanding of foreground characters. Figure 6 The functional modules of the image blending system based on foreground person semantic understanding according to an embodiment of the present invention are shown.

[0070] like Figure 6 As shown, the image blending system 600 based on foreground person semantic understanding provided by the present invention can be installed in an electronic device. Depending on the functionality implemented, the image blending system based on foreground person semantic understanding can include a joint training unit 610, a feature transfer unit 620, and an auxiliary blending unit 630. The units described in the present invention, also known as modules, refer to a series of computer program segments that can be executed by a processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0071] In this embodiment, the functions of each module / unit are as follows:

[0072] a joint training unit 610 configured to employ a joint training mechanism to perform semantic segmentation of foreground people in an image using a foreground person segmentation network while performing image harmonization using the image harmonization network, thereby segmenting the foreground people into different categories according to a preset segmentation rule, and obtaining semantic information of the foreground people based on the different categories of the segmented foreground people;

[0073] A feature transfer unit 620, configured to transfer the semantic information of the foreground person from the foreground person segmentation network to the image harmonization network;

[0074] The auxiliary reconciliation unit 630 is used to assist the image reconciliation task in the image reconciliation network through the semantic information of the foreground person.

[0075] As an example, the joint training unit includes an encoder and two decoders. The image blending network and the foreground person segmentation network share the same encoder. The pre-blend image and the foreground person mask are concatenated and encoded by the encoder. The image blending and foreground person semantic segmentation are then performed by two decoders, respectively. In one specific embodiment of the present invention, the encoder comprises an eight-layer convolutional neural network, and the two decoders each comprise an eight-layer deconvolutional neural network. Residual connections are used between the encoder and the two decoders to connect the corresponding convolutional layers and deconvolution layers to restore image details.

[0076] More specific implementations of the image harmonization system based on foreground person semantic understanding provided in this application can all refer to the above-mentioned embodiment of the image harmonization method based on foreground person semantic understanding.

[0077] As can be seen from the above examples, the image blending system based on semantic understanding of foreground characters proposed in this application, by adding a branch for foreground character segmentation to the image blending network, obtains semantic information of foreground characters, and blends the foreground characters based on this semantic information. This alleviates the problem of skin color distortion of foreground characters after blending to a certain extent, and also improves the objective evaluation indicators mean square error (MSE) and peak signal-to-noise ratio (PSNR). This effectively solves the problem of foreground character skin color distortion caused by the background image, which is caused by the existing deep learning-based image blending methods not distinguishing the type of foreground.

[0078] Figure 7 Schematic diagram of the internal structure of an electronic device for implementing an image blending method based on foreground character semantic understanding according to an embodiment of the present invention. Figure 7 As shown, the electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as an image blending program 12 based on semantic understanding of foreground characters.

[0079] The memory 11 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1. Furthermore, the memory 11 may include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can be used not only to store application software installed in the electronic device 1 and various types of data, such as the code of an image blending program based on semantic understanding of foreground characters, but also to temporarily store data that has been output or is about to be output.

[0080] In some embodiments, the processor 10 may be comprised of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and circuits. It executes or runs programs or modules stored in the memory 11 (such as an image blending program based on semantic understanding of foreground characters) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0081] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory 11 and at least one processor 10, etc.

[0082] Figure 7 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 7The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0083] For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for powering the various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0084] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0085] Optionally, the electronic device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0086] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0087] The memory 11 in the electronic device 1 is a computer-readable storage medium storing at least one instruction. The at least one instruction is executed by a processor in the electronic device to implement the aforementioned image blending method based on foreground person semantic understanding. Specifically, as an example, the image blending program 12 based on foreground person semantic understanding stored in the memory 11 is a combination of multiple instructions. When executed by the processor 10, it can implement the following steps:

[0088] S110: Using a joint training mechanism, while performing image harmonization using the image harmonization network, semantically segmenting foreground people in the image using the foreground person segmentation network, so as to segment the foreground people into different categories according to a preset segmentation rule, and obtaining semantic information of the foreground people based on the different categories of the segmented foreground people;

[0089] S120: transferring the semantic information of the foreground person from the foreground person segmentation network to the image harmonization network;

[0090] S130: Assisting the image reconciliation task in the image reconciliation network by using the semantic information of the foreground person.

[0091] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0092] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0093] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0094] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0095] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0096] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0097] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Second-order terms are used to indicate names and do not imply any particular order.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An image blending method based on semantic understanding of foreground characters, applied to electronic devices, characterized in that: The method comprises: A joint training mechanism is adopted, in which a foreground person segmentation branch is added to the image harmonization network. While performing image harmonization through the image harmonization network, a foreground person segmentation network is used to perform semantic segmentation on the foreground person in the image, so as to segment the foreground person into different categories according to a preset segmentation rule, and obtain semantic information of the foreground person according to the different categories of the segmented foreground person. The image harmonization network and the foreground person segmentation network share the same encoder, the pre-harmonic image and the foreground person mask are concatenated together and encoded by the encoder, and then image harmonization and foreground person semantic segmentation are performed respectively by two decoders. The encoder is composed of an eight-layer convolutional neural network, and the two decoders are both composed of an eight-layer deconvolutional neural network. A residual connection is used between the encoder and the two decoders to connect the corresponding convolutional layer and the deconvolution layer to restore the details in the image. Transferring semantic information of the foreground person from the foreground person segmentation network to the image reconciliation network; wherein, feature transfer is performed at higher layers of the foreground person segmentation network and the image reconciliation network; wherein, features of the fifth, sixth, and seventh layers of the foreground person segmentation network are respectively transferred to the fifth, sixth, and seventh layers of the image reconciliation network, cascaded with features of the fifth, sixth, and seventh layers of the image reconciliation network, and then input into the corresponding next layer of the image reconciliation network; In the image reconciliation network, the semantic information of the foreground person is used to assist the image reconciliation task.

2. The image blending method based on foreground character semantic understanding according to claim 1, characterized in that: After performing image reconciliation through the image reconciliation network, a first image reconciliation result is output, and the image reconciliation loss is calculated by comparing the first image reconciliation result with the true value of the preset image reconciliation. : in, is the first image blending result, The true value of the preset image reconciliation.

3. The image blending method based on foreground character semantic understanding according to claim 2, characterized in that: The output of the foreground person segmentation network includes three channels 0, 1, and 2 representing the probabilities of the pixel being clothing, skin, and hair, respectively. The channel values ​​of channels 0, 1, and 2 are the probabilities of the pixel being clothing, skin, and hair. In the process of using the foreground person segmentation network to perform semantic segmentation on the foreground person in the image, the output of the foreground person segmentation network is the result of segmenting the foreground person into three categories: clothing, skin, and hair in units of pixels. The loss function of the output of the foreground person segmentation network is: Wherein, h, w are the height and width of the output image of the foreground person segmentation network, X is the output of the foreground person segmentation network, including three channels, the channel value of the corresponding category is 1, and the other channel values ​​are 0, is the true value of image segmentation; the final loss function L of the entire network is the weighted sum of the loss functions of the image reconciliation network and the foreground character segmentation network: in, is the weight coefficient of the loss function of the image reconciliation network, is the weight coefficient of the loss function of the foreground person segmentation network.

4. The image blending method based on foreground character semantic understanding according to claim 3, characterized in that: In the process of transferring the semantic information of the foreground person from the foreground person segmentation network to the image reconciliation network, In the image harmony decoder with a hierarchical structure composed of a series of deconvolution layers, the layer close to the output end is the high-level network layer, and the layer far from the output end is the low-level network layer; wherein, the features of the high-level network layer are related to the low-dimensional features of the output image, and the features of the low-level network layer are related to the high-dimensional semantic features of the output image. Expressed as: Where z is the image code output by the encoder, n is the number of layers of the image harmonization decoder, is the nth layer of deconvolution neural network in the image harmony decoder; at the same time, the image harmony decoder Also expressed as: in, is the high-level network of the image reconciliation network, The lower layers of the image harmonization network.

5. The image blending method based on foreground character semantic understanding according to claim 4, characterized in that: The process of assisting the image reconciliation task by using the semantic information of the foreground person in the image reconciliation network includes: In the image reconciliation network, an attention module is used to process features from different networks; wherein, The deconvolution layer near the output end of the image harmony network simultaneously receives three different features: the image harmony feature from the previous layer, the residual connection feature from the encoder, and the foreground semantic feature from the foreground person segmentation network; The image harmonization network concatenates the three features and passes them through a 1×1 convolutional layer to obtain a feature map with unchanged feature dimension as the weight value of each feature, and obtains: Among them, the 1×1 convolution layer is , is the feature map, is the cascade feature; then the output feature of the attention module for: Finally, the concatenated features are element-wise multiplied with the feature map as the output of the attention module and the input of the next deconvolution layer.

6. An image blending system based on semantic understanding of foreground characters, characterized by: The system comprises: A joint training unit is configured to adopt a joint training mechanism, by adding a foreground person segmentation branch to an image harmonization network, and simultaneously performing image harmonization through the image harmonization network and semantically segmenting the foreground person in the image using the foreground person segmentation network, so as to segment the foreground person into different categories according to a preset segmentation rule, and obtain semantic information of the foreground person according to the different categories of the segmented foreground person; wherein the image harmonization network and the foreground person segmentation network share the same encoder, the pre-harmonic image and the foreground person mask are concatenated together and encoded by the encoder, and then image harmonization and foreground person semantic segmentation are performed respectively by two decoders; the encoder is composed of an eight-layer convolutional neural network, and the two decoders are both composed of an eight-layer deconvolutional neural network; a residual connection is used between the encoder and the two decoders to connect the corresponding convolutional layer and the deconvolution layer to restore details in the image; a feature transfer unit, configured to transfer semantic information of the foreground person from the foreground person segmentation network to the image harmony network; wherein the feature transfer is performed at a higher layer of the foreground person segmentation network and the image harmony network; wherein the features of the fifth, sixth, and seventh layers of the foreground person segmentation network are respectively transferred to the fifth, sixth, and seventh layers of the image harmony network, cascaded with the features of the fifth, sixth, and seventh layers of the image harmony network, and then input into the corresponding next layer of the image harmony network; An auxiliary reconciliation unit is used to assist the image reconciliation task in the image reconciliation network through the semantic information of the foreground person.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps in the image blending method based on foreground character semantic understanding as described in any one of claims 1 to 5.

Images