Image fusion method, device and storage medium

Abstract

The present invention relates to the field of image processing, and specifically provides an image fusion method, device and storage medium, aiming to solve the technical problem that the text effect obtained by the existing image fusion method is poor. To this end, the image fusion method of the present invention includes the following steps: obtaining a first style image and a text image to be transferred, the first style image includes a first text content, the text image to be transferred includes a second text content, and the style patterns of the first style image and the text image to be transferred are different; erasing the first text content of the first style image based on a text erasure model to obtain a background image; obtaining a second style image based on the background image and the first style image; and using a text style transfer fusion model to fuse the second style image, the text image to be transferred and the background image to obtain a current fused image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and specifically provides an image fusion method, device and storage medium. Background Art

[0002] At present, the traditional image fusion method mainly uses the inpaint method in OpenCV to erase the text content in the slice after binarizing the text area, and then uses the preset model to analyze the font, composition pattern, color and height of the original text, generate the corresponding field content and paste it on the erased slice background, and then restore it to the original image to generate new data for training. However, inpaint has a good effect on erasing images with solid color backgrounds. If the background has a texture structure, the background will disappear completely after erasing, and there will be obvious boundary features when training the detection model, so the trained model has poor robustness. In addition, the field synthesized based on the field content related information obtained by the preset model has poor effect, especially when the height and color of the characters are too different from the background itself, which will affect the synthesis effect to a certain extent.

[0003] Accordingly, the art needs a new image fusion solution to solve the above problems. Summary of the invention

[0004] In order to overcome the above defects, the present invention is proposed to solve or at least partially solve the technical problem that the text effect obtained by the existing image fusion method is poor. The present invention provides an image fusion method, device and storage medium.

[0005] In a first aspect, the present invention provides an image fusion method, comprising the following steps: obtaining a first style image and a text image to be transferred, the first style image including a first text content, the text image to be transferred including a second text content, and the first style image and the text image to be transferred having different style patterns; erasing the first text content of the first style image based on a text erasure model to obtain a background image; obtaining a second style image based on the background image and the first style image; and fusing the second style image, the text image to be transferred and the background image using a text style transfer fusion model to obtain a current fused image.

[0006] In one embodiment, acquiring the second style image based on the background image and the first style image includes: performing a difference calculation on the background image and the first style image to obtain the second style image.

[0007] In one embodiment, the second style image, the text image to be migrated and the background image are fused using a text style transfer fusion model to obtain a current fused image, including: using a text style transfer fusion model to migrate the second text content of the text image to be migrated to the second style image in the style of the second style image to obtain a migrated text image; and fusing the migrated text image with the background image to obtain a current fused image.

[0008] In one embodiment, a text style transfer fusion model is used to migrate the second text content of the text image to be migrated to the second style image in the style style of the second style image to obtain the migrated text image, which includes: using a text style transfer fusion model, migrating the second text content of the text image to be migrated to a target position in the second style image in the style style of the second style image to obtain the migrated text image, wherein the target position is a preset position in the second style image.

[0009] In one embodiment, the text erasure model includes a generative adversarial network, and the text style transfer fusion model includes a Resnet network.

[0010] In one embodiment, the method further includes: extracting text content and boundary features of the current fused image using a feature extraction network; splicing the current fused image and the extracted text content and boundary features to obtain a spliced ​​image; and performing convolution processing on the spliced ​​image to obtain a final fused image.

[0011] In one embodiment, the feature extraction network comprises an encoder and a decoder, the encoder comprises at least one convolutional layer, and the decoder comprises at least one deconvolutional layer.

[0012] In a second aspect, the present invention provides an image fusion device, comprising: an input module, configured to acquire a first style image and a text image to be migrated, the first style image including a first text content, the text image to be migrated including a second text content, and the first style image and the text image to be migrated have different style patterns; an erasing module, configured to erase the first text content of the first style image based on a text erasure model to obtain a background image; an acquisition module, configured to acquire a second style image based on the background image and the first style image; and a fusion module, configured to fuse the second style image, the text image to be migrated and the background image using a text style migration fusion model to obtain a current fused image.

[0013] In a third aspect, an electronic device is provided, the electronic device comprising a processor and a storage device, the storage device being suitable for storing a plurality of program codes, the program codes being suitable for being loaded and run by the processor to execute any of the aforementioned image fusion methods.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored in the computer-readable storage medium, wherein the program codes are suitable for being loaded and run by a processor to execute any of the aforementioned image fusion methods.

[0015] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0016] The image fusion method of the present invention first uses a text erasure model to erase the text content of a first style image to obtain a background image, and then uses a text style transfer fusion model to fuse the second style image with the background image deleted, the text image to be transferred, and the background image with the text information erased, and finally obtains the current fused image, thereby avoiding the problem of poor effect of text images synthesized using existing image fusion methods and improving the accuracy of image fusion.

[0017] By performing a differential calculation on the background image and the first style image, a second style image containing only text content is obtained. This method of highlighting only the text itself is conducive to improving the accuracy of extracting the text skeleton image by the subsequent text style transfer fusion model.

[0018] On the basis of obtaining the current fused image, the text information and boundary information of the current fused image are further optimized. Specifically, the text content and boundary information of the current fused image are extracted, and the extracted information is spliced ​​with the current fused image and then convolution is performed to obtain the final fused image, thereby achieving the enhancement effect of the text information and boundary information of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which:

[0020] Figure 1 is a flowchart of an image fusion method according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of a text erasure model structure according to an embodiment of the present invention;

[0022] Figure 3is a schematic diagram of the structure of a text style transfer fusion model according to an embodiment of the present invention;

[0023] FIG4( a ) is a schematic diagram of a first style image according to an embodiment of the present invention;

[0024] FIG4( b ) is a schematic diagram of a background image according to an embodiment of the present invention;

[0025] FIG5( a ) is a schematic diagram of a second style image according to an embodiment of the present invention;

[0026] FIG5( b ) is a schematic diagram of a current fused image according to an embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of a process of optimizing a current fused image according to an embodiment of the present invention;

[0028] Figure 7 is a schematic structural diagram of an image fusion device according to an embodiment of the present invention.

[0029] Reference numerals list :

[0030] 11: Input module; 12: Erasing module; 13: Acquisition module; 14: Fusion module. DETAILED DESCRIPTION

[0031] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0032] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as program codes, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program codes, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and the like. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.

[0033] At present, the traditional image fusion method has a poor erasing effect on style images with texture structures. At the same time, the synthesis effect of the existing image fusion method is poor. To this end, the present application provides an image fusion method, device and storage medium.

[0034] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of an image fusion method according to an embodiment of the present invention. Figure 1 As shown, the image fusion method in the embodiment of the present invention mainly includes the following steps S101 to S104.

[0035] Step S101: Obtain a first style image and a text image to be migrated, wherein the first style image includes a first text content, the text image to be migrated includes a second text content, and the style patterns of the first style image and the text image to be migrated are different. Specifically, the first style image includes the first text content, and the text image to be migrated includes the second text content. Generally speaking, the first text content and the second text content are different. In addition, the style patterns of the first style image and the text image to be migrated are different, wherein the style patterns may be fonts, font sizes, etc., but are not limited thereto.

[0036] Step S102: Erasing the first text content of the first style image based on the text erasure model to obtain a background image. The text erasure model of the present application includes a generative adversarial network (GAN). GAN (Generative Adversarial Networks) is a deep learning model and one of the most promising methods for unsupervised learning on complex distributions in recent years. GAN generates output through the mutual game learning of at least two modules in the framework: a generative model and a discriminative model. The text erasure model of the present application is implemented based on a generative adversarial network (GAN). Specifically, Figure 2 As shown, the text erasure model in this embodiment includes a generator and a discriminator, wherein the generator is composed of an encoder (Encoder), a convolutional layer (Conv) and a decoder (Decoder). When training the text erasure model, the training data required by the text erasure network is generated online by collecting commonly used background images of fixed layouts and nearly 300 printed fonts for training, which further improves the accuracy of model training. The training data may include a first style image and a background image label. The first style image is input into the text erasure model for training, and the loss function is calculated using the output result and the real background image label. The specific loss function L BThe calculation formula of is shown in the following formula (1). After the model is trained, the style image to be erased is input into the trained text erasure model to obtain the background image.

[0037]

[0038] In the above formula, D B Represents the result of the discriminator, T b is the background image tag, I s is the first style image, O b is the result output by the generator, and β is a coefficient, which can be set to 10 in this embodiment.

[0039] Step S103: Obtain a second style image based on the background image and the first style image. Specifically, the background image and the first style image are differentially calculated to obtain the second style image. The differential calculation described here is to subtract the corresponding positions of the background image and the first style image, thereby erasing the background image in the first style image to obtain the second style image. By performing a differential calculation on the background image and the first style image, a second style image containing only text content is obtained. This method of highlighting only the text itself is conducive to improving the accuracy of the skeleton image extracted by the subsequent text style transfer fusion model.

[0040] Step S104: Using the text style transfer fusion model to fuse the second style image, the text image to be transferred, and the background image to obtain a current fused image. The specific process of using the text style transfer fusion model to fuse the second style image, the text image to be transferred, and the background image to obtain the current fused image can be implemented through the following steps S1041 to S1042, which will not be repeated here.

[0041] The text style transfer fusion model of this application is also implemented based on the generative adversarial network (GAN), as shown below: Figure 3 As shown, the text style transfer fusion model includes multiple generators and discriminators, wherein the generator is composed of an encoder (Encoder) and a decoder (Decoder), wherein the encoder (Encoder) and the decoder (Decoder) are specifically composed of a Resnet network. Specifically, the encoder (Encoder) can be composed of three convolutional layers (Conv), and the decoder can be composed of three deconvolutional layers (Deconv). The text style transfer fusion model of the present application can be implemented in the training process using conventional network training methods, and all loss functions in the training process are shown in the following formulas (2)-(5).

[0042]

[0043] L Vgg=α1L per +α2L style (3)

[0044]

[0045]

[0046] In the above formula, φ i Represents the result of the network output, G represents the Gram matrix, α1 is 1, and α2 is 500.

[0047] In addition, TV loss can also be used to train the text style transfer fusion model to reduce noise through adjacent pixel values, thereby maintaining the smoothness of the current fused image. The specific formula is shown in the following formula (6).

[0048]

[0049] In the above formula, coefficient α = 2 × 10 -8 , i, j represent the row number and column number of the image respectively, x i,j-1 Represents the element in the i-th row and j-1-th column.

[0050] Based on the above steps S101 to S104, the text erasure model is first used to erase the text content of the first style image to obtain a background image, and then the text style transfer fusion model is used to fuse the second style image with the background image deleted, the text image to be transferred, and the background image with the text information erased, and finally the current fused image is obtained, which avoids the problem of poor text effect synthesized by existing image fusion methods and improves the accuracy of image fusion.

[0051] Step S1041: Using the text style transfer fusion model, the second text content of the text image to be transferred is transferred to the second style image in the style of the second style image to obtain a transferred text image. In one embodiment, using the text style transfer fusion model, the second text content of the text image to be transferred is transferred to the target position in the second style image in the style of the second style image to obtain a transferred text image, and the target position is a preset position in the second style image. Specifically, Figure 3As shown, first, the second style image is passed through the encoder-decoder structure in the text style transfer fusion model to obtain a text skeleton, and then the second text content of the text image to be transferred is transferred to the target position in the second style image according to the text skeleton in the style of the second style image to obtain a transferred text image, wherein the target position in the second style image here refers to the preset position in the second style image. Since the second style image is an image generated by removing the background image from the first style image, this preset position can also be the position of the first text content in the first style image.

[0052] Step S1042: Fusing the migrated text image with the background image to obtain the current fused image. Specifically, the fusion here is also performed by fusing the migrated text image with the background image through an encoder-decoder structure, and the current fused image can be obtained after the fusion, that is, Figure 3 The output results in .

[0053] In addition, the text erasure model and text style transfer fusion model trained in the present application are placed in the text erasure module and image fusion module in the automated training platform respectively to perform text erasure and image fusion. It has been proved that the text erasure model trained in the present application has a good erasure effect. Specifically, for the first style image, as shown in FIG4(a), the trained text erasure model can well erase the text information in the first style image, thereby obtaining a background image, as shown in FIG4(b). In addition, the text style transfer fusion model of the present application has a good migration effect. Specifically, for the second style image, as shown in FIG5(a), the trained text style transfer fusion model can well migrate the text content in the text image to be migrated to the second style in the style style in the second style image, and finally obtain the current fused image, as shown in FIG5(b).

[0054] After generating the current fused image based on the above steps, the present application further optimizes the current fused image, specifically including: firstly, extracting the text content and boundary features of the current fused image using a feature extraction network, then splicing the current fused image and the extracted text content and boundary features to obtain a spliced ​​image, wherein the splicing can be achieved using conventional splicing methods. Finally, convolution processing is performed on the spliced ​​image to obtain the final fused image, specifically as follows: Figure 6As shown, the convolution processing of the current fused image and text content and boundary features can be specifically implemented by a convolution layer (Conv). In addition, the feature extraction network in this embodiment includes an encoder and a decoder, the encoder includes at least one convolution layer, and the decoder includes at least one deconvolution layer, wherein the number of convolution layers and the number of deconvolution layers can be specifically set according to actual conditions. For the feature extraction network, during training, it can be trained by using the collected real images and their labels (label), and during testing, the output result of the text style transfer fusion model can be directly used as the input of the feature extraction network for testing.

[0055] In this step, based on the current fused image, the text information and boundary information of the current fused image are further optimized. Specifically, the text content and boundary information of the current fused image are extracted, and the extracted information is spliced ​​with the current fused image and then convolution is performed to obtain the final fused image, thereby achieving the enhancement effect of the text information and boundary information of the fused image.

[0056] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0057] Furthermore, the present invention also provides an image fusion device. Figure 7 , Figure 7 FIG. 1 is a main structural block diagram of an image fusion device according to an embodiment of the present invention. Figure 7 As shown, the image fusion device in the embodiment of the present invention mainly includes an input module 11, an erasing module 12, an acquisition module 13 and a fusion module 14. In some embodiments, one or more of the input module 11, the erasing module 12, the acquisition module 13 and the fusion module 14 can be combined into one module. In some embodiments, the input module 11 can be configured to acquire a first style image and a text image to be transferred, the first style image includes a first text content, the text image to be transferred includes a second text content, and the style styles of the first style image and the text image to be transferred are different. The erasing module 12 can be configured to erase the first text content of the first style image based on the text erasure model to obtain a background image. The acquisition module 13 can be configured to obtain a second style image based on the background image and the first style image. The fusion module 14 can use the text style transfer fusion model to fuse the second style image, the text image to be transferred and the background image to obtain a current fused image. In one embodiment, the description of the specific implementation function can refer to the description of steps S101-step S104.

[0058] The image fusion device is used to perform Figure 1 The image fusion method embodiments shown in the figure have similar technical principles, technical problems solved and technical effects produced. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process and related instructions of the image fusion device can refer to the contents described in the embodiment of the image fusion method, which will not be repeated here.

[0059] It is understood by those skilled in the art that the present invention implements all or part of the processes in the method of the above embodiment, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0060] Furthermore, the present invention also provides an electronic device. In an electronic device embodiment according to the present invention, the electronic device includes a processor and a storage device, the storage device can be configured to store a program for executing the image fusion method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to a program for executing the image fusion method of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention.

[0061] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the image fusion method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned image fusion method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0062] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0063] Those skilled in the art will appreciate that the modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present invention.

[0064] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. An image fusion method, characterized in that: The steps include: Acquire a first style image and a text image to be migrated, wherein the first style image includes a first text content, the text image to be migrated includes a second text content, and the first style image and the text image to be migrated have different style styles; Erasing the first text content of the first style image based on a text erasure model to obtain a background image; Acquire a second style image based on the background image and the first style image; Using a text style transfer fusion model to fuse the second style image, the text image to be transferred, and the background image to obtain a current fused image; Also includes: Extracting text content and boundary features of the current fused image using a feature extraction network; Splicing the current fused image with the extracted text content and boundary features to obtain a spliced ​​image; The convolution process is performed on the stitched image to obtain a final fused image.

2. The image fusion method according to claim 1, characterized in that: Acquiring the second style image based on the background image and the first style image includes: performing a difference calculation on the background image and the first style image to obtain the second style image.

3. The image fusion method according to claim 1, characterized in that: Using the text style transfer fusion model to fuse the second style image, the text image to be transferred, and the background image to obtain the current fused image includes: using the text style transfer fusion model to transfer the second text content of the text image to be transferred to the second style image in the style of the second style image to obtain a transferred text image; The migrated text image is fused with the background image to obtain a current fused image.

4. The image fusion method according to claim 3, characterized in that: Using a text style transfer fusion model, the second text content of the text image to be transferred is transferred to the second style image in the style style of the second style image to obtain the transferred text image, including: using a text style transfer fusion model, the second text content of the text image to be transferred is transferred to a target position in the second style image in the style style of the second style image to obtain the transferred text image, and the target position is a preset position in the second style image.

5. The image fusion method according to claim 1, characterized in that: The text erasure model includes a generative adversarial network, and the text style transfer fusion model includes a Resnet network.

6. The image fusion method according to claim 1, characterized in that: The feature extraction network includes an encoder and a decoder, the encoder includes at least one convolutional layer, and the decoder includes at least one deconvolutional layer.

7. An image fusion device, characterized in that: include: An input module is configured to obtain a first style image and a text image to be migrated, wherein the first style image includes a first text content, the text image to be migrated includes a second text content, and the first style image and the text image to be migrated have different style styles; an erasing module, configured to erase the first text content of the first style image based on a text erasure model to obtain a background image; An acquisition module, configured to acquire a second style image based on the background image and the first style image; A fusion module is configured to fuse the second style image, the text image to be transferred, and the background image using a text style transfer fusion model to obtain a current fused image; Also includes: Extracting text content and boundary features of the current fused image using a feature extraction network; Splicing the current fused image with the extracted text content and boundary features to obtain a spliced ​​image; The convolution process is performed on the stitched image to obtain a final fused image.

8. An electronic device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the image fusion method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the image fusion method according to any one of claims 1 to 6.

Images