A method, device, electronic device and storage medium for face image fusion
By using Gaussian residual maps and fuzzy masking techniques, the problems of high time cost and poor fusion effect of existing image fusion technologies in real-time scenarios are solved, achieving low-cost and efficient image fusion effect and highlighting the texture and detail features of the image to be fused.
Patent Information
- Application Number
- CN202010598241.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-28
- Publication Date
- 2024-11-01
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing image fusion techniques, such as Poisson fusion, are time-consuming and difficult to apply in real-time scenarios, and the fusion effect is poor, easily resulting in problems such as internal color differences and abrupt edge transitions.
By employing Gaussian residual image and blur mask techniques, the Gaussian residual image and facial key points of the image to be fused are determined. The image to be fused is then merged into the target image using a mask after blurring. Finally, a Gaussian residual image is added for enhancement, thus achieving image fusion.
It achieves efficient image fusion with low time cost, can highlight the texture and detail features of the image to be fused, eliminate the problems of color differences within the image and abrupt edge transitions, and is suitable for real-time scenarios.
Smart Images

Figure CN111768356B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of image processing and deep learning, and in particular to the field of image fusion. Background Art
[0002] Image fusion is an important part of image processing. It can output a fused image suitable for human visual perception or further processing and analysis by computer for multiple image information of the same scene or different image information in different scenes. Image fusion has a wide range of application scenarios: for example, image stitching, multiple images taken by the same device in a short period of time need to be fused to obtain a panoramic view; in augmented reality applications, virtual images produced by mobile phones or other devices need to be fused before they can be added to real images.
[0003] Currently, common image fusion technologies include RGB difference, Laplace pyramid fusion, histogram matching and Poisson fusion. Among them, Poisson fusion can achieve seamless fusion and style transfer, and is the best image fusion algorithm in current technology. However, the characteristic of Poisson fusion is that it needs to solve the gradient field and the equation group, which requires a high time cost. This characteristic makes Poisson fusion difficult to apply in real-time scenarios. Summary of the invention
[0004] The present application provides a face image fusion method, device, electronic device and storage medium.
[0005] According to one aspect of the present application, a face image fusion method is provided, comprising:
[0006] Determine a Gaussian residual map of the image to be fused; and obtain facial key points of the image to be fused, and synthesize an original mask using the facial key points;
[0007] Performing a fuzzy operation on the original mask to obtain a fuzzy mask;
[0008] According to the mask after the blur operation, the image to be fused is fused into the target image to obtain a fused image;
[0009] According to the mask after the blurring operation, the Gaussian residual map of the image to be fused is added to the fused image.
[0010] According to another aspect of the present application, a face image fusion device is provided, comprising:
[0011] A processing module, used to determine the Gaussian residual map of the image to be fused; and obtain the facial key points of the image to be fused, and synthesize the original mask using the facial key points;
[0012] A fuzzy operation module, used for performing a fuzzy operation on the original mask to obtain a mask after the fuzzy operation;
[0013] A fusion module, used for fusing the image to be fused into the target image according to the mask after the blur operation to obtain a fused image;
[0014] The enhancement module is used to add the Gaussian residual map of the image to be fused to the fused image according to the mask after the blurring operation.
[0015] According to another aspect of the present application, there is provided an electronic device, including:
[0016] at least one processor; and
[0017] a memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above methods.
[0019] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any of the above methods.
[0020] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.
[0021] According to the face image fusion method of the present application, after the image to be fused is fused into the target image, the Gaussian residual map of the image to be fused is used to enhance the fused image to highlight the texture and detail features of the image to be fused, thereby completing the final image fusion. The face image fusion method using the embodiment of the present application has a small amount of calculation, requires a low time cost, and has a good image fusion effect.
[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0024] Figure 1 A face image fusion method implementation process of the present application embodiment Figure 1 ;
[0025] Figure 2 A face image fusion method implementation process of the present application embodiment Figure 2 ;
[0026] Figure 3 This is a schematic diagram of the effect of implementing face image fusion in an embodiment of the present application;
[0027] Figure 4 This is a schematic diagram of the structure of a face image fusion device according to an embodiment of the present application;
[0028] Figure 5 This is a schematic diagram of the structure of another face image fusion device according to an embodiment of the present application;
[0029] Figure 6 It is a block diagram of an electronic device used to implement the face image fusion method of the embodiment of the present application. DETAILED DESCRIPTION
[0030] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0031] This application embodiment proposes a face image fusion method. Figure 1 A face image fusion method implementation process of the present application embodiment Figure 1 ,include:
[0032] Step S101: determining the Gaussian residual map of the image to be fused; obtaining the facial key points of the image to be fused, and synthesizing the original mask using the facial key points;
[0033] Step S102: performing a fuzzy operation on the original mask to obtain a fuzzy mask;
[0034] Step S103: fusing the image to be fused into the target image according to the mask after the fuzzy operation to obtain a fused image;
[0035] Step S104: adding the Gaussian residual image of the image to be fused to the fused image according to the mask after the blurring operation.
[0036] The embodiment of the present application has an excellent fusion effect for two highly similar face images (i.e., the target image and the image to be fused), and can solve the following problems that arise in existing face fusion technologies: First, the performance and time-consuming problem. The fusion efficiency of the embodiment of the present application is fast enough to provide more time for other processing algorithms. Second, the fusion effect problem. The embodiment of the present application can eliminate the perceived differences between the two images as much as possible while ensuring the characteristics of the two images. Specifically, it can fully achieve fusion of the interior and edges of the images.
[0037] Optionally, the image fusion technology proposed in the embodiment of the present application is designed and implemented based on parallel computing of a graphics processing unit (GPU), wherein the operation can be accelerated by an open graphics library (OpenGL, Open Graphics Library).
[0038] Optionally, fusing the image to be fused into the target image in the above step S103 includes: fusing the image to be fused into the target image with low transparency.
[0039] When performing fusion, by adjusting the transparency of the mask, the image to be fused is fused into the target image with low transparency (10% to 30%), so that the target image will not produce an abnormal feeling due to excessive color differences while retaining the lighter content of the image to be fused. Afterwards, the embodiment of the present application adopts step S104 to add the Gaussian residual map previously extracted from the image to be fused to the fused image according to the mask after the blur operation, to enhance the lighter features of the image to be fused, highlight its texture details, and thus complete the final image fusion.
[0040] Figure 2 A face image fusion method implementation process of the present application embodiment Figure 2 ,like Figure 2 As shown, the face image fusion method proposed in the embodiment of the present application mainly includes the following three main stages:
[0041] Phase 1:
[0042] Gaussian residual is used to extract the texture and brightness details of the image from the image to be fused.
[0043] Specific correspondence Figure 1 The Gaussian residual image of the image to be fused is determined in step S101. Optionally, Figure 1 Determining the Gaussian residual image of the image to be fused in step S101 includes:
[0044] Perform Gaussian convolution on the image to be fused to obtain a blurred image of the image to be fused;
[0045] The Gaussian residual image of the image to be fused is determined by using the image to be fused and the blurred image of the image to be fused.
[0046] The Gaussian convolution operation can make each pixel value take the average value of the surrounding colors according to the normal distribution probability and distance. Gaussian convolution can be used to blur the image to obtain good results. The blurred image of the image to be fused can be obtained by performing Gaussian convolution on the image to be fused, and the Gaussian residual image of the image to be fused can be obtained by subtracting the image to be fused from the blurred image of the image to be fused.
[0047] In the Gaussian residual map of the image to be fused, each pixel retains the color difference between the blurred image to be fused and the image to be fused; for pixels and regions with smooth colors, there tends to be no difference, that is, the pixel value of the pixel or region in the Gaussian residual map is equal to or close to 0; for pixels and regions with sharp colors and rich features, there is a value, that is, the pixel value of the pixel or region in the Gaussian residual map is not equal to 0. The Gaussian residual map contains the feature information of the image to be fused in the three RGB channels, which has little relationship with brightness, is less affected by light, and has more shape and texture information.
[0048] Phase 2:
[0049] A face landmark detector is used to extract facial landmarks of the image to be fused, such as extracting the area from eyebrows to chin to synthesize the original mask; the original mask is blurred, and the image to be fused and the target image are weightedly added at low transparency according to the mask after the blurred operation.
[0050] Specifically corresponds to Figure 1 The steps of step S101 are as follows: obtaining facial key points of the image to be fused, synthesizing the original mask by using the facial key points, and step S102 and step S103.
[0051] Optionally, Figure 1 Step 102 includes:
[0052] Perform pose alignment on the original mask to align it to the face area of the image to be fused;
[0053] Gaussian blur is performed on the original mask after the pose alignment operation.
[0054] A mask is a black and white image that represents the mixing ratio of the image to be fused and the target image pixel by pixel, that is, transparency, and the value of a single channel is between 0 and 1. The mask image can be used to conveniently specify the size and shape of the fusion area, and hollowing and special transition curves can be formulated. In the embodiment of the present application, the edge of the original mask is softened by Gaussian blur to obtain a mask after the blur operation, so that the transition of image fusion is softer. In addition, the embodiment of the present application performs fusion by masking, which avoids the linear traversal of edge pixel calculation interpolation compared to the usual fusion method, and can reduce the time cost of image fusion.
[0055] Optionally, the embodiment of the present application may use the following formula for fusion:
[0056] Pixel color of fused image = pixel color of target image * first transparency + pixel color of image to be fused * (1.0 – first transparency);
[0057] The first transparency is the transparency of the mask after the blur operation.
[0058] Figure 3 This is a schematic diagram of the image fusion effect achieved in an embodiment of the present application. Figure 3 The face images in are artificially synthesized, not real people. Figure 3 In the figure, the first row of images is the effect of the existing Poisson fusion, and the second row of images is the image fusion effect of the embodiment of the present application. Figure 3 In the fusion image of the present application, except for the glasses, the highlights of the nose tip, the shape of the corners of the eyes, the thickness of the eyebrows, etc., all incorporate the features of the image to be fused.
[0059] Phase 3:
[0060] The Gaussian residual image of the image to be fused is added to the fused image, that is, the fused image is enhanced to obtain an enhanced image.
[0061] Optionally, the embodiment of the present application may use the following formula to perform image enhancement:
[0062] Enhanced pixel color = fused image pixel color + Gaussian residual image pixel color
[0063] Second transparency = (1.0-pixel value of mask image after blurring)*2.0
[0064] The final pixel color of the enhanced image = enhanced pixel color * second transparency + fused image pixel color * (1.0 – second transparency).
[0065] By adopting the steps of the third stage, the features of the image to be fused that are lighter in the fused image obtained in the second stage can be enhanced, thereby highlighting the texture details of the image to be fused and avoiding the problem of loss of features of the image to be fused.
[0066] The embodiment of the present application effectively provides a fusion solution that is easy to implement, has good fusion effect and is extremely efficient, so that the fusion operation of the image to be fused to the target image can be performed in real time, and the fusion effect of the embodiment of the present application is good.
[0067] In particular, the embodiments of the present application can solve the performance time-consuming problem of the prior art. The operation steps adopted in the embodiments of the present application, such as Gaussian convolution, Gaussian residual, image fusion, fuzzy calculation, texture enhancement, etc., are all pixel-level operations at the lowest level, so they can meet the GPU acceleration conditions from the design point of view. In terms of fusion effect, the embodiments of the present application avoid the problem of internal color difference that is easy to occur in common image fusion through low-transparency fusion; solve the problem of abrupt transition of the edge of the fused image through fuzzy masking, and avoid the problem of loss of features of the image to be fused or the target image that may occur due to insufficient fusion through texture enhancement.
[0068] The present application also provides a face image fusion device. Figure 4 This is a schematic diagram of the structure of a face image fusion device according to an embodiment of the present application, comprising:
[0069] The processing module 410 is used to determine the Gaussian residual map of the image to be fused; and obtain the facial key points of the image to be fused, and synthesize the original mask using the facial key points;
[0070] A fuzzy operation module 420, configured to perform a fuzzy operation on the original mask to obtain a fuzzy-operated mask;
[0071] A fusion module 430 is used to fuse the image to be fused into the target image according to the mask after the blur operation to obtain a fused image;
[0072] The enhancement module 440 is used to add the Gaussian residual map of the image to be fused to the fused image according to the mask after the blurring operation.
[0073] Optionally, the fusion module 430 is used to fuse the image to be fused into the target image with low transparency.
[0074] Figure 5 FIG. 1 is a schematic diagram of another structure of a face image fusion device according to an embodiment of the present application. Figure 5 As shown, optionally, the processing module 410 includes:
[0075] A Gaussian convolution submodule 411 is used to perform Gaussian convolution on the image to be fused to obtain a blurred image of the image to be fused;
[0076] The Gaussian residual determination submodule 412 is used to determine the Gaussian residual map of the image to be fused by using the image to be fused and the blurred image of the image to be fused.
[0077] like Figure 5 As shown, optionally, the processing module 410 includes:
[0078] The detection submodule 413 is used to detect the image to be fused using a facial key point detector to obtain facial key points of the image to be fused.
[0079] like Figure 5 As shown, optionally, the fuzzy operation module 420 includes:
[0080] A posture alignment submodule 421 is used to perform a posture alignment operation on the original mask so as to align the original mask to the face area of the image to be fused;
[0081] The Gaussian blur submodule 422 is used to perform a Gaussian blur operation on the original mask after the pose alignment operation.
[0082] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and will not be repeated here.
[0083] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.
[0084] like Figure 6 , is a block diagram of an electronic device according to a face image fusion method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0085] like Figure 6As shown, the electronic device includes: one or more processors 601, a memory 602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 601 is taken as an example.
[0086] The memory 602 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the face image fusion method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the face image fusion method provided in the present application.
[0087] The memory 602 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the face image fusion method in the embodiment of the present application (for example, the attached Figure 4 The processor 601 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 602, that is, the face image fusion method in the above method embodiment is implemented.
[0088] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the electronic device for face image fusion, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the electronic device for face image fusion via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0089] The electronic device of the face image fusion method may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.
[0090] The input device 603 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device fused with the face image, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 604 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0091] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0092] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0094] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0095] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.
[0096] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0097] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A face image fusion method, comprising: Determine the Gaussian residual map of the image to be fused; And obtaining the facial key points of the image to be fused, and synthesizing the original mask using the facial key points; Performing a posture alignment operation on the original mask so that the original mask is aligned to the face area of the image to be fused; Perform Gaussian blur operation on the original mask after the pose alignment operation to obtain the mask after the blur operation; According to the mask after the blur operation, the image to be fused is fused into the target image to obtain a fused image; According to the mask after the blurring operation, the Gaussian residual map of the image to be fused is added to the fused image.
2. The method according to claim 1, wherein: The fusing the image to be fused into the target image includes: fusing the image to be fused into the target image with low transparency.
3. The method according to claim 1 or 2, wherein: The step of determining the Gaussian residual map of the image to be fused includes: Performing Gaussian convolution on the image to be fused to obtain a blurred image of the image to be fused; The Gaussian residual image of the image to be fused is determined by using the image to be fused and the blurred image of the image to be fused.
4. The method according to claim 1 or 2, wherein: The step of obtaining facial key points of the image to be fused comprises: The image to be fused is detected using a facial key point detector to obtain facial key points of the image to be fused.
5. A facial image fusion device, comprising: A processing module, used for determining a Gaussian residual map of the image to be fused; And obtaining the facial key points of the image to be fused, and synthesizing the original mask using the facial key points; A fuzzy operation module, used for performing a posture alignment operation on the original mask to align the original mask to the face area of the image to be fused, and performing a Gaussian blur operation on the original mask after the posture alignment operation to obtain a mask after the fuzzy operation; A fusion module, used for fusing the image to be fused into the target image according to the mask after the blur operation to obtain a fused image; The enhancement module is used to add the Gaussian residual map of the image to be fused to the fused image according to the mask after the blurring operation.
6. The device according to claim 5, wherein: The fusion module is used to fuse the image to be fused into the target image with low transparency.
7. The device according to claim 5 or 6, wherein: The processing module comprises: A Gaussian convolution submodule, used for performing Gaussian convolution on the image to be fused to obtain a blurred image of the image to be fused; The Gaussian residual determination submodule is used to determine the Gaussian residual map of the image to be fused by using the image to be fused and the blurred image of the image to be fused.
8. The device according to claim 5 or 6, wherein: The processing module comprises: The detection submodule is used to detect the image to be fused using a facial key point detector to obtain the facial key points of the image to be fused.
9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.
10. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4.
