Image harmonization processing method and related equipment

Abstract

The invention provides an image harmonization processing method and related equipment. The method comprises the following steps: acquiring a to-be-processed composite image; the composite image comprises a composite area and a background area, and the composite area comprises a part of image clipped and pasted from other images; outputting and displaying the composite image to a user, and obtaining a target reference area selected by the user from the composite image; wherein the target reference area comprises at least part of the background area; and obtaining image parameters of the target reference area, and carrying out image harmonization processing on the synthesis area based on the image parameters of the target reference area to obtain a harmonized image.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image processing technology, and more particularly, to an image harmonization processing method and related devices. Background Art

[0002] Image composition refers to cutting out part of an image (such as a portrait) from one image and pasting it onto another image to obtain a composite image. The composite image often includes the original background area and the foreground area containing the texture (i.e., the composite area). Image synthesis technology can be used to obtain images of interest to users, and can also be used for data augmentation, and has broad application prospects. However, there are many problems with the composite images obtained through PS or textures, such as the unreasonable size or position of the foreground, and the color brightness distribution of the foreground and background looks disharmonious and inconsistent.

[0003] Image harmonization refers to adjusting the composite area of ​​PS or clipping in the composite image to make it more harmoniously integrated with the image background to improve the quality of the composite image. Conventional image harmonization methods always force the overall color brightness distribution of the composite area to be consistent with that of the entire background area. However, the range of the background area is often large, which may contain multiple image targets, each of which may have a different color brightness distribution. If the overall color brightness distribution of the composite area is consistent with that of the entire background area, and the composite area is affected by the entire background area, it is very likely that the color brightness distribution of the composite area will be completely unreasonable, making the effect of image harmonization completely uncontrollable, and thus resulting in poor image harmonization effect. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide an image harmonization processing method and related devices.

[0005] In a first aspect, the present specification provides an image harmonization processing method, the method comprising:

[0006] Acquire a composite image to be processed; the composite image includes a composite area and a background area, and the composite area includes a partial image clipped from other images;

[0007] Outputting and displaying the composite image to a user, and acquiring a target reference area selected by the user from the composite image; wherein the target reference area includes at least a portion of the background area;

[0008] The image parameters of the target reference area are acquired, and based on the image parameters of the target reference area, the image harmonization processing is performed on the synthesis area to obtain a harmonized image.

[0009] In a second aspect, the present specification provides an image harmonization processing device, the device comprising:

[0010] A composite image acquisition unit, used to acquire a composite image to be processed; the composite image includes a composite area and a background area, and the composite area includes a partial image clipped from other images;

[0011] A reference region acquisition unit, configured to output the composite image to a user and acquire a target reference region selected by the user from the composite image; wherein the target reference region includes at least a portion of the background region;

[0012] The image harmonization processing unit is used to obtain the image parameters of the target reference area, and based on the image parameters of the target reference area, perform image harmonization processing on the synthesis area to obtain a harmonized image.

[0013] Accordingly, the present specification also provides a computing device, comprising: a memory and a processor; the memory stores a computer program / instruction that can be executed by the processor; when the processor executes the computer program / instruction, it executes the image harmonization processing method described in the first aspect above.

[0014] Accordingly, the present specification also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the image harmonization processing method as described in the first aspect above is executed.

[0015] Accordingly, the present specification also provides a computer program product, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, the image harmonization processing method described in the first aspect is executed.

[0016] In summary, when performing image harmonization processing, the present specification can first output and display the composite image to be processed to the user, and allow the user to independently select the local area (such as the target reference area) that needs to be referenced during the image harmonization processing from the entire composite image, and then perform image harmonization processing on the composite area in a targeted manner based on the image parameters of the local reference area. In this way, the present specification realizes a flexible and configurable reference area for image harmonization processing through human-computer interaction, thereby realizing more accurate image harmonization processing that meets user expectations, so as to improve the quality of the composite image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment;

[0018] Figure 2 is a flowchart of an image harmonization processing method provided by an exemplary embodiment;

[0019] Figure 3 is a schematic diagram of a composite image provided by an exemplary embodiment;

[0020] Figure 4 is a schematic diagram of another composite image provided by an exemplary embodiment;

[0021] Figure 5 is a schematic diagram of selecting a target reference area provided by an exemplary embodiment;

[0022] Figure 6 is a schematic diagram of selecting another target reference area provided by an exemplary embodiment;

[0023] Figure 7 is a flowchart of another image harmonization processing method provided by an exemplary embodiment;

[0024] Figure 8 is a structural schematic diagram of an image harmonization processing device provided by an exemplary embodiment;

[0025] Fig. 9 It is a structural diagram of a computing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0028] It should be noted that the “plurality” mentioned in this application refers to two or more than two.

[0029] Image harmonization technology can improve the quality of the composite image by adjusting the composite area of ​​PS or clipping in the composite image to make it blend more harmoniously with the image background. Conventional image harmonization methods always force the overall color brightness distribution of the composite area to be consistent with the entire background area.

[0030] However, the background area is often large and may contain multiple image targets, each of which may have a different color brightness distribution. If the overall color brightness distribution of the synthesized area is integrated with that of the entire background area and the synthesized area is affected by the entire background area, it is very likely that the color brightness distribution of the synthesized area will be completely unreasonable, making the image harmonization effect completely uncontrollable, which will lead to a poor image harmonization effect.

[0031] Based on this, this specification provides a technical solution, which allows the user to select a local area from the entire composite image through human-computer interaction that can be used as a reference for image harmonization processing, so as to achieve more accurate image harmonization processing that meets user expectations based on the image parameters of the local area, thereby improving the quality of the composite image.

[0032] During implementation, the present specification may first obtain a composite image to be processed, wherein the composite image includes a composite area and a background area. The composite area includes a portion of an image clipped from other images. Further, the present specification may output and display the composite image to a user, and obtain a target reference area selected by the user from the composite image. The target reference area may include at least a portion of the above-mentioned background area. Further, the present specification may obtain image parameters of the target reference area, and based on the image parameters of the target reference area, perform image harmonization processing on the composite area, thereby obtaining a harmonized image.

[0033] In the above technical solution, when performing image harmonization processing, this manual can first output the composite image to be processed to the user, and allow the user to independently select the local area (such as the target reference area) that needs to be referenced during the image harmonization processing from the entire composite image, and then perform image harmonization processing on the composite area in a targeted manner based on the image parameters of the local reference area. In this way, this manual realizes a flexible and configurable reference area for image harmonization processing through human-computer interaction, thereby realizing more accurate image harmonization processing that meets user expectations, so as to improve the quality of the composite image.

[0034] See also Figure 1 , Figure 1FIG. 1 is a schematic diagram of a system architecture provided by an exemplary embodiment. Figure 1 As shown, the system may include a computing device 100. The computing device 100 may run corresponding image processing software, and the user may further perform image harmonization processing on the synthesized image that has been produced through the image processing software.

[0035] In one illustrated embodiment, the computing device 100 obtains a finished composite image and may output and display the composite image to a user via an output device (eg, a display screen) in the computing device 100 .

[0036] Among them, the composite image may include a composite area and a background area, and the composite area includes a partial image cut from other images. Exemplarily, the background area in the composite image may be a landscape image, and the composite area may include several portraits cut from a group photo. Exemplarily, the background area in the composite image may be a group photo of multiple people, and the composite area may include portraits of other people cut from other group photos. Exemplarily, the background area in the composite image may be a document image containing identity information, and the composite area may include facial images of other people cut from other document images, etc., which are not specifically limited in this specification.

[0037] In one illustrated embodiment, the composite image may be a composite image produced by a user through the computing device 100 and stored locally on the computing device 100, or the composite image may be a composite image obtained by the computing device 100 from other remote devices through a network connection, etc. This specification does not specifically limit this.

[0038] Further, after the computing device 100 outputs and displays the composite image to the user, the user can select a target reference area corresponding to the current image harmonization process from the displayed composite image through an input device (such as a mouse or a touch screen, etc.) in the computing device 100. Accordingly, the computing device 100 can obtain the target reference area selected by the user from the composite image. The target reference area may include at least a portion of the background area of ​​the composite image.

[0039] It should be noted that this specification does not specifically limit the specific implementation method for the user to select the target reference area from the composite image.

[0040] In an illustrated embodiment, the user can select any area in the composite image by himself through an input device (such as a mouse or a touch screen, etc.) in the computing device 100. Accordingly, the computing device 100 can determine the area as a target reference area in response to the user's selection operation on the area. For details, please refer to the following Figure 5 The description of the corresponding embodiments will not be repeated here.

[0041] In an illustrated embodiment, the computing device 100 may also first divide the composite image into multiple selectable regions, and then output and display the multiple regions to the user. Accordingly, the user may select any region from the multiple regions of the composite image through an input device (such as a mouse or a touch screen, etc.) in the computing device 100. Accordingly, the computing device 100 may determine the region as a target reference region in response to the user's selection operation on the region. For details, please refer to the following Figure 6 The description of the corresponding embodiments will not be repeated here.

[0042] Furthermore, the computing device 100 may obtain the image parameters of the target reference area selected by the user, and based on the image parameters of the target reference area, perform image harmonization processing on the synthesized area, so that the color brightness distribution of the synthesized area is adapted to the color brightness distribution of the target reference area, so that the synthesized area and the image background are more harmonious and harmonious, thereby obtaining a harmonized image. For details, please refer to the following Figure 2 The description of the corresponding embodiments will not be repeated here.

[0043] Furthermore, the computing device 100 may also output the processed harmonized image for display to the user.

[0044] As described above, this manual uses human-computer interaction to interactively allow the user to independently select a local area (such as a target reference area) from the entire composite image that can be used as a reference for the current image harmonization processing, so that more accurate and user-friendly image harmonization processing can be achieved based on the image parameters of the local area, thereby improving the quality of the composite image.

[0045] In one illustrated embodiment, assuming that the composite image is a composite image containing a face map or an ID map (i.e., a forged ID image), then after these composite images are harmonized using the image harmonization processing method provided in this specification, the resulting high-quality harmonized images can be used as training samples or test samples for anti-counterfeiting and anti-tampering models to detect the model's recognition effect on high-quality forged IDs, thereby improving the model's defense capabilities against forged IDs, effectively avoiding the dangerous situation where malicious users use synthetic forged IDs to impersonate real users for identity authentication, ensuring the security and reliability of electronic identity authentication, and thus protecting user interests and network security.

[0046] In one illustrated embodiment, the computing device 100 may be a smart wearable device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a server, etc. having the above functions, and this specification does not specifically limit this.

[0047] It should be noted that Figure 1 The system architecture shown is only for illustrative purposes and does not constitute a specific limitation on the system architecture of image harmonization processing in this specification. In some possible implementations, the system architecture of image harmonization processing may also include Figure 1 More devices as shown. For example, when the hardware conditions of the computing device 100 are not sufficient to support complex tasks such as image harmonization (for example, the computing device 100 is just a smart phone with limited computing power), the computing device 100 can send the synthesized image and the indication information corresponding to the target reference area selected by the user to other computing devices connected to the computing device 100 (such as a remote server or cloud computing center, etc.), and the other computing devices will perform the above-mentioned image harmonization process. Correspondingly, the harmonized image finally obtained by other computing devices can also be further sent to the computing device 100, and so on, which is not specifically limited in this specification.

[0048] See also Figure 2 , Figure 2 is a flowchart of an image harmonization processing method provided by an exemplary embodiment. The method can be applied to Figure 1 The system architecture shown can be applied to Figure 1 In the computing device 100 shown. Figure 2 As shown, the method may specifically include the following steps S201 to S203.

[0049] Step S201, obtaining a composite image to be processed; the composite image includes a composite area and a background area, and the composite area includes a partial image clipped from other images.

[0050] In an illustrated embodiment, the computing device 100 may first obtain a composite image to be processed, where the composite image may include a composite area and a background area, and the composite area may include a partial image clipped from other images.

[0051] For example, see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a composite image provided by an exemplary embodiment. Figure 3 As shown, the original background area of ​​the composite image may include images such as the sky, trees, vehicles and crowds, and the composite area may include crowd images clipped from other images, which is not specifically limited in this specification.

[0052] For example, see Figure 4 , Figure 4 FIG. 1 is a schematic diagram of another composite image provided by an exemplary embodiment. Figure 4As shown, the original background area of ​​the composite image may include the identity information image of the certificate, and the composite area may include other people's face images cut from other certificate images, which is not specifically limited in this specification.

[0053] Step S202: outputting the composite image to the user, and acquiring a target reference area selected by the user from the composite image; wherein the target reference area includes at least a portion of the background area.

[0054] Furthermore, the computing device may output the composite image to the user and obtain the target reference area selected by the user from the composite image, wherein the target reference area may include at least a part of the background area of ​​the composite image.

[0055] It should be noted that this specification does not specifically limit the specific implementation method for the user to select the target reference area from the composite image.

[0056] In one illustrated embodiment, a user can select any area in the composite image according to his or her needs through an input device in the computing device (such as a mouse or a touch screen, etc.). Accordingly, the computing device can determine the area as a target reference area in response to the user's selection operation on the area.

[0057] For example, Figure 3 For example, see the composite image shown in Figure 5 , Figure 5 FIG. 1 is a schematic diagram of selecting a target reference area provided by an exemplary embodiment. Figure 5 As shown, the user can perform corresponding gesture operations (such as sliding or smearing, etc.) on the touch screen of the computing device to select a target reference area (such as Figure 5 Alternatively, in an embodiment shown, the user may select the target reference area in the composite image by performing operations such as mouse sliding and clicking. Alternatively, in an embodiment shown, the user may select the target reference area in the composite image by voice control, etc., which is not specifically limited in this specification.

[0058] In an illustrated embodiment, the computing device may also first divide the composite image into multiple optional regions, and output the multiple regions to the user. Accordingly, the user may select any region from the multiple regions of the composite image through an input device (such as a mouse or a touch screen, etc.) in the computing device according to his or her needs, and accordingly, the computing device may determine the region as the target reference region in response to the user's selection operation on the region.

[0059] In an illustrated embodiment, the composite image may include multiple image objects (or referred to as image instances). Specifically, the computing device may segment the composite image into multiple regions corresponding to the multiple image objects.

[0060] It should be noted that the present specification does not particularly limit the specific implementation method of segmenting the composite image into multiple regions. In one illustrated embodiment, image segmentation can be performed using a trained image segmentation model.

[0061] In an illustrated embodiment, the computing device may input the composite image into the trained image segmentation model, and the image segmentation model may extract the image features contained in the composite image. The image features may include semantic information corresponding to multiple image objects in the composite image. Furthermore, the image segmentation model may determine the boundary information of multiple image objects in the composite image based on the semantic information, and divide the composite image into multiple regions corresponding to the multiple image objects based on the boundary information, etc., which is not specifically limited in this specification.

[0062] For example, Figure 3 For example, see the composite image shown in Figure 6 , Figure 6 FIG. 1 is a schematic diagram of another target reference area selection provided by an exemplary embodiment. Figure 6 As shown in FIG. 1 , the synthetic image can be segmented into six regions by the image segmentation model, namely, region 1 including the sky, region 2 including trees, region 3 including buses, region 4 including cars, region 5 including crowd A, and region 6 including crowd B. Region 6 may include part of the synthetic region. Figure 6 As shown, the user can perform a corresponding operation gesture (such as a click operation) on the touch screen of the computing device to select a target reference area (such as Figure 6 Alternatively, in an embodiment shown, the user may select a target reference area from the multiple segmented areas by clicking a mouse or the like. Alternatively, in an embodiment shown, the user may select a target reference area from the multiple segmented areas by voice control, etc., which is not specifically limited in this specification.

[0063] It should be understood that Figure 6 The region segmentation of the composite image shown is only for illustrative purposes. In one embodiment shown, the region segmentation of the composite image can be more fine-grained, for example Figure 6Each tree in the crowd A can be divided into an independent area, each car can be divided into an independent area, and each portrait in the crowd A and the crowd B can be divided into an independent area, and so on. This manual does not make specific limitations on this.

[0064] In addition, in an illustrated embodiment, after the computing device outputs and displays multiple areas segmented from a composite image to the user, if the user is not satisfied with the multiple areas, the user may also select a target reference area that meets his or her needs in the composite image, and this manual does not specifically limit this.

[0065] Step S203, acquiring image parameters of the target reference area, and performing image harmonization processing on the synthesis area based on the image parameters of the target reference area to obtain a harmonized image.

[0066] Furthermore, after obtaining the target reference area selected by the user from the composite image, the computing device can further obtain image parameters of the target reference area, and perform image harmonization processing on the composite area based on the image parameters of the target reference area, thereby obtaining a harmonized image.

[0067] In an illustrated embodiment, the image parameters may include any one or more combinations of the following parameters: any possible image parameters such as color, brightness, contrast, and saturation, which are not specifically limited in this specification.

[0068] In an illustrated embodiment, when the computing device performs image harmonization processing on the synthesized area based on the image parameters of the target reference area, the processing may specifically include: the computing device adjusts the image parameters of the synthesized area based on the image parameters of the target reference area, so that the color brightness distribution of the synthesized area is adapted to the color brightness distribution of the target reference area. It is understandable that after the image harmonization processing, the difference between the image parameters of the synthesized area in the harmonized image and the image parameters of the target reference area is smaller than the difference between the image parameters of the synthesized area in the original synthesized image and the image parameters of the target reference area.

[0069] In an illustrated embodiment, considering that the pixel values ​​of the pixels included in the image express information such as the color, brightness and saturation of the pixels, when the computing device adjusts the image parameters of the synthesized area based on the image parameters of the target reference area, it may specifically include: adjusting the pixel values ​​of the pixels of the synthesized area based on the pixel values ​​of the pixels of the target reference area. Accordingly, the difference between the pixel values ​​of the pixels of the synthesized area in the harmonized image and the pixel values ​​of the pixels of the target reference area is smaller than the difference between the pixel values ​​of the pixels of the synthesized area in the original synthesized image and the pixel values ​​of the pixels of the target reference area.

[0070] It should be noted that the present specification does not particularly limit the specific implementation method of the image harmonization processing. In one illustrated implementation, the image harmonization processing can be performed by using a trained image harmonization model.

[0071] In an illustrated embodiment, the computing device may first generate indication information corresponding to the target reference area selected by the user. The indication information may be used to indicate an area that needs to be referenced when performing image harmonization processing. Further, the computing device may input the synthesized image and the indication information into a trained image harmonization model, and the image harmonization model determines the target reference area corresponding to this image harmonization processing based on the indication information. Further, the image harmonization model may obtain image parameters of the target reference area, and perform image harmonization processing on the synthesized area based on the image parameters of the target reference area, for example, adjusting the pixel values ​​of the pixels of the synthesized area based on the pixel values ​​of the pixels of the target reference area so that the color brightness distribution of the synthesized area matches the color brightness distribution of the target reference area.

[0072] In an illustrated embodiment, the indication information corresponding to the target reference area may include an attention distribution map corresponding to the target reference area. The size of the attention distribution map may be consistent with the size of the composite image, and the pixels included in the attention distribution map correspond one-to-one to the pixels included in the composite image. It should be understood that the pixel values ​​of the pixels included in the attention distribution map are not used to represent the color, but to represent the reference degree (or attention degree) to the pixel when performing image harmonization processing. Therefore, the pixel value of the pixel included in the attention distribution map is actually the attention weight value of the pixel. Based on this, the pixel values ​​of the pixels corresponding to the target reference area of ​​the composite image in the attention distribution map are often greater than the pixel values ​​of the pixels corresponding to other areas of the composite image in the attention distribution map, so as to accurately distinguish the target reference area from other areas.

[0073] In one illustrated embodiment, the pixel values ​​of the pixels included in the attention distribution map may be 0 or 1 (ie, the attention weight values ​​of the pixels may be 0 or 1), and the attention distribution map may be regarded as a mask map.

[0074] Among them, the attention weight value of the pixel corresponding to the target reference area in the attention distribution map can be 1. When performing image harmonization processing, the actual reference pixel value of the pixel contained in the target reference area = the original pixel value * 1. This means that during the image harmonization processing, it is necessary to refer to the pixel value of the pixel in the target reference area in the synthesized image to adjust the pixel value of the pixel in the synthesized area.

[0075] Among them, the attention weight value of the pixels corresponding to other areas in the attention distribution map can be 0. When performing image harmonization processing, the actual reference pixel value of the pixels contained in other areas = original pixel value * 0, which means that during the image harmonization processing, there is no need to refer to the pixel values ​​of the pixels in other areas of the synthesized image to adjust the pixel values ​​of the pixels in the synthesized area, that is, other areas are excluded from the reference of the image harmonization processing.

[0076] Correspondingly, when the computing device inputs the synthetic image and indication information into the trained image harmonization model, and the image harmonization model determines the target reference area corresponding to the current image harmonization processing based on the indication information, it can specifically include: inputting the synthetic image and the generated attention distribution map into the trained image harmonization model, and the image harmonization model determines the target reference area corresponding to the current image harmonization processing based on the pixel values ​​of the pixels contained in the attention distribution map (i.e., the attention weight values). Exemplarily, the area composed of multiple pixels in the synthetic image corresponding to multiple pixels with an attention weight value of 1 in the attention distribution map can be determined as the target reference area.

[0077] In one illustrated embodiment, see Figure 7 , Figure 7 FIG. 1 is a flow chart of another image harmonization processing method provided by an exemplary embodiment. Figure 7 The interactive image harmonization processing method provided in this specification is described.

[0078] like Figure 7 As shown, after the entire interactive image harmonization processing link is started, the computing device can output the composite image to be processed to the user, and prompt the user to select the target reference area L for image harmonization processing from the composite image through a pop-up box on the human-computer interface, for example, the user can be prompted in various possible ways such as text, pattern information or voice information, which is not specifically limited in this specification. In addition, the computing device can also input the composite image into an image segmentation model, and the image segmentation model divides the composite image into multiple areas corresponding to multiple image targets contained in the composite image, such as area A1, area A3 and area A3, etc., and outputs the multiple areas obtained by segmentation to the user.

[0079] Furthermore, if Figure 7 As shown, the user can directly select the target reference area L in the composite image (for example, Figure 5 In the corresponding embodiment), a target reference area L (such as the above Figure 6 Corresponding embodiments), this specification does not make specific limitations on this.

[0080] Furthermore, if Figure 7 As shown, the computing device can generate a corresponding attention distribution map M according to the target reference area L selected by the user. The attention weight value of the pixels corresponding to the target reference area in the attention distribution map M can be 1, and the attention weight value of the pixels corresponding to other areas can be 0.

[0081] Furthermore, if Figure 7 As shown, the synthesized image and the generated attention distribution map M are input into the image harmonization model, and the image harmonization model determines the target reference area L for image harmonization processing according to the attention weight values ​​of the pixels in the attention distribution map M, and performs image harmonization processing on the synthesized area based on the image parameters of the target reference area L.

[0082] Finally, if Figure 7 As shown, the image harmonization model outputs the harmonized image after processing, and accordingly, the computing device can output and display the harmonized image to the user.

[0083] In addition, in some possible implementations, the target reference area selected by the user may be a synthetic area (eg Figure 6 As shown in the area 6 containing the crowd B, the process of image harmonization processing may be the reverse operation of the process described in the above step S203, that is, it is necessary to adjust the image parameters of the background area based on the image parameters of the synthetic area, so that the color brightness distribution of the background area is adapted to the color brightness distribution of the synthetic area, and so on. This specification does not make any specific limitation on this.

[0084] In addition, in some possible implementations, for the image harmonization processing of a composite image, the user can also select multiple target reference areas, and then perform image harmonization processing on the composite area based on the image parameters of these multiple target reference areas to achieve more flexible and comprehensive image harmonization processing.

[0085] In an illustrated embodiment, after the user selects multiple target reference areas from the composite image, the user can also configure corresponding reference weights (or attention levels) for the multiple target reference areas. Accordingly, the computing device can obtain the multiple target reference areas selected by the user, and the reference weights configured by the user for the multiple target reference areas. Among them, the reference weight corresponding to any target reference area can be used to indicate the degree of reference to the image parameters of the target reference area when performing image harmonization processing. In other words, the larger the reference weight of any target reference area, the more emphasis should be placed on adjusting the image parameters of the composite area based on the image parameters of the target reference area when performing image harmonization processing, so that the color brightness distribution of the composite area is closer to the color brightness distribution of the target reference area.

[0086] In an illustrated embodiment, the computing device may generate multiple attention distribution maps corresponding to the multiple target reference areas based on the multiple target reference areas selected by the user. In an illustrated embodiment, the attention weight value of the pixel corresponding to any target reference area in the attention distribution map corresponding to the target reference area may be 1, and the attention weight value of the pixel corresponding to other areas may be 0.

[0087] Furthermore, the computing device may input the synthesized image, multiple attention distribution maps corresponding to multiple target reference areas, and their respective reference weights into the image harmonization model, and the image harmonization model may determine multiple target reference areas selected by the user based on the attention weight values ​​of the pixels included in the multiple attention distribution maps. Furthermore, the image harmonization model may obtain image parameters of the multiple target reference areas, and perform image harmonization processing on the synthesized area based on the image parameters of the multiple target reference areas and their respective reference weights.

[0088] Alternatively, in an illustrated embodiment, the computing device may also generate multiple attention distribution maps corresponding to the multiple target reference areas selected by the user and their respective reference weights. In an illustrated embodiment, the attention weight value of the pixel corresponding to any target reference area in the attention distribution map corresponding to the target reference area may be proportional to the reference weight of the target reference area. That is to say, the attention distribution map corresponding to any target reference area further includes information on the reference weight of the target reference area, and the attention weight value of the pixel in the attention distribution map is no longer either 0 or 1, but can cover multiple attention levels. For example, the attention weight values ​​of different pixels can be 0, 0.5, 1, and 2, etc. This specification does not specifically limit this.

[0089] Furthermore, the computing device may input the synthesized image and the generated multiple attention distribution maps into the image harmonization model, and the image harmonization model may determine multiple target reference areas selected by the user and their reference weights based on the attention weight values ​​of the pixels included in the multiple attention distribution maps. Furthermore, the image harmonization model may obtain image parameters of the multiple target reference areas, and perform image harmonization processing on the synthesized area based on the image parameters of the multiple target reference areas and their reference weights.

[0090] For example, Figure 6Taking the multiple areas shown in FIG. 1 as an example, it is assumed that the user selects area 3 containing a bus and area 5 containing a crowd A from the multiple areas as target reference areas, and configures a reference weight of 1 for area 3 and a reference weight of 2 for area 5. Further, the computing device can generate two attention distribution maps corresponding to the two target reference areas based on the two target reference areas and their respective reference weights.

[0091] For example, in the attention distribution map M3 corresponding to area 3 generated by the computing device, the attention weight value of the pixels corresponding to area 3 can be 0.5, and the attention weight values ​​of the pixels in other areas are 0; in the attention distribution map M5 corresponding to area 5 generated by the computing device, the attention weight value of the pixels corresponding to area 5 can be 1, and the attention weight values ​​of the pixels in other areas are 0. Obviously, the attention level of area 5 is higher, and the attention weight value of each pixel is larger.

[0092] Or, for example, in the attention distribution map M3, the attention weight value of the pixels corresponding to area 3 can be 1, and the attention weight value of the pixels in other areas can be 0; in the attention distribution map M5, the attention weight value of the pixels corresponding to area 5 can be 2, and the attention weight value of the pixels in other areas can be 0, and so on. This specification does not make any specific limitations on this.

[0093] Further, the composite image and the generated attention distribution map M3 and attention distribution map M5 can be input into the image harmonization model. Based on the attention weight values ​​of the pixels contained in the attention distribution map M3 and the attention distribution map M5, the image harmonization model can determine that region 3 and region 5 in the composite image are target reference regions, and can determine different reference weights for region 3 and region 5. Further, the image harmonization model can obtain the image parameters of region 3 and region 5 respectively, and perform image harmonization processing on the composite region based on the image parameters of region 3 and region 5 and their respective reference weights. Exemplarily, the image harmonization model can focus on referring to the image parameters of region 5, adjust the image parameters of the composite region, and finally make the color brightness distribution of the composite region compatible with region 3 and region 5, and make the color brightness distribution of the composite region closer to the color brightness distribution of region 5.

[0094] In summary, this specification provides an interactive image harmonization processing method with a flexibly configurable reference area. When performing image harmonization processing, the composite image to be processed can be output to the user first, and the user can independently select the local area (such as the target reference area) that needs to be referenced during this image harmonization processing from the entire composite image. Subsequently, the composite area can be targeted based on the image parameters of the local reference area. In this way, this specification realizes a flexibly configurable reference area for image harmonization processing through human-computer interaction, thereby realizing more accurate image harmonization processing that meets user expectations, so as to improve the quality of the composite image.

[0095] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides an image harmonization processing device. Figure 8 , Figure 8 is a schematic diagram of a structure of an image harmonization processing device provided by an exemplary embodiment. The device 80 can be applied to Figure 1 The system architecture shown can be applied to Figure 1 The computing device 100 shown. Figure 8 As shown, the device 80 includes:

[0096] The composite image acquisition unit 801 is used to acquire a composite image to be processed; the composite image includes a composite area and a background area, and the composite area includes a partial image clipped from other images;

[0097] A reference region acquisition unit 802 is used to output the composite image to the user and acquire a target reference region selected by the user from the composite image; wherein the target reference region includes at least a part of the background region;

[0098] The image harmonization processing unit 803 is used to obtain the image parameters of the target reference area, and perform image harmonization processing on the synthesis area based on the image parameters of the target reference area to obtain a harmonized image.

[0099] In an illustrated embodiment, the image harmonization processing unit 803 is specifically configured to:

[0100] Based on the image parameters of the target reference area, the image parameters of the synthesized area are adjusted; wherein the difference between the image parameters of the synthesized area in the harmonized image and the image parameters of the target reference area is smaller than the difference between the image parameters of the synthesized area in the synthesized image and the image parameters of the target reference area.

[0101] In an illustrated embodiment, the reference region acquisition unit 802 is specifically configured to:

[0102] In response to a user's selection operation on any area in the composite image, the area is determined as the target reference area.

[0103] In an illustrated embodiment, the composite image includes a plurality of image objects; the reference area acquisition unit 802 is specifically configured to:

[0104] segmenting the composite image into a plurality of regions corresponding to the plurality of image objects;

[0105] Outputting the plurality of areas to a user;

[0106] In response to a user's selection operation on any area among the multiple areas, the area is determined as the target reference area.

[0107] In an illustrated embodiment, the reference region acquisition unit 802 is specifically configured to:

[0108] The composite image is input into a trained image segmentation model, and the image segmentation model extracts image features contained in the composite image, determines boundary information of the multiple image targets in the composite image based on the image features, and segments the composite image into multiple regions corresponding to the multiple image targets based on the boundary information.

[0109] In an illustrated embodiment, the image harmonization processing unit 803 is specifically configured to:

[0110] Generate indication information corresponding to the target reference area; wherein the indication information is used to indicate the area to be referenced when performing image harmonization processing;

[0111] The composite image and the indication information are input into the trained image harmonization model, and the image harmonization model determines the target reference area based on the indication information, obtains the image parameters of the target reference area, and performs image harmonization processing on the composite area based on the image parameters of the target reference area.

[0112] In an illustrated embodiment, the indication information includes an attention distribution map corresponding to the target reference area; the pixels contained in the attention distribution map correspond one-to-one to the pixels contained in the composite image; the pixel values ​​of the pixels contained in the attention distribution map are used to indicate the degree of reference to the pixel when performing image harmonization processing; wherein the pixel values ​​of the pixels in the attention distribution map corresponding to the target reference area are greater than the pixel values ​​of the pixels corresponding to other areas.

[0113] In one illustrated embodiment, the pixel values ​​of the pixels corresponding to the target reference area in the attention distribution map are 1, and the pixel values ​​of the pixels corresponding to other areas are 0.

[0114] In an illustrated embodiment, the image harmonization processing unit 803 is specifically configured to:

[0115] The synthesized image and the attention distribution map are input into a trained image harmonization model, and the image harmonization model determines the target reference area based on pixel values ​​of pixels included in the attention distribution map.

[0116] In an illustrated embodiment, the reference region acquisition unit 802 is specifically configured to:

[0117] Acquire multiple target reference areas selected by a user from the composite image, and reference weights respectively configured by the user for the multiple target reference areas; wherein the reference weight corresponding to any target reference area is used to indicate the degree of reference to the image parameters of the target reference area when performing image harmonization processing;

[0118] The indication information includes a plurality of attention distribution maps corresponding to the plurality of target reference areas; wherein the pixel value of a pixel corresponding to any target reference area in the attention distribution map corresponding to the target reference area is proportional to the reference weight of the target reference area;

[0119] The image harmonization processing unit 803 is specifically used for:

[0120] The composite image and the multiple attention distribution maps are input into the image harmonization model, and the image harmonization model determines the multiple target reference areas and their reference weights based on the pixel values ​​of the pixels contained in the multiple attention distribution maps, obtains the image parameters of the multiple target reference areas, and performs image harmonization processing on the composite area based on the image parameters of the multiple target reference areas and their reference weights.

[0121] In an illustrated embodiment, the image parameters include any one or more combinations of the following parameters: color, brightness, contrast, and saturation.

[0122] The implementation process of the functions and effects of each unit in the above-mentioned device 80 is specifically described in the above-mentioned embodiment, and will not be repeated here. It should be understood that the above-mentioned device 80 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor (CPU) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0123] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0124] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, a vehicle-mounted computer, or a combination of any of these devices.

[0125] Corresponding to the above method embodiment, the embodiment of this specification also provides a computing device. Fig. 9 , Fig. 9 is a schematic diagram of a computing device provided by an exemplary embodiment. The computing device may be Figure 1 The computing device 100 in the system architecture shown. Fig. 9 As shown, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected via a bus or other means. Fig. 9As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be run by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the image harmonization processing method in the embodiment of this specification, including: obtaining a composite image to be processed; the composite image includes a composite area and a background area, and the composite area includes a part of the image clipped from other images; outputting the composite image to the user, and obtaining the target reference area selected by the user from the composite image; wherein the target reference area includes at least a part of the background area; obtaining the image parameters of the target reference area, and based on the image parameters of the target reference area, performing image harmonization processing on the composite area to obtain a harmonized image, and so on.

[0126] For a detailed description of each step of the above-mentioned image harmonization processing method, please refer to the previous content, which will not be repeated here.

[0127] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and when these computer programs are executed by a processor, the various steps of the image harmonization processing method in the embodiment of this specification are executed. Please refer to the description of the above embodiment for details, which will not be repeated here.

[0128] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0129] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0130] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0131] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0132] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0133] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0134] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A method for image harmonization, characterized in that: The method comprises: Acquire a composite image to be processed; the composite image includes a composite area and a background area, and the composite area includes a partial image clipped from other images; Outputting and displaying the composite image to a user, and acquiring a target reference area selected by the user from the composite image; wherein the target reference area includes at least a portion of the background area; The image parameters of the target reference area are acquired, and based on the image parameters of the target reference area, the image harmonization processing is performed on the synthesis area to obtain a harmonized image.

2. The method according to claim 1, characterized in that The performing image harmonization processing on the synthesis area based on the image parameters of the target reference area includes: Based on the image parameters of the target reference area, the image parameters of the synthesized area are adjusted; wherein the difference between the image parameters of the synthesized area in the harmonized image and the image parameters of the target reference area is smaller than the difference between the image parameters of the synthesized area in the synthesized image and the image parameters of the target reference area.

3. The method according to claim 1, characterized in that The obtaining of the target reference area selected by the user from the composite image comprises: In response to a user's selection operation on any area in the composite image, the area is determined as the target reference area.

4. The method according to claim 1, characterized in that: The composite image includes a plurality of image targets; and obtaining a target reference area selected by a user from the composite image includes: segmenting the composite image into a plurality of regions corresponding to the plurality of image objects; Outputting the plurality of areas to a user; In response to a user's selection operation on any area among the multiple areas, the area is determined as the target reference area.

5. The method according to claim 4, characterized in that The step of dividing the composite image into a plurality of regions corresponding to the plurality of image objects comprises: The composite image is input into a trained image segmentation model, and the image segmentation model extracts image features contained in the composite image, determines boundary information of the multiple image targets in the composite image based on the image features, and segments the composite image into multiple regions corresponding to the multiple image targets based on the boundary information.

6. The method according to claim 1, characterized in that The acquiring the image parameters of the target reference area and performing image harmonization processing on the synthesis area based on the image parameters of the target reference area includes: Generate indication information corresponding to the target reference area; wherein the indication information is used to indicate the area to be referenced when performing image harmonization processing; The composite image and the indication information are input into the trained image harmonization model, and the image harmonization model determines the target reference area based on the indication information, obtains the image parameters of the target reference area, and performs image harmonization processing on the composite area based on the image parameters of the target reference area.

7. The method according to claim 6, characterized in that The indication information includes an attention distribution map corresponding to the target reference area; the pixels contained in the attention distribution map correspond one-to-one to the pixels contained in the composite image; the pixel values ​​of the pixels contained in the attention distribution map are used to indicate the degree of reference to the pixel when performing image harmonization processing; wherein, the pixel values ​​of the pixels corresponding to the target reference area in the attention distribution map are greater than the pixel values ​​of the pixels corresponding to other areas.

8. The method according to claim 7, characterized in that The pixel values ​​of the pixels corresponding to the target reference area in the attention distribution map are 1, and the pixel values ​​of the pixels corresponding to other areas are 0.

9. The method according to claim 7, characterized in that: The step of inputting the synthesized image and the indication information into a trained image harmonization model, and allowing the image harmonization model to determine the target reference area based on the indication information, includes: The synthesized image and the attention distribution map are input into a trained image harmonization model, and the image harmonization model determines the target reference area based on pixel values ​​of pixels included in the attention distribution map.

10. The method according to claim 7, characterized in that The obtaining of the target reference area selected by the user from the composite image comprises: Acquire multiple target reference areas selected by a user from the composite image, and reference weights respectively configured by the user for the multiple target reference areas; wherein the reference weight corresponding to any target reference area is used to indicate the degree of reference to the image parameters of the target reference area when performing image harmonization processing; The indication information includes a plurality of attention distribution maps corresponding to the plurality of target reference areas; wherein the pixel value of a pixel corresponding to any target reference area in the attention distribution map corresponding to the target reference area is proportional to the reference weight of the target reference area; The step of inputting the synthesized image and the indication information into a trained image harmonization model, determining the target reference area based on the indication information by the image harmonization model, acquiring image parameters of the target reference area, and performing image harmonization processing on the synthesized area based on the image parameters of the target reference area comprises: The composite image and the multiple attention distribution maps are input into the image harmonization model, and the image harmonization model determines the multiple target reference areas and their reference weights based on the pixel values ​​of the pixels contained in the multiple attention distribution maps, obtains the image parameters of the multiple target reference areas, and performs image harmonization processing on the composite area based on the image parameters of the multiple target reference areas and their reference weights.

11. The method according to any one of claims 1 to 10, characterized in that: The image parameters include any one or more combinations of the following parameters: color, brightness, contrast, and saturation.

12. An image harmonization processing device, characterized in that: The device comprises: A composite image acquisition unit, used to acquire a composite image to be processed; the composite image includes a composite area and a background area, and the composite area includes a partial image clipped from other images; A reference region acquisition unit, configured to output the composite image to a user and acquire a target reference region selected by the user from the composite image; wherein the target reference region includes at least a portion of the background region; The image harmonization processing unit is used to obtain the image parameters of the target reference area, and based on the image parameters of the target reference area, perform image harmonization processing on the synthesis area to obtain a harmonized image.

13. A computing device, characterized in that: include: Memory and processor; The memory has stored thereon computer programs / instructions executable by the processor; When the processor runs the computer program / instructions, the method according to any one of claims 1 to 11 is performed.

14. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images