Image processing method, apparatus, device, and medium

Abstract

The present disclosure provides an image processing method, device, equipment and medium, a specific embodiment of the method includes: acquiring a background image and a foreground image to be fused, and acquiring a noise image; determining a target region in the background image to fuse the foreground image; based on the target region, determining a target position code of the foreground image relative to the background image; using a target model to denoise the noise image multiple times based on the background image, the foreground image and the target position code, to fuse the foreground image into the target region of the background image, to obtain a target image. This embodiment fuses the foreground image and the background image with high quality, improves the fusion effect, makes the target image more realistic, and improves the user experience.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of image processing technology, and more particularly to image processing methods, apparatus, devices, and media. Background Technology

[0002] With the continuous development of network technology and image processing technology, digital images are increasingly used in people's lives, providing entertainment services, bringing numerous conveniences, and adding more fun to people's lives. When processing images, people often need to merge different images, for example, merging the foreground of one image into the background of another. Currently, there is a need for a solution for image fusion processing. Summary of the Invention

[0003] Embodiments of this disclosure describe an image processing method, apparatus, device, and medium.

[0004] According to a first aspect, an image processing method is provided, the method comprising: acquiring a background image and a foreground image to be fused, and acquiring a noisy image; determining a target region in the background image to fuse the foreground image; determining a target position code of the foreground image relative to the background image based on the target region; and performing multiple denoising operations on the noisy image using a target model based on the background image, the foreground image, and the target position code, so as to fuse the foreground image into the target region of the background image to obtain a target image.

[0005] According to a second aspect, an image processing apparatus is provided, the apparatus comprising: an acquisition unit configured to acquire a background image and a foreground image to be fused, and to acquire a noisy image; a first determination unit configured to determine a target region in the background image to be fused with the foreground image; a second determination unit configured to determine a target position code of the foreground image relative to the background image based on the target region; and a fusion unit configured to perform multiple denoising operations on the noisy image using a target model based on the background image, the foreground image, and the target position code, so as to fuse the foreground image into the target region of the background image to obtain a target image.

[0006] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform any of the methods described in the first aspect.

[0007] According to a fourth aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any one of the first aspects.

[0008] According to an embodiment of this disclosure, an image processing scheme is provided. By acquiring a background image and a foreground image to be fused, a target region for fusing the foreground image in the background image is determined. Based on the target region, a target position code of the foreground image relative to the background image is determined. A target model is then used to perform multiple denoising operations on the noisy image based on the background image, the foreground image, and the target position code, thereby fusing the foreground image into the target region of the background image to obtain the target image. This achieves high-quality fusion of the foreground and background images, improving the fusion effect, making the target image more realistic, and enhancing the user experience. Attached Figure Description

[0009] Figure 1 This is a schematic diagram illustrating an application scenario of an image processing scheme according to an exemplary embodiment of the present disclosure;

[0010] Figure 2 This is a schematic diagram of an exemplary system architecture for applying embodiments of this disclosure;

[0011] Figure 3 This is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure;

[0012] Figure 4 This disclosure is a schematic diagram illustrating an image processing scenario according to an exemplary embodiment;

[0013] Figure 5 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure;

[0014] Figure 6 This is a schematic block diagram of an electronic device provided in some embodiments of this disclosure. Detailed Implementation

[0015] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0016] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as electronic devices, applications, servers, or storage media, that perform the operations of the technical solutions disclosed herein, based on the prompt message.

[0017] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0018] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0019] The technical solutions provided in this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0020] With the continuous development of network technology and image processing technology, digital images are increasingly used in people's lives, providing entertainment services, bringing numerous conveniences, and adding more fun to people's lives. When processing images, people often need to merge different images, for example, merging the foreground of one image into the background of another. In related technologies, background and foreground in different images can generally be merged using image matting techniques. However, image matting techniques have certain limitations. The merged image obtained in this way has poor blending of background and foreground, lacks realism, and is difficult to meet users' requirements for image quality.

[0021] This disclosure provides an image processing scheme that acquires a background image and a foreground image to be fused, determines the target region in the background image for fusing the foreground image, determines the target position code of the foreground image relative to the background image based on the target region, and uses a target model to perform multiple denoising operations on the noisy image based on the background image, foreground image, and target position code to fuse the foreground image into the target region of the background image, thereby obtaining the target image. This achieves high-quality fusion of the foreground and background images, improves the fusion effect, makes the target image more realistic, and enhances the user experience.

[0022] See Figure 1 This is a schematic diagram illustrating an application scenario of an image processing scheme according to an exemplary embodiment. Figure 1As shown, this scheme involves a diffusion model for image fusion processing. Therefore, the figure illustrates any time step of the reverse process (image fusion process) and any time step of the forward process (noise addition process) of the diffusion model. In the scenario of using the diffusion model for image fusion processing, refer to the reverse process; in the scenario of model training, refer to both the reverse and forward processes. The image fusion process and the model training process are described below.

[0023] like Figure 1 As shown, specifically in image fusion application scenarios, see [link to relevant documentation]. Figure 1 The reverse process illustrated begins by having the user input the background image 101 and foreground image 102 to be fused, as well as the region 103 in the background image 101 to be fused with the foreground image 102, into the image processing client via a terminal device. Optionally, the user can also input text guidance information 104, which can be used to constrain certain features of the generated fused image. For example, Figure 1 The text guidance information 104 can be used to constrain the color of the motorcycle in the fused image. The image processing client can obtain the position code 105 corresponding to the background image 101, and the position code 106 of region 103 in the background image 101. And based on the position code 105 and the position code 106, the position code 107 of the foreground image 102 relative to the background image 101 is determined.

[0024] After entering the denoising stage, the image processing client can perform multiple denoising processes, each corresponding to a time step. At any given time step, the image processing client can acquire the image to be denoised 108 and its corresponding position code 109. Specifically, in the first time step, the image to be denoised 108 can be randomly acquired Gaussian noise; in subsequent time steps, the image to be denoised 108 can be the denoised result obtained from the previous time step.

[0025] Next, the image processing client can stitch together the background image 101, foreground image 102, and the image to be denoised 108 to obtain stitching features 110. Position codes 105, 107, and 109 are added to the stitching features 110. The text guidance information 104 is then encoded to obtain text encoding features 111. The stitched image 110 and text encoding features 111 are input into the diffusion model M. The diffusion model M, through a shared attention mechanism, calculates shared features based on the stitched image 110 and text encoding features 111, and predicts the noise to be removed from the image to be denoised 108 based on the shared features and time step information. Finally, the diffusion model M can denoise the image to be denoised 108 to obtain the denoised result image 112 corresponding to the current time step.

[0026] It should be noted that the diffusion model M can be trained using conventional methods. However, in order to achieve better fusion results, the inventors further improved the training method for the diffusion model M. The specific process is as follows:

[0027] See Figure 1 The reverse and forward processes are illustrated. First, initial training can be performed in the first stage, using conventional training methods to obtain an intermediate model m. This intermediate model m can fuse the background image 101 and the foreground image 102, achieving a good result. Next, optimization training is performed in the second stage. Specifically, an intermediate model m can be obtained as a diffusion model M1, which may be a model requiring further training. In the second stage, the model parameters of diffusion model M1 need to be further updated. Another intermediate model m is obtained as a diffusion model M2, which can be used as a reference model. In the second stage, the model parameters of diffusion model M2 do not need to be updated. In addition, the background image 101 and the foreground image 102 to be fused, as well as the region 103 in the background image 101 fused with the foreground image 102, can be obtained. Fusion sample images 113 and 114 are also obtained.

[0028] In this embodiment, fused sample image 113 and fused sample image 114 are both images obtained by fusing background image 101 and foreground image 102. The background and foreground in fused sample image 113 are more harmoniously blended, but the background of fused sample image 113 differs significantly from the background of background image 101. Fusion sample image 113 can be an image obtained through manual processing, or it can be an image selected from multiple images obtained by fusing background image 101 and foreground image 102 from an intermediate model m; this embodiment is not limited in this regard. The background and foreground in fused sample image 114 are poorly blended, but the background of fused sample image 114 has a high degree of consistency with the background of background image 101. Fusion sample image 114 can be an image obtained by directly pasting the foreground of foreground image 102 into region 103 of background image 101.

[0029] Referring to the forward process, the fused sample image 113 can first undergo n-step noise addition processing to obtain the first intermediate image corresponding to each of the n time steps. The first intermediate image corresponding to the r-th time step can be represented as intermediate image 115tr (r represents the specific time step). Then, the fused sample image 114 can undergo n-step noise addition processing to obtain the second intermediate image corresponding to each of the n time steps. The second intermediate image corresponding to the r-th time step can be represented as intermediate image 116tr.

[0030] Next, the first and second intermediate images corresponding to any time step can be selected. For example, intermediate image 115tk and intermediate image 116tk corresponding to the k-th time step can be selected. Using... Figure 1 The reverse process is illustrated below. Using diffusion model M1, the noise to be removed from the intermediate image 115tk is predicted, resulting in prediction result G1. Using diffusion model M1, the noise to be removed from the intermediate image 116tk is predicted, resulting in prediction result G2. Using diffusion model M2, the noise to be removed from the intermediate image 115tk is predicted, resulting in prediction result G3. Using diffusion model M2, the noise to be removed from the intermediate image 116tk is predicted, resulting in prediction result G4.

[0031] Finally, based on Figure 1 In the forward process shown, the actual noise added at the k-th time step, prediction results G1, G2, G3, and G4 are used to calculate the prediction loss. The model parameters of the diffusion model M1 are then updated to optimize the training of the diffusion model M1, with the aim of reducing the prediction loss.

[0032] It should be noted that, in Figure 1In the image fusion application scenario described in this embodiment, the image processing client directly performs image fusion as an example. In other embodiments, the image processing client can also transmit the background image 101, foreground image 102, region 103, and text guidance information 104 to the image processing server deployed on the service platform via the network. The image processing server then generates a target image based on the background image 101, foreground image 102, region 103, and text guidance information 104, and transmits the target image to the image processing client via the network to provide the target image to the user. See details below. Figure 2 Example.

[0033] Figure 2 This is a schematic diagram of an exemplary system architecture for applying embodiments of this disclosure.

[0034] like Figure 2 As shown, system architecture 200 may include terminal device 202, network 203, and server 204. It should be understood that... Figure 2 The number or type of terminal devices, networks, and servers shown are merely illustrative. Depending on implementation needs, there can be any number or type of terminal devices, networks, and servers.

[0035] Network 203 is a medium used to provide communication links between terminal devices and servers. Network 203 can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0036] The terminal device 202 is equipped with an image processing client. The terminal device 202 can interact with the server via network 203 to receive or send requests or information. The terminal device 202 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, and smart wearable devices.

[0037] Server 204 houses an image processing server. Server 204 can store, analyze, and process received data, and can also send control commands or requests to terminal devices or other servers. The server can provide image processing services in response to user service requests. It is understood that a single server can provide one or more services, and the same service can be provided by multiple servers.

[0038] based on Figure 2In the system architecture shown in this embodiment, user 201 can input instruction information for image fusion via terminal device 202. This instruction information may include, but is not limited to, the background image and foreground image to be fused, the target region in the background image for fusion with the foreground image, and text guidance information to constrain the image fusion effect. Terminal device 202 can transmit this instruction information to server 204 via network 203. After receiving the instruction information, server 204 can generate a target image based on the instruction information. Finally, server 204 can return the target image to terminal device 202 via network 203, allowing user 201 to view and save the target image via terminal device 202.

[0039] The present disclosure will now be described in detail with reference to specific embodiments.

[0040] Figure 3 This is a flowchart illustrating an image processing method according to an exemplary embodiment. The method can be executed by an image processing client or an image processing server. In this embodiment, the image processing client is installed on a terminal device, which may include, but is not limited to, mobile terminal devices such as smartphones, smart wearable devices, tablets, laptops, and desktop computers. The image processing server is deployed in a service platform, which can be any device, server, or device cluster with computing and processing capabilities. It should be noted that this method can be applied to the image fusion application stage or the model training stage, and the method may include the following steps:

[0041] like Figure 3 As shown, in step 301, a background image and a foreground image to be fused are acquired, and a noise image is acquired; and in step 302, a target region in the background image to be fused with the foreground image is determined.

[0042] In this embodiment, the noisy image can be an image randomly generated using Gaussian white noise. This embodiment does not limit the specific method of generating the noisy image. The background image and foreground image can be images that need to be fused. Foreground objects can be extracted from the foreground image and fused into the target region of the background image. For example, see... Figure 4 The foreground object in the foreground image 402 is a car, and the background image 401 includes region 403. The car in the foreground image 402 can be merged into region 403 of the background image 401 to obtain the target image 404.

[0043] In step 303, the target position encoding of the foreground image relative to the background image is determined based on the target region.

[0044] In this embodiment, the target position code of the foreground image relative to the background image can be determined based on the target region in the background image. Specifically, firstly, a first position code of the background image, a second position code of the foreground image, and a third position code of the target region can be obtained. The position code of the image can be the specific position information of pixels in the image; for example, this specific position information can be coordinate information. Therefore, the first position code can be the coordinate information of each pixel in the background image in the coordinate system corresponding to the background image. The second position code can be the coordinate information of each pixel in the foreground image in the coordinate system corresponding to the foreground image. The third position code can be the coordinate information of each pixel in the target region in the coordinate system corresponding to the target region.

[0045] Next, the transformation relationship information of the third position code relative to the first position code can be determined, and the target position code can be determined based on this transformation relationship information and the second position code. Specifically, the pixels in the target region can be mapped to the coordinate system corresponding to the background image to determine the transformation relationship information of the third position code relative to the first position code. This transformation relationship information can be, for example, a transformation matrix. Then, each pixel in the foreground image can be mapped to the coordinate system corresponding to the target region, and then, according to the above transformation relationship information, each pixel in the foreground image can be mapped to the coordinate system corresponding to the background image to obtain the target position code of the foreground image relative to the background image.

[0046] Since this embodiment references the target position encoding of the foreground image relative to the background image when performing image fusion, there is no need to scale the foreground image to adapt to the background image, thus avoiding the loss of information in the foreground image and improving the image fusion effect.

[0047] In step 304, the target image is obtained by using the target model to perform multiple denoising operations on the noisy image based on the background image, foreground image, and target location encoding.

[0048] In this embodiment, a target model can be used to perform multiple denoising processes on a noisy image based on the background image, foreground image, and target location encoding through a shared attention mechanism. This process fuses the foreground image into the target region of the background image, thereby obtaining the target image. The target model can be a diffusion model. Specifically, for any one of the multiple denoising processes, the following steps can be included: If denoising is performed for the first time, i.e., in the denoising process corresponding to the first time step, the noisy image can be acquired as the image to be processed. If denoising is not performed for the first time, i.e., in the denoising process corresponding to a time step other than the first time step, the intermediate image obtained from the previous denoising process, i.e., the denoising result obtained in the denoising process corresponding to the previous time step, can be acquired as the image to be processed.

[0049] Next, the background image, foreground image, and image to be processed can be stitched together to obtain stitched features. These stitched features are then input into the target model. Through a shared attention mechanism, shared features are calculated based on the stitched features and the target location encoding. Based on these shared features, the noise to be removed is predicted, and the image to be processed is denoised according to the required noise removal, resulting in the denoising result for the current time step.

[0050] Since this embodiment stitches together the background image, foreground image, and image to be processed to obtain stitched features, and calculates shared features based on the stitched features through a shared attention mechanism, the target model can better predict the noise to be removed based on the shared features, thereby further improving the image fusion effect.

[0051] It should be noted that this method can also be used in the training process of the target model. Specifically, the inventors discovered through creative work that after training the target model using conventional training methods, the target images generated by the target model typically have two problems. One is that the foreground and background of the target image blend well, making the target image look more harmonious; however, the background of this type of target image usually differs too much from the background image, resulting in poor consistency. The other is that the background of the target image differs little from the background image, resulting in good consistency; however, the foreground and background of this type of target image blend poorly, making the target image look stiff and disharmonious. Therefore, the inventors added a process of updating the target model through a local loss term, solving the above problems and improving the performance of the target model.

[0052] Specifically, during the training of the target model, the following target update operation for the model to be trained can be performed iteratively multiple times: using the model to be trained to predict noise at a specified time step to obtain a first prediction result; based on the first prediction result, obtaining a first loss term for the foreground and a second loss term for the background; and updating the model to be trained based on the first and second loss terms. Here, both the first and second loss terms are local loss terms that can reflect local losses in the image.

[0053] In other embodiments, the training process may include a first stage and a second stage. In the first stage, the target model can be initially trained using conventional training methods to obtain an intermediate model. In the second stage, the intermediate model can be further trained by performing the target update operation described above multiple times to obtain a target model with better performance. Specifically, in the second stage, the target update operation may include the following steps: First, obtain an intermediate model as the model to be trained, which needs to update its model parameters. Then, obtain another intermediate model as a reference model, which does not need to update its model parameters.

[0054] Next, the noise at a specified time step is predicted using the model to be trained, resulting in a first prediction result. The noise at the same specified time step is then predicted using a reference model, resulting in a second prediction result. Based on the first prediction result, a first loss term for the foreground and a second loss term for the background are obtained. Based on the second prediction result, a third loss term for the foreground and a fourth loss term for the background are obtained. Finally, the model to be trained is updated with the aim of reducing the sum of the first, second, third, and fourth loss terms.

[0055] In this embodiment, the target model is initially trained in the first stage to obtain an intermediate model. Then, based on the intermediate model, the target model is optimized in the second stage, resulting in better performance of the target model.

[0056] In other embodiments, a first sample image and a second sample image may be acquired before performing the target update operation. The first sample image is a negative sample for the background and a positive sample for the foreground, while the second sample image is a positive sample for the background and a negative sample for the foreground. Specifically, both the first and second sample images are obtained by fusing the background and foreground images. The background and foreground in the first sample image are more harmoniously blended, but the background of the first sample image differs significantly from the background of the background image. The first sample image can be a manually processed image or an image selected from multiple images obtained by fusing the background and foreground images from an intermediate model; this embodiment is not limited in this regard. The background and foreground in the second sample image are less blended, but the background of the second sample image has higher consistency with the background of the background image. The second sample image can be an image obtained by directly pasting the foreground of the foreground image onto the target area of ​​the background image.

[0057] Next, the first sample image undergoes multi-step noise addition processing to obtain multiple first intermediate images corresponding to multiple time steps. The second sample image undergoes the same multi-step noise addition processing to obtain multiple second intermediate images corresponding to multiple time steps. The model to be trained can then perform predictions based on the first intermediate images corresponding to a specified time step to obtain first prediction noise, and perform predictions based on the second intermediate images corresponding to a specified time step to obtain second prediction noise. Finally, a first loss term for the background can be determined based on the first prediction noise, and a second loss term for the foreground can be determined based on the second prediction noise.

[0058] Specifically, for example, a first real noise can be obtained, which is the real noise corresponding to a specified time step during the noise addition process for the first sample image. A second real noise can be obtained, which is the real noise corresponding to a specified time step during the noise addition process for the second sample image. A first difference between the first real noise and the first predicted noise is calculated. Using a mask corresponding to the target region, the difference between the background and foreground in the first difference is separated. Based on the difference between the foreground and background in the first difference, a first loss term for the foreground is determined. A second difference between the second real noise and the second predicted noise is calculated. Using a mask corresponding to the target region, the difference between the background and foreground in the second difference is separated. Based on the difference between the background and background in the second difference, a second loss term for the background is determined. Finally, the model to be trained can be updated with the aim of reducing the sum of the first and second loss terms.

[0059] Because this embodiment uses the first sample image and the second sample image, and adopts the method of local positive and negative samples, it achieves optimized training of the target model, thereby further optimizing the performance of the target model.

[0060] In other embodiments, the training process includes a first stage and a second stage. In the first stage, the target model is initially trained using conventional training methods to obtain an intermediate model. In the second stage, the intermediate model is further trained by performing a target update operation through multiple iterations. Specifically, in the second stage, before performing the target update operation, a first sample image and a second sample image can be acquired. The first sample image represents negative samples for the background and positive samples for the foreground, while the second sample image represents positive samples for the background and negative samples for the foreground. An intermediate model is then acquired as the model to be trained, and this model requires parameter updates. Another intermediate model is acquired as a reference model, which does not require parameter updates.

[0061] Next, the first sample image undergoes multi-step noise addition processing to obtain multiple first intermediate images corresponding to multiple time steps. The second sample image undergoes the same multi-step noise addition processing to obtain multiple second intermediate images corresponding to multiple time steps. The model to be trained can then be used to make predictions based on the first intermediate images corresponding to a specified time step, resulting in first prediction noise. Prediction can then be made based on the second intermediate images corresponding to the specified time step, resulting in second prediction noise. Using the reference model, predictions can then be made based on the first intermediate images corresponding to the specified time step, resulting in third prediction noise. Prediction can then be made based on the second intermediate images corresponding to the specified time step, resulting in fourth prediction noise.

[0062] Then, a first real noise can be obtained, which is the real noise corresponding to a specified time step during the noise addition process of the first sample image. A second real noise can be obtained, which is the real noise corresponding to a specified time step during the noise addition process of the second sample image. A first difference between the first real noise and the first predicted noise is calculated. Using a mask corresponding to the target region, the difference between the background and foreground in the first difference is separated. Based on the difference between the foreground in the first difference, a first loss term for the foreground is determined. A second difference between the second real noise and the second predicted noise is calculated. Using a mask corresponding to the target region, the difference between the background and foreground in the second difference is separated. Based on the difference between the background in the second difference, a second loss term for the background is determined. A third difference between the first real noise and the third predicted noise is calculated. Using a mask corresponding to the target region, the difference between the background and foreground in the third difference is separated. Based on the difference between the foreground in the third difference, a third loss term for the foreground is determined. A fourth difference between the second real noise and the fourth predicted noise is calculated. Using a mask corresponding to the target region, the difference between the background and foreground in the fourth difference is separated. Based on the difference between the background in the fourth difference, a fourth loss term for the background is determined. Finally, the model to be trained can be updated with the aim of reducing the sum of the first, second, third, and fourth loss terms.

[0063] Because this embodiment uses the prediction results of the intermediate model and the model to be trained to jointly determine the loss term, the target model prediction becomes more stable, thus improving the prediction ability of the target model.

[0064] This disclosure provides an image processing method that acquires a background image and a foreground image to be fused, determines a target region in the background image for fusing the foreground image, determines a target position code of the foreground image relative to the background image based on the target region, and uses a target model to perform multiple denoising operations on the noisy image based on the background image, foreground image, and target position code to fuse the foreground image into the target region of the background image, thereby obtaining the target image. This achieves high-quality fusion of the foreground and background images, improves the fusion effect, makes the target image more realistic, and enhances the user experience.

[0065] It should be noted that although the operations of the methods of this disclosure embodiment are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0066] Corresponding to the foregoing embodiments of the image processing method, this disclosure also provides embodiments of the image processing apparatus.

[0067] like Figure 5 As shown, Figure 5 This is a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure. The apparatus may include: an acquisition unit 501, a first determination unit 502, a second determination unit 503, and a fusion unit 504.

[0068] The acquisition unit 501 is configured to acquire the background image and foreground image to be fused, as well as the noisy image.

[0069] The first determining unit 502 is configured to determine the target region in the background image that is fused with the foreground image.

[0070] The second determining unit 503 is configured to determine the target position encoding of the foreground image relative to the background image based on the target region.

[0071] The fusion unit 504 is configured to use the target model to perform multiple denoising operations on the noisy image based on the background image, the foreground image, and the target location encoding, so as to fuse the foreground image into the target region of the background image to obtain the target image.

[0072] In some embodiments, the second determining unit 503 is configured to: acquire a first position code of the background image, a second position code of the foreground image, and a third position code of the target region; determine transformation relationship information of the third position code relative to the first position code; and determine the target position code based on the transformation relationship information and the second position code.

[0073] In other embodiments, the fusion unit 504 is configured to: utilize the target model, through a shared attention mechanism, to perform multiple denoising processes on the noisy image based on the background image, foreground image, and target location encoding.

[0074] In other embodiments, for any one of the multiple denoising processes, the fusion unit 504 performs the following steps: acquiring the image to be processed, wherein if denoising is being performed for the first time, the noisy image is acquired as the image to be processed; if denoising is not being performed for the first time, the intermediate image obtained from the previous denoising process is acquired as the image to be processed. The background image, foreground image, and image to be processed are stitched together to obtain stitched features. A shared attention mechanism is used to calculate shared features based on the stitched features and the target location encoding; based on the shared features, denoising is performed on the image to be processed.

[0075] In other embodiments, the apparatus can be used in the training process of a target model. The apparatus further includes an update unit that iteratively performs the following target update operation for the model to be trained: using the model to be trained to predict noise at a specified time step to obtain a first prediction result; based on the first prediction result, obtaining a first loss term for the foreground and a second loss term for the background; and updating the model to be trained based on the first loss term and the second loss term.

[0076] In other implementations, the training process includes a first stage and a second stage. In the first stage, the initial model corresponding to the target model is updated to obtain an intermediate model. In the second stage, the update unit iteratively performs a target update operation multiple times. The target update operation further includes: using the intermediate model to predict noise at a specified time step to obtain a second prediction result; and based on the second prediction result, obtaining a third loss term for the foreground and a fourth loss term for the background. Specifically, updating the model to be trained based on the first and second loss terms includes updating the model to be trained with the aim of reducing the sum of the first, second, third, and fourth loss terms.

[0077] In other embodiments, before the update unit performs the target update operation, the following operations are also performed: acquiring a first sample image and a second sample image. The first sample image is a negative sample for the background and a positive sample for the foreground, and the second sample image is a positive sample for the background and a negative sample for the foreground. The first sample image undergoes multi-step noise addition processing to obtain multiple first intermediate images corresponding to multiple time steps, and the second sample image undergoes multi-step noise addition processing to obtain multiple second intermediate images corresponding to multiple time steps.

[0078] The process involves using the model to be trained to predict noise at a specified time step to obtain a first prediction result. Based on the first prediction result, a first loss term for the foreground and a second loss term for the background are obtained. This includes: using the model to be trained to predict noise based on a first intermediate image corresponding to the specified time step to obtain first prediction noise; and using a second intermediate image corresponding to the specified time step to predict noise to obtain second prediction noise. Based on the first prediction noise, a first loss term is determined; and based on the second prediction noise, a second loss term is determined.

[0079] In other implementations, the training process includes a first stage and a second stage. In the first stage, the initial model corresponding to the target model is updated to obtain an intermediate model. In the second stage, the update unit iteratively performs a target update operation multiple times. The target update operation further includes: using the intermediate model, making a prediction based on a first intermediate image corresponding to a specified time step to obtain a third prediction noise; making a prediction based on a second intermediate image corresponding to a specified time step to obtain a fourth prediction noise; determining a third loss term based on the third prediction noise; and determining a fourth loss term based on the fourth prediction noise. Updating the model to be trained, at least based on the first and second loss terms, includes updating the model to be trained with the aim of reducing the sum of the first, second, third, and fourth loss terms.

[0080] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiments of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0081] The following is for reference. Figure 6 , Figure 6 This is a schematic block diagram of an electronic device provided for some embodiments of this disclosure. The electronic device 920 is, for example, suitable for implementing the image processing method provided in the embodiments of this disclosure. The electronic device 920 can be a terminal device, etc., and can be used to implement a client or server. The electronic device 920 can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that... Figure 6 The illustrated electronic device 920 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0082] like Figure 6As shown, the electronic device 920 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 into a random access memory (RAM) 923. The RAM 923 also stores various programs and data required for the operation of the electronic device 920. The processing unit 921, ROM 922, and RAM 923 are interconnected via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0083] Typically, the following devices can be connected to I / O interface 925: input devices 926 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 927 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 928 including, for example, magnetic tapes, hard disks, etc.; and communication devices 929. Communication device 929 allows electronic device 920 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 6 An electronic device 920 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 920 may alternatively implement or have more or fewer devices. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0084] According to embodiments of this disclosure, the image processing method described above can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the image processing method described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When the computer program is executed by the processing device 921, the functions defined in the image processing method provided by embodiments of this disclosure can be implemented.

[0085] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods provided in this disclosure.

[0086] It should be noted that the computer-readable medium described in the embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0087] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0088] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0089] Those skilled in the art will recognize that the functions described in the embodiments of this disclosure in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0090] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this disclosure. It should be understood that the above descriptions are merely specific implementations of the embodiments of this disclosure and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of this disclosure should be included within the scope of protection of this invention.

Claims

1. An image processing method, the method comprising: Obtain the background and foreground images to be fused, as well as the noisy image; Determine the target region in the background image that will be fused with the foreground image; Based on the target region, the target position code of the foreground image relative to the background image is determined; The noisy image is denoised multiple times using a target model based on the background image, the foreground image, and the target location encoding, so that the foreground image is fused into the target region of the background image to obtain the target image.

2. The method according to claim 1, wherein, Determining the target position encoding of the foreground image relative to the background image based on the target region includes: Obtain the first position code of the background image, the second position code of the foreground image, and the third position code of the target region; Determine the transformation relationship information of the third position code relative to the first position code; The target location code is determined based on the transformation relationship information and the second location code.

3. The method according to claim 1, wherein, The step of using a target model to perform multiple denoising operations on the noisy image based on the background image, the foreground image, and the target location encoding includes: Using the target model, and through a shared attention mechanism, multiple denoising processes are performed on the noisy image based on the background image, the foreground image, and the target location encoding.

4. The method according to claim 3, wherein, Each of the multiple denoising processes includes the following steps: Obtain the image to be processed; wherein, if the denoising process is performed for the first time, the noisy image is obtained as the image to be processed; if the denoising process is not performed for the first time, the intermediate image obtained from the previous denoising process is obtained as the image to be processed. The background image, the foreground image, and the image to be processed are stitched together to obtain stitched features; A shared attention mechanism is used to calculate shared features based on the splicing features and the target position encoding; Based on the shared features, the image to be processed is subjected to noise reduction.

5. The method according to claim 1, wherein, The method is used in the training process of the target model; the method further includes iteratively performing the following target update operation on the model to be trained: The noise at a specified time step is predicted using the model to be trained, and a first prediction result is obtained; Based on the first prediction result, a first loss term for the foreground and a second loss term for the background are obtained; The model to be trained is updated based on the first loss term and the second loss term.

6. The method according to claim 5, wherein, The training process includes a first stage and a second stage; in the first stage, the initial model corresponding to the target model is updated to obtain an intermediate model; In the second phase, the target update operation is executed iteratively multiple times; The target update operation also includes: The noise at the specified time step is predicted using the intermediate model to obtain a second prediction result; Based on the second prediction result, a third loss term for the foreground and a fourth loss term for the background are obtained; The step of updating the model to be trained based on the first loss term and the second loss term includes: The model to be trained is updated with the aim of reducing the sum of the first loss term, the second loss term, the third loss term, and the fourth loss term.

7. The method according to claim 5, wherein, Before performing the target update operation, the following is also included: Acquire a first sample image and a second sample image; wherein, the first sample image is a negative sample for the background and a positive sample for the foreground; the second sample image is a positive sample for the background and a negative sample for the foreground. The first sample image is subjected to multi-step noise addition processing to obtain multiple first intermediate images corresponding to multiple time steps; the second sample image is subjected to multi-step noise addition processing to obtain multiple second intermediate images corresponding to multiple time steps. The step of using the model to be trained to predict noise at a specified time step to obtain a first prediction result; and based on the first prediction result, obtaining a first loss term for the foreground and a second loss term for the background, includes: Using the model to be trained, a first prediction noise is obtained by predicting based on the first intermediate image corresponding to the specified time step; and a second prediction noise is obtained by predicting based on the second intermediate image corresponding to the specified time step. Based on the first prediction noise, the first loss term is determined, and based on the second prediction noise, the second loss term is determined.

8. The method according to claim 7, wherein, The training process includes a first stage and a second stage; in the first stage, the initial model corresponding to the target model is updated to obtain an intermediate model; In the second phase, the target update operation is executed iteratively multiple times; The target update operation also includes: Using the intermediate model, a third prediction noise is obtained by predicting based on the first intermediate image corresponding to the specified time step; and a fourth prediction noise is obtained by predicting based on the second intermediate image corresponding to the specified time step. Based on the third prediction noise, the third loss term is determined; based on the fourth prediction noise, the fourth loss term is determined. Updating the model to be trained based at least on the first loss term and the second loss term includes: The model to be trained is updated with the aim of reducing the sum of the first loss term, the second loss term, the third loss term, and the fourth loss term.

9. An image processing apparatus, the apparatus comprising: The acquisition unit is configured to acquire the background image and foreground image to be fused, as well as the noisy image; The first determining unit is configured to determine a target region in the background image that is fused with the foreground image; The second determining unit is configured to determine the target position code of the foreground image relative to the background image based on the target region; The fusion unit is configured to perform multiple denoising operations on the noisy image based on the background image, the foreground image, and the target location encoding using a target model, so as to fuse the foreground image into the target region of the background image to obtain a target image.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-8.

11. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-8.

Images