Background image generation, image fusion method and device, electronic equipment and readable medium

Abstract

The present disclosure discloses a background image generation method and device, an image fusion method and device, an electronic device and a readable medium. The method comprises: performing instance segmentation on each of at least two target images to obtain a background segmentation map corresponding to each target image without instances; for each target image, filling in the area where the removed instances are located in the target image according to the background segmentation map of a set image to obtain a filling result of the target image, wherein the set image comprises at least two target images different from the target image; and generating a background image according to the filling results of the target images. The above technical solution uses the background segmentation map of the set image to fill in the area where the removed instances of each target image are located, and generates a background image by comprehensively considering the filling results of the target images, fully reuses the features of the background in the target images, makes the segmentation between instances and background smoother, and thus generates a high-quality background image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a background image generation, image fusion method, apparatus, electronic device, and readable medium. Background Technology

[0002] Image fusion refers to image processing of acquired related images to extract the most useful information from each image and obtain a comprehensive image. For example, when generating a cover image for a video, multiple frames from the video can be fused to obtain a high-quality fused image that reflects the key content of the video as the cover. In the process of fusing multiple frames into a single frame, a unified background image is usually required.

[0003] Currently, the main method for generating background images is to smooth foreground objects (noise points) using a large amount of data; that is, averaging all images to obtain the background image. This method has significant limitations: it requires each frame to correspond to the same viewpoint and a sufficient number of images to guarantee a smoothing effect. However, in practical applications, multiple frames are not necessarily at the same viewpoint, and video scenes are complex and varied. Background images generated using this method often exhibit poor smoothing, especially in areas where the instance and background meet, where distortion or deformation is common, failing to guarantee the quality of the generated background image. Summary of the Invention

[0004] This disclosure provides a background image generation, image fusion method, apparatus, electronic device, and readable medium for generating high-quality background images.

[0005] In a first aspect, embodiments of this disclosure provide a background image generation method, including:

[0006] Perform instance segmentation on each of the target images in at least two frames to obtain a background segmentation map with instances removed for each target image;

[0007] For each frame of target image, the region where the removed instance is located in the target image is filled according to the background segmentation map of the set image to obtain the filling result of the target image, wherein the set image includes target images that are different from the target image in the at least two frames of target images;

[0008] A background image is generated based on the filling results of each target image.

[0009] Secondly, embodiments of this disclosure also provide an image fusion method, including:

[0010] Acquire at least two frames of the target image;

[0011] A background image is generated based on the filling results of the regions where the removed instances are located in each of the target images;

[0012] The instances from each of the target images are fused into the background image to obtain a fused image.

[0013] Thirdly, embodiments of this disclosure also provide a background image generation apparatus, comprising:

[0014] The segmentation module is used to perform instance segmentation on each frame of the target image in at least two frames of target images to obtain a background segmentation map with instances removed for each frame of target image.

[0015] The filling module is used to fill in the region where the removed instance is located in the target image for each frame of target image according to the background segmentation map of the set image, so as to obtain the filling result of the target image, wherein the set image includes target images that are different from the target image in the at least two frames of target images;

[0016] The generation module is used to generate a background image based on the filling results of each target image.

[0017] Fourthly, embodiments of this disclosure also provide a fusion apparatus, comprising:

[0018] The acquisition module is used to acquire at least two frames of the target image;

[0019] The background image generation module is used to generate a background image based on the filling results of the regions where the removed instances are located in each of the target images;

[0020] The fusion module is used to fuse instances from each of the target images into the background image to obtain a fused image.

[0021] Fifthly, embodiments of this disclosure also provide an electronic device, including:

[0022] One or more processors;

[0023] Storage device for storing one or more programs;

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement the background image generation method as described in the first aspect or the image fusion method as described in the second aspect.

[0025] In a sixth aspect, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the background image generation method as described in the first aspect or the image fusion method as described in the second aspect.

[0026] This disclosure discloses a background image generation and image fusion method, apparatus, electronic device, and readable medium. The method includes: performing instance segmentation on each of at least two target images to obtain a background segmentation map for each target image corresponding to the removed instances; for each target image, filling the region where the removed instances are located in the target image according to the background segmentation map of a set image to obtain a filling result for the target image, wherein the set image includes target images different from the target image in at least two target images; and generating a background image based on the filling results of each target image. The above technical solution fills the region where the removed instances are located in each target image using the background segmentation map of a set image, and combines the filling results of each target image to generate a background image, fully utilizing the background features in each target image, making the segmentation between instances and background smoother, thereby generating a high-quality background image. Attached Figure Description

[0027] Figure 1 This is a flowchart of the background image generation method in Embodiment 1 of this disclosure;

[0028] Figure 2 This is a flowchart of the background image generation method in Embodiment 2 of this disclosure;

[0029] Figure 3 This is a schematic diagram illustrating the filling of the area where the removed instance is located in the target image in Embodiment 2 of this disclosure;

[0030] Figure 4 This is a flowchart of the background image generation method in Embodiment 3 of this disclosure;

[0031] Figure 5 This is a schematic diagram of the dilated region corresponding to an instance in the target image in Embodiment 3 of this disclosure;

[0032] Figure 6 This is a flowchart illustrating the process of obtaining the restoration results of each target image based on the filling results of each target image in Embodiment 3 of this disclosure;

[0033] Figure 7 This is a schematic diagram of the background image generated from the target image in Embodiment 3 of this disclosure;

[0034] Figure 8 This is a flowchart of the image fusion method in Embodiment 4 of this disclosure;

[0035] Figure 9 This is a schematic diagram of the fused image in Embodiment 4 of this disclosure;

[0036] Figure 10 This is a schematic diagram of the background image generation device in Embodiment 5 of this disclosure;

[0037] Figure 11This is a schematic diagram of the background image generation device in Embodiment Six of this disclosure;

[0038] Figure 12 This is a schematic diagram of the hardware structure of the electronic device in Embodiment 7 of this disclosure. Detailed Implementation

[0039] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0040] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0041] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0042] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0043] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0044] In the following embodiments, each embodiment provides optional features and examples. The various features described in the embodiments can be combined to form multiple optional solutions. Each numbered embodiment should not be regarded as only one technical solution. Furthermore, unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be combined with each other.

[0045] Example 1

[0046] Figure 1This is a flowchart of a background image generation method according to Embodiment 1 of this disclosure. This method is applicable to situations where a background image is extracted from multiple frames of images. Specifically, the method obtains a background image by fusing features from the background segmentation images of multiple frames. This method can be executed by a background image generation device, which can be implemented in software and / or hardware and integrated into an electronic device. In this embodiment, the electronic device can be a computer, laptop, server, tablet computer, or smartphone, or other device with image processing capabilities.

[0047] like Figure 1 As shown, the background image generation method in Embodiment 1 of this disclosure specifically includes the following steps:

[0048] S110. Perform instance segmentation on each of the target images in at least two frames to obtain a background segmentation map with instances removed for each target image.

[0049] In this embodiment, the target image mainly refers to the image containing feature information used to generate the background image. By fusing the background feature information from each target image together, a unified background image can be generated. The backgrounds in each target image are for the same or similar scenes, but the viewpoints may differ. In addition, each target image may include instances and background, and the position of the same instance in each target image may be different.

[0050] The main purpose of instance segmentation is to identify instances in each target image and separate these instances from the background. The remaining portion after removing the instances is the background segmentation map. Optionally, the Separate Object Instances by Location and Size (SOLO) algorithm can be used to segment each target image. Specifically, the SOLOv2 algorithm can be used to segment instances according to their location and size, which has high accuracy and real-time performance, thus improving the efficiency of generating the background map.

[0051] S120. For each frame of target image, fill in the area where the removed instance is located in the target image according to the background segmentation map of the set image to obtain the filling result of the target image, wherein the set image includes target images that are different from the target image in the at least two frames of target images.

[0052] In this embodiment, for any frame of the target image, the region where the removed instance is located has no features. The region where the removed instance is located can be filled using the background segmentation map of a set image (which can be all target images other than the target image, a portion of all target images other than the target image, or a set number of target images). This results in the filled-in target image. During this process, features from the background segmentation map of the set image are transferred and fused into the region where the removed instance is located in the target image. Based on this, a background map can be obtained by combining the filled-in results of each target image.

[0053] It is understood that in the background segmentation map of the set image, the region containing the features used to fill in the area where the removed instance in the target image is located corresponds to the area where the removed instance in the target image is located. For example, if the area where the removed instance in the target image is located is a region with a top-left corner size of A*A, then the feature information of the region with a top-left corner size of A*A in the background segmentation map of the set image can be used to fill in the area where the removed instance in the target image is located. It should be noted that, unless otherwise specified, the corresponding region mentioned in the following embodiments refers to the region in the background segmentation map of the set image that corresponds to the area where the removed instance in the target image is located.

[0054] Optionally, filling in the region in the target image containing the removed instances based on the background segmentation map of the set image can be achieved by averaging the feature information of the background segmentation map of the set image (mainly the corresponding region in the background segmentation map of the set image), and then using the average result to fill in the region in the target image containing the removed instances. For example, the set image has X background segmentation maps (X is greater than or equal to 1), denoted as B1, B2, ..., B X The average of the features of the background segmentation map of the given image can be expressed as: The features of the corresponding region in the averaged result B can be used to fill in the region where the removed instance is located in the target image.

[0055] Filling in the region of the removed instance in the target image based on the background segmentation map of the set image can also be done by: dividing the region of the removed instance in the target image into N parts, either evenly or randomly; and then using the feature information of the background segmentation map of each set image (mainly the corresponding region in the background segmentation map of the set image) to fill in one of those parts. For example, if the region of the removed instance in the target image is divided into A1 and A2, then the background segmentation map of set image 1 can be used to fill in one of those parts. Figure 1 Feature filling A1 is performed in the corresponding region, and background segmentation of Image 2 is used as a reference. Figure 2Feature filling is performed in the corresponding region A2. It's important to note that since the background segmentation map of the target image also has instances removed, to ensure the effectiveness of the filling, when assigning the regions containing the removed instances from the target image to different target images, it's necessary to ensure that the background segmentation map of the assigned target image contains features within the corresponding regions, rather than being completely blank. For example, to ensure the background segmentation of target image 1... Figure 1 In the example, the corresponding area of ​​A1 contains background content and cannot be a completely blank area where the instance has been removed.

[0056] Filling the region in the target image containing the removed instances using the background segmentation map of the set image can also be done by sequentially filling the region in the target image containing the removed instances using the feature information of the background segmentation map of each set image (mainly the corresponding region in the background segmentation map of the set image) until the region in the target image containing the removed instances is completely filled, or by using all set images for filling. For example, for the region in the target image containing the removed instances, first use the background segmentation map of set image 1. Figure 1 The corresponding region's features are filled in, but due to the background segmentation settings in Image 1... Figure 1 Since instances have been removed, if the removed instance is located within the corresponding region, then some features within that region are missing. Therefore, even after filling the region containing the removed instance in the target image with the features from that region, gaps still exist in the region containing the removed instance in the target image. In this case, background segmentation of Image 2 can be used again. Figure 2 The corresponding features are filled in, and so on, until the area where the removed instance in the target image is located is completely filled, or all the set images have been used for filling.

[0057] Filling the region in the target image containing the removed instances based on the background segmentation map of the set image can also be done as follows: For the region in the target image containing the removed instances, average the features of the common regions of the background segmentation maps of each set image (mainly the corresponding regions in the background segmentation maps of the set images) and fill the region in the target image containing the removed instances. Then, for the remaining unfilled regions, further fill them using the feature information of the background segmentation maps of each set image. For example, if the region in the target image containing the removed instances is a region with a size of A*A at the top left corner, and set images 1 to N all contain a common region of size A'*A' within the region with a size of A*A at the top left corner, where A' is smaller than A, then the feature information of the common region with a size of A'*A' at the top left corner of each set image can be averaged and filled into the A*A region in the target image. For the remaining part of the A*A region in the target image excluding the A'*A' region, refer to the above-described filling process and use the background segmentation maps of each set image to fill it together.

[0058] S130. Generate a background image based on the filling results of each target image.

[0059] In this embodiment, the features in the filling results of each target image are further fused to generate a background image. For example, the background image can be obtained by averaging the filling results of each target image, so as to fully reuse the background features in each target image.

[0060] Optionally, the process of generating the background image can be divided into two stages. In the first stage, for the region where instances have been removed in each frame of the target image, the background segmentation map of the set image can be used to fill in the gaps, resulting in a filling result for each frame of the target image. This filling result can be understood as a rough background image. In the second stage, the background image is generated based on the filling results of each target image. This stage can be understood as a repair process for the rough background image, which can further optimize the background features in each target image, resulting in a more refined background image. For example, the filling results of each target image can be averaged to obtain the background image; or, to make the segmentation between instances and background smoother, the region where the instances are located in each target image can be dilated, and then a second round of filling or averaging operations can be performed on the dilated region to further integrate the features of each filling result and obtain a high-quality background image.

[0061] In this embodiment, the background image generation method fills in the area where the instance is removed in each frame of the target image using the background segmentation map of the set image, and generates a background image by combining the filling results of each frame of the target image. This fully utilizes the background features in each target image, making the segmentation between the instance and the background smoother, thereby generating a high-quality background image.

[0062] Example 2

[0063] Figure 2 This is a flowchart of the background image generation method in Embodiment 2 of this disclosure. Based on the above embodiments, Embodiment 2 specifies the process of filling in the region where instances of the target image have been removed according to a background segmentation map of a set image. Technical features not described in detail in this embodiment can be found in any of the above embodiments.

[0064] In this embodiment, for each frame of the target image, the region containing the removed instances in the target image is filled in according to the background segmentation map of the set image to obtain the filling result of the target image. This includes: for each frame of the target image, sequentially filling the region containing the removed instances in the target image according to the feature information of the corresponding region in the background segmentation map of each set image, until the filling operation based on the feature information of the corresponding region in the background segmentation map of the last set image is completed, or until the region containing the removed instances in the target image is completely filled, to obtain the filling result of the target image. Based on this, the features of each background segmentation map can be utilized to the maximum extent, and a high-quality background image can be generated efficiently.

[0065] like Figure 2 As shown, the background image generation method in Embodiment 2 of this disclosure includes the following steps:

[0066] S210. Perform instance segmentation on each of the target images in at least two frames to obtain a background segmentation map with instances removed for each target image.

[0067] S220. For the current target image, fill in the region where the removed instance is located in the current target image based on the feature information of the corresponding region in the background segmentation map of the current set image.

[0068] Figure 3 This is a schematic diagram illustrating the filling of the region containing the removed instance in the target image, as shown in Embodiment 2 of this disclosure. Figure 3 As shown, assuming there are N target images (N is an integer greater than 2), the blank areas resembling human figures in each target image represent the regions where the removed human figures are located. The position or action of the human figures may differ in different target images. The feature information in the background segmentation image after removing human figures in target image 1 is represented by a grid; the feature information in the background segmentation image after removing human figures in target image 2 is represented by diagonal lines; the feature information in the background segmentation image after removing human figures in target image N-1 is represented by dotted textures; and the feature information in the background segmentation image after removing human figures in target image N is represented by vertical lines.

[0069] Taking filling the area where human figures have been removed from target image 1 (i.e., the current target image) as an example, in the background segmentation map of target image 2 (i.e., the current set image), the human figure shape indicated by the dashed line is the corresponding area. The feature information represented by the diagonal lines in this area can be used to fill the area in target image 1 after removing human figures. However, it is clear that the human figure shape indicated by the dashed line in the background segmentation map of target image 2 also contains some blank space (due to the removal of human figures in target image 2). Therefore, using only the feature information of the corresponding area in the background segmentation map of target image 2 cannot completely fill the area in target image 1 after removing human figures. We can continue to use the feature information of the corresponding area in the background segmentation map of the next target image to fill it. Assuming the next set image is target image N-1, the feature represented by the dotted texture within the human figure shape indicated by the dashed line in the background segmentation map of target image N-1 can be used to continue filling the area in target image 1 after removing human figures. However, it still cannot completely fill the area. Therefore, we also need to use the feature information represented by the vertical lines within the human figure shape indicated by the dashed line in the background segmentation map of target image N to continue filling the area in target image 1 after removing human figures. Thus, we can obtain the filling result of target image 1. In the filling results, the feature information of the diagonal part comes from the corresponding area of ​​the background segmentation map of the target image 2, the feature information of the dotted part comes from the corresponding area of ​​the background segmentation map of the target image N-1, and the feature information of the vertical part comes from the corresponding area of ​​the background segmentation map of the target image N.

[0070] Based on a similar principle, the filling results for target images 2 to N can be obtained. A background image can then be generated based on the filling results for each target image.

[0071] S230. Has the region containing the removed instance in the current target image been completely filled? If yes, proceed to S250; otherwise, proceed to S240.

[0072] In this embodiment, if the area in the current target image after removing human instances is completely filled after filling with the feature information of the corresponding region of the background segmentation map of the current set image, the filling operation on the current target image can be ended, and the filling result of the current target image can be obtained without using the background segmentation map of the subsequent set image for filling; if the area in the current target image after removing human instances is not completely filled after filling with the feature information of the corresponding region of the background segmentation map of the current set image, it can be further determined whether there are still background segmentation maps of set images that have not been used for filling.

[0073] S240. Is the currently set image the last set image? If yes, proceed to S250; otherwise, proceed to S290.

[0074] In this embodiment, if there are still background segmentation maps of the set image that have not been used for filling (i.e., the current set image is not the last set image), the background segmentation map of the next set image can be used for filling until the feature information of the corresponding area of ​​the background segmentation map of the last set image is used to complete the filling. At this point, regardless of whether it can be completely filled, the filling operation of the current target image can be ended, and the filling result of the current target image can be obtained.

[0075] S250, Obtain the filling result of the target image.

[0076] S260. Is the current target image the last target image? If yes, proceed to S280; otherwise, proceed to S270.

[0077] S270. Use the next target image as the current target image.

[0078] S280. Generate a background image based on the filling results of each target image.

[0079] S290: Set the next setting image as the current setting image.

[0080] The background image generation method in this embodiment performs instance segmentation on each frame of the target image. For each frame of the target image, the region where the removed instance is located is filled in the target image according to the feature information of the corresponding region in the background segmentation map of each set image. This can maximize the use of the features of each background segmentation map and efficiently generate a high-quality background image. On this basis, the background image is generated by using the background segmentation map of each target image. This can integrate the features of the background part of each target image, ensure the consistency between the background image and the background of each target image, and generate a high-quality background image.

[0081] Example 3

[0082] Figure 4 This is a flowchart of the background image generation method in Embodiment 3 of this disclosure. Embodiment 3 elaborates on the process of generating a background image based on the filling results of each target image, building upon the above embodiments. Technical features not described in detail in this embodiment can be found in any of the above embodiments.

[0083] In this embodiment, the process of generating a background image can be divided into two stages. In the first stage, for the region of the removed instance in each frame of the target image, the background segmentation map of the set image can be used to fill it, and the filling result corresponding to each frame of the target image is obtained. In the second stage, a background image is generated based on the filling result of each target image.

[0084] In this embodiment, generating a background image based on the infilling results of each target image includes: dilating the region where the instance is located in each target image to obtain the dilated region corresponding to each target image; for each frame of target image, repairing the dilated region corresponding to the target image based on the feature information of the corresponding region in the infilling results of each target image to obtain the repaired result of the target image; and generating a background image based on the repaired result of each target image. Based on this, repairing the infilling results of each target image can further smooth the edges of the instances, resulting in a more accurate background image.

[0085] like Figure 4 As shown, the background image generation method in Embodiment 3 of this disclosure includes the following steps:

[0086] S310. Perform instance segmentation on each of the target images in at least two frames to obtain a background segmentation map with instances removed for each target image.

[0087] S320. For each frame of the target image, fill in the area where the removed instance is located in the target image according to the background segmentation map of the set image to obtain the filling result of the target image.

[0088] S330. Dilate the region where the instance is located in each target image to obtain the dilated region corresponding to each target image.

[0089] In this embodiment, dilation is performed on instances in each target image. This can be understood as adding pixel values ​​to the edges of the instances, thereby expanding the overall pixel area of ​​the instances so that the dilated area includes as many instance edges as possible that are difficult to repair. Adding pixel values ​​can be achieved using a convolution template or a convolution kernel.

[0090] Figure 5 This is a schematic diagram of the dilated region corresponding to an instance in the target image of Embodiment 3 of this disclosure. For example... Figure 5 As shown, the region depicted by the bold dashed line, representing the shape of the person, is the expanded region obtained after dilution of the area containing the instance in the target image. This expanded region should be as large as possible compared to the original instance's region (the region of the person shape formed by diagonal lines, dotted textures, and vertical lines), and the edges of the original instance should be contained within the expanded region. Based on this, the expanded regions corresponding to each target image can be repaired using the filling results obtained in the first stage, making the instance edges smoother.

[0091] S340. For each frame of the target image, repair the corresponding dilated region of the target image based on the feature information of the corresponding region in the filling result of each target image, and obtain the repair result of the target image.

[0092] Optionally, in this embodiment, for each frame of target image, the infilling results of each target image are used to repair its dilated region, including: averaging the feature information of the region corresponding to the dilated region in the infilling results of each target image (including the currently repaired target image and other target images), and filling the dilated region corresponding to the target image with the averaged result, thereby obtaining the repair result of the target image.

[0093] In some embodiments, for each frame of the target image, the inflated region is repaired using the infilling results of each target image. This can also be a similar infilling operation to the first stage, for example, using feature information from the infilling results of other target images corresponding to the inflated region to fill the inflated region again. Specifically, this could involve averaging the features from the infilling results of other target images corresponding to the inflated region and then filling it again, or dividing the inflated region into several parts, either evenly or randomly, and then using the features from the infilling results of each other target image corresponding to the inflated region to fill one part respectively. Based on this, the repaired result of the target image can be obtained.

[0094] S350. Generate a background image based on the restoration results of each target image.

[0095] For example, the highest quality restoration result from the restoration results of each target image can be selected as the background image according to the requirements.

[0096] Optionally, in this embodiment, generating a background image based on the restoration results of each target image includes averaging the restoration results of each target image to obtain the background image. Based on this, the edges of the instance can be smoothed by fully utilizing the feature information of other target images.

[0097] In one embodiment, for each frame of target image, the dilated region corresponding to the target image is repaired based on the feature information of the corresponding region in the infilling result of each target image to obtain the repair result of the target image. This includes: in each iteration, for each frame of target image, the feature information of the region corresponding to the dilated region in the infilling result of each target image is averaged, and the averaged result is filled into the dilated region corresponding to the target image to obtain the repair result of the target image in this iteration; if the set conditions are not met, the next iteration process is entered; if the set conditions are met, the iteration is stopped, and the repair result of the target image in this iteration process is taken as the repair result of the target image.

[0098] In this embodiment, the second-stage repair operation can be performed iteratively multiple times until a set condition is met. For example, the set condition is: if the feature difference between the repair result obtained in this iteration and the repair result in the previous iteration of any target image is within the allowable range, the iteration can be stopped. At this time, the repair result corresponding to each target image has fully integrated the feature information in each filling result, and the edge transition is smooth, the accuracy is higher, and a higher quality repair result can be obtained.

[0099] Figure 6 This is a flowchart illustrating the process of obtaining the restoration results of each target image based on the filling results of each target image in Embodiment 3 of this disclosure. For example... Figure 6 As shown, the restoration results for each target image are obtained based on the filling results of each target image, including:

[0100] S410. Dilate the region where the instance is located in each target image to obtain the dilated region corresponding to each target image.

[0101] S420. For the current target image, the feature information of the region corresponding to the dilated region in the filling results of each target image is averaged, and the averaged result is filled into the dilated region corresponding to the current target image to obtain the repair result of the current target image in this iteration process.

[0102] S430. Are the set conditions met? If yes, proceed to S460 to obtain the final repair result of the target image; otherwise, proceed to S440.

[0103] Optionally, the conditions include: the feature differences between the repair result of the target image in the current iteration and the repair result in the previous iteration are within the allowable range.

[0104] In some embodiments, the set condition may also be that the number of iterations reaches a specified number, or the iteration duration reaches a specified duration, etc.

[0105] S440. Is the current target image the last target image? If yes, proceed to S460; otherwise, proceed to S450.

[0106] S450: Use the next target image as the current target image.

[0107] S460, Proceed to the next iteration.

[0108] In this embodiment, if the error between the repair result obtained by the current target image in this iteration and the repair result obtained in the previous iteration is small, the iteration can be stopped. At this time, the repair result obtained by each target image in the last repair can be used as the final repair result. If the error is large, the iteration will not be stopped. Furthermore, if there are still target images whose filling results have not been repaired in this iteration, the next target image can be selected as the current target image, and its filling results can be repaired. If the current target image is the last target image, that is, the filling results of each target image in this iteration have been repaired, then the current iteration process is completed, and the next iteration begins.

[0109] For example, the iterative repair process for the infilling results of each target image includes:

[0110] Assume there are N target images, corresponding to N infilling results;

[0111] In the first iteration, for the dilated region in the filling result Bj (1≤j≤N) of the target image j obtained in the first stage, the feature information of the corresponding region in the filling results B1, B2...BN of each target image is averaged and filled into the dilated region to repair the dilated region, so as to obtain the repair result Bj1 of the target image j in the first iteration.

[0112] Then, in the second iteration, similarly, for the dilated region in the repair result Bj1 of the target image j, the feature information of the corresponding regions in B1, B2...BN is averaged and filled into the dilated region to repair the dilated region in Bj1, and the repair result Bj2 of the target image j in the second iteration is obtained.

[0113] This process continues until, during an iteration, the feature difference between the restoration result of any target image and the restoration result of the previous iteration is within an acceptable range. Then, the iteration stops, and the restoration results of all target images at this point are averaged to obtain the background image.

[0114] It should be noted that the number of times each target image has been repaired may differ when the repair operation stops. For example, in the first iteration, the infill results of 10 target images are repaired sequentially. In the second iteration, when the repair of the third target image is reached, the error between the repair result obtained in the second iteration and the repair result obtained in the first iteration is already small, so the iteration can be stopped. At this point, the repair results of the target images from the first to the third frame have actually undergone two iterations of repair, while the infill results of the target images from the fourth to the tenth frame have actually undergone one iteration of repair.

[0115] In some embodiments, it is also possible that, when certain conditions are met, the repair operation may be stopped only after the inpainting results of each target image have been repaired in the current iteration. In this case, when the repair operation is stopped, each target image has been repaired the same number of times.

[0116] It should be noted that the filling result obtained in the first stage is actually a rough background image. The repair operation in the second stage can further improve the filling accuracy. Incorrect pixel values ​​in the expansion area will be gradually corrected by the correct pixel values, while the correct pixel values ​​in the background outside the instance will not change with iteration, ensuring that the generated background image fully integrates the feature information of each target image, and the edge processing effect is better, and the transition between the instance and the background is more natural.

[0117] The principle of the two-stage algorithm (filling and repair) provided in this embodiment is as follows:

[0118] Suppose there are N target images, denoted as I1, I2, ..., I... N The masks of instances in each frame of the target image are denoted as M1, M2, ..., M. N Then, the rough filling result B corresponding to the target image in the i-th (i = 1, 2, ... N) frame can be obtained by the following formula. i :

[0119] B i =B i,N ;

[0120] Among them, B i,k F represents the filling result obtained by filling the region of the removed instance in the target image of the i-th frame with the background segmentation map of the target image of the k-th frame. i,k This represents the remaining region in the target image after filling the region of the removed instances in the target image of the i-th frame with the background segmentation map of the target image of the k-th frame (i.e., the region that still needs to be filled using the background segmentation maps of other frames), where k is not equal to i, k = 1, 2, ..., N. This represents element-wise matrix multiplication.

[0121] Since instance segmentation algorithms are often inaccurate at instance contour edges, further fine-tuning is required. The mask for the dilated region corresponding to each instance in the target image of each frame is denoted as M1. * M2 * ,…,M N * The infilling results of the target images in each frame are denoted as B1, B2, ..., B N Then, the restoration result of the target image in the i-th frame can be obtained according to the following formula:

[0122] The iteration ends and the repair operation stops when the following condition is met in the m-th iteration:

[0123] Mean(·) represents the matrix average function.

[0124] After the iteration is complete, the restoration results of each target image are averaged to obtain the final background image:

[0125]

[0126] Optionally, after obtaining the background segmentation map of the removed instance corresponding to each target image frame, the method further includes: selecting a target image frame as a reference frame, determining the affine transformation matrix between each target image other than the reference frame and the reference frame according to a feature point matching algorithm; and aligning the background segmentation maps of each target image other than the reference frame with the background segmentation map of the reference frame according to the affine transformation matrix.

[0127] In this embodiment, although the target images are for the same scene, due to different shooting angles, camera shake, or errors, the backgrounds of the target images are not perfectly aligned. Directly generating a background image based on the background segmentation maps of each target image will result in local distortion, deformation, or blurring, affecting the accuracy and visual effect of the background image. Therefore, before generating the background image based on the background segmentation maps of each target image, a target image frame can be selected as a reference frame, and other background segmentation maps of each target image can be aligned with this reference frame. The reference frame can be the target image with the highest image quality, the first target image, the last target image, or a target image located in the middle, etc.

[0128] For example, an affine transformation matrix between each target image and a reference frame is determined using a feature point matching algorithm. This affine transformation matrix describes the transformation relationship of matched feature points from the target image to the reference frame. Affine transformations include linear transformations and translation transformations. The feature point matching algorithm can be a scale-invariant feature transform (SIFT) algorithm. Specifically, first, key feature points in the background of each target image are extracted. These key feature points do not disappear due to factors such as illumination, scale, or rotation. Then, based on the feature vectors of each key point, pairwise comparisons are performed between the key points in the target image and the reference frame to find several pairs of matching feature points, thus establishing the correspondence between the feature points and obtaining the affine transformation matrix. Optionally, if the number of key feature points available for registration in a target image frame is less than a set threshold, that target image frame can be discarded.

[0129] Figure 7This is a schematic diagram of the background image generated from the target image in Embodiment 3 of this disclosure. For example... Figure 7 As shown, after registering (aligning) multiple target images, removing instances, and utilizing the feature information of the background, a high-quality background image can be obtained through a two-stage algorithm (i.e., filling and repair operations). This image can fully preserve the features of the background in the original target images and has a good smoothing effect on the edges of instances.

[0130] The background image generation method in this embodiment improves the accuracy and image quality of the generated background image by selecting a target image frame as a reference frame and aligning other background segmentation images of each target image with the reference frame. By obtaining a rough background image of each target image in the first stage, the dilated region of the instance is iteratively repaired in the second stage to fuse the features of each filling result. This allows the generated background image to fully reuse the feature information of the background segmentation images of each target image, and the processing effect of the instance edges is smoother, and the transition between the instance and the background is more natural, further improving the quality of the background image.

[0131] Example 4

[0132] Figure 8 This is a flowchart of the image fusion method in Embodiment 4 of this disclosure. This method is applicable to situations where multiple frames of images are fused into a single image. Specifically, a unified background image is generated based on the multiple frames of images, and instances from each frame are fused into the generated background image. Application scenarios for this method include extracting multiple frames of images from a video and generating a fused image based on the extracted frames as the video cover; it can also be used to generate a fused image based on a set of images as an identifier or folder icon for that set of images, or further to obtain thumbnails that reflect the main content of that set of images. This method can be executed by an image fusion device, which can be implemented by software and / or hardware and integrated into an electronic device. The electronic device in this embodiment can be a computer, laptop, server, tablet computer, or smartphone, or other device with image processing capabilities. It should be noted that technical details not described in detail in this embodiment can be found in any of the above embodiments.

[0133] like Figure 8 As shown, the background image generation method in Embodiment 1 of this disclosure specifically includes the following steps:

[0134] S510: Acquire at least two frames of target images.

[0135] In this embodiment, the target image mainly refers to an image containing background features. By fusing the background features from various target images, a unified background image can be extracted. The backgrounds in each target image refer to the same scene, but the viewpoints may differ. Furthermore, a target image may contain instance portions and background portions, and the positions of the same instance in each target image may differ. The target image can be read from an electronic device or downloaded from a database; it can be a series of captured frames or a series of frames extracted from video, etc.

[0136] Optionally, acquiring at least two target images includes: identifying action sequence frames in the video based on an action recognition algorithm, and using the action sequence frames as target images.

[0137] In this embodiment, an action recognition algorithm can be used to identify valid action sequence frames from a video. Each action sequence frame, when linked sequentially with the human figures, can express a complete action or behavior, and these action sequence frames can serve as target images. For example, an open-pose human pose recognition algorithm can be used to estimate the pose of human figures in a video. Specifically, firstly, the coordinates of the human joints in each frame of the video are extracted, and the distance change matrix between adjacent frames is calculated. Then, the video is segmented, and video features are generated using the distance change matrix corresponding to each segment. Finally, a trained classifier is used to classify the video features. If a segment of video is identified as having action or behavior feature sequences from a pre-defined behavior library, then the frames corresponding to that segment are action sequence frames. Another example is using an instance segmentation algorithm to extract the contours of the human figure in each keyframe and express their pose. A clustering algorithm is used to extract key features of the pose, and based on these key features, a dynamic time warping (DTW) algorithm is used to complete action recognition. For example, action recognition algorithms can be implemented using the Temporal Segment Network (TSM) model, which is trained on the Kinetics-400 dataset and can be used to recognize 400 actions, meeting the requirement of recognizing and displaying the actions of instances on the cover.

[0138] Optionally, if valid action sequence frames are identified, the degree of background difference between each action sequence frame can be further determined. If the degree of background difference is within the allowable range, image fusion is performed on each action sequence frame.

[0139] Optionally, acquiring at least two target images includes: determining the similarity between keyframes in the video based on a pre-trained network; dividing the keyframes into multiple groups according to the similarity; and selecting the keyframes in the group with the most keyframes as the target images.

[0140] In this embodiment, keyframes mainly refer to frames that reflect key content or scene changes in the video. Examples include frames containing the main characters, frames representing exciting or classic scenes, frames showing significant scene changes, and frames containing key character actions. At least two keyframes are extracted from the video. Based on this, the keyframes can be grouped, and the keyframe in the group with the most keyframes is used as the target image for image fusion.

[0141] For example: Select one frame at a set interval (e.g., 20 frames) from the video to reduce the data volume, and extract keyframes using an image evaluation algorithm. Then, cluster the keyframes based on their inter-frame similarity, such as color tone, scene content, or whether they contain the same instances. Finally, perform similarity measurement in the feature space based on a pre-trained convolutional neural network. For example, use a Visual Geometry Group (VGG) network, specifically a VGG19 network. Each frame image input to VGG19 yields a 1000-dimensional vector, and the angle between two vectors represents their similarity. Assume the feature vector of the i-th frame is F. i Assume the feature vector of the j-th frame is F j Then the similarity is expressed as: Here, <·> represents the inner product operation, and ||·|| represents the magnitude of the vector. Based on similarity, the images in the video can be divided into several groups, and the group with the most frames is selected as the target image to be fused.

[0142] S520. Generate a background image based on the filling results of the regions where the removed instances are located in each of the target images.

[0143] In this embodiment, the regions where removed instances are located in each frame of the target image can be filled in. For example, for a frame of the target image, the texture features of the background portion after instance removal can be used to fill in the regions where the removed instances are located, thereby completing the background restoration of the target image and obtaining the filled-in result of the target image. Alternatively, for a frame of the target image, the feature information of the background segmentation map of a set image (which can be all target images other than the target image, or a portion of all target images other than the target image, or a set number of target images) can be used to fill in the regions where the removed instances are located in the target image. That is, the features in the background segmentation map of the set image are transferred and fused into the regions where the removed instances are located in the target image, resulting in the filled-in result of the target image. Based on this, the features in the filled-in results of each target image can be further fused to generate a background image. In this process, the background image is generated by combining the filled-in results of each frame of the target image, fully utilizing the features of each target image to generate a high-quality background image.

[0144] S530. The instances in each of the target images are fused into the background image to obtain a fused image.

[0145] In this embodiment, instances in each target image are separated from the background, wherein each background can be used to generate a unified background image; each instance can be merged into the background image to obtain a fused image.

[0146] For example, instances can be extracted from each target image and added to a background image generated based on the infilling results of each target image. In this case, instances and background from multiple frames of target images can be displayed using a single static image, effectively reducing computational resources and storage space usage. During this process, operations such as cropping, scaling, rotating, and stitching can also be performed on the instances to be added.

[0147] Optionally, instances in each target image can be arranged sequentially in the background image (e.g., from left to right, or from right to left), and the arrangement of each instance in the background image can be made consistent with its relative position in the original target image, so that the fused image is visually closer to the shape of each instance in the original target image; or, instances in each target image can be arranged freely in the background image.

[0148] Taking the application scenario of generating video cover images as an example, an image fusion method can be used to extract the background image of any video based on temporal redundancy information, and then fuse instances from multiple target images into this background image. This process can include:

[0149] Frame extraction: Extract multiple frames from the video at set intervals (e.g., 20 frames), and select keyframes based on image quality algorithms;

[0150] Scene clustering: Clustering is performed based on the inter-frame similarity of each keyframe, and the keyframes in the class (i.e., a group) containing the most keyframes are taken as the target images.

[0151] Instance segmentation: Separating instances from the background in each target image;

[0152] Image registration: Align the background segmentation maps of each target image according to the affine transformation matrix;

[0153] Two-stage algorithm: Filling in and repairing the regions where instances were removed in each target image to obtain the background image;

[0154] Instance fusion: Add instances from each target image to the background image to obtain a fused image.

[0155] Optionally, the degree of fusion between the instances in each target image and the background image decreases sequentially according to the temporal order of each target image.

[0156] Figure 9 This is a schematic diagram of the fused image in Embodiment 4 of this disclosure. For example... Figure 9 As shown, the five human figures in the fused image can originate from five target images, which can be derived from a video clip depicting a skateboard jump. To make the fused image generated from the target images more closely resemble the original video content, after obtaining a unified background image, the figures from each target image can be arranged in appropriate positions within that background image. It is understandable that, typically, using five target images from a video to represent the action of a human figure requires creating animated images from these target images, which is computationally intensive and requires significant storage space. However, the image fusion method in this embodiment effectively integrates the feature information of multiple target images using a fused image, displaying rich image content with limited resources.

[0157] Furthermore, the five character instances in the fused image, from right to left, complete a skateboard jump action from takeoff, flight, to landing. The leftmost character instance is further back in the sequence, with the leftmost instance corresponding to the last target image. The further left the character instance is, the lower its blending degree with the background image, which can also be interpreted as lower transparency. Based on this, while displaying multiple action sequence frames through a static fused image, the temporal sequence of each instance is also evident, creating a visual persistence effect that makes the displayed actions or behaviors more concrete and vivid.

[0158] Optionally, the method for generating a background image based on the filling result of the region where the removed instance is located in each of the target images is determined according to any of the above embodiments.

[0159] The image fusion method in this embodiment can display the rich features of multiple target images by using the fused image. In addition, by generating a background image by integrating the filling results of each target image, the features of each target image are fully utilized to generate a high-quality background image, which can also improve the quality of the fused image.

[0160] Example 5

[0161] Figure 10 This is a schematic diagram of the background image generation device in Embodiment 5 of this disclosure. For details not covered in this embodiment, please refer to the above embodiments.

[0162] like Figure 10 As shown, the device includes:

[0163] The segmentation module 610 is used to perform instance segmentation on each frame of the target image in at least two frames of target images to obtain a background segmentation map with instances removed for each frame of target image.

[0164] The filling module 620 is used to fill in the area where the removed instance is located in the target image according to the background segmentation map of the set image for each frame of target image, so as to obtain the filling result of the target image, wherein the set image includes target images that are different from the target image in the at least two frames of target images;

[0165] The generation module 630 is used to generate a background image based on the filling results of each of the target images.

[0166] The background image generation device in this embodiment fills in the background segmentation map of each frame of target image and generates a background image by combining the filling results of each frame of target image. It makes full use of the background features in each target image, making the segmentation between the instance and the background smoother, thereby generating a high-quality background image.

[0167] Based on the above, module 620 is filled in, specifically for:

[0168] For each frame of the target image, the region where the removed instance is located in the target image is filled in sequentially according to the feature information of the corresponding region in the background segmentation map of each set image, until the filling operation based on the feature information of the corresponding region in the background segmentation map of the last set image is completed, or until the region where the removed instance is located in the target image is completely filled, to obtain the filling result of the target image.

[0169] Based on the above, module 630 is generated, including:

[0170] The dilation unit is used to dilate the region where the instance is located in each of the target images to obtain the dilated region corresponding to each of the target images;

[0171] The repair unit is used to repair the dilated region corresponding to the target image for each frame of target image based on the feature information of the corresponding region in the filling result of each target image, so as to obtain the repair result of the target image;

[0172] The generation unit is used to generate the background image based on the restoration results of each target image.

[0173] Based on the above, the repair unit is specifically used for:

[0174] The feature information of the region corresponding to the dilated region in the filling results of each target image is averaged, and the averaged result is filled into the dilated region corresponding to the target image to obtain the repair result of the target image.

[0175] Based on the above, the generation unit is specifically used to: average the repair results of each target image to obtain the background image.

[0176] Based on the above, the repair unit is specifically used for:

[0177] In each iteration, for each frame of the target image, the feature information of the region corresponding to the dilated region in the filling result of each target image is averaged, and the averaged result is filled into the dilated region corresponding to the target image to obtain the repair result of the target image in this iteration.

[0178] If the set conditions are not met, proceed to the next iteration.

[0179] If the set conditions are met, the iteration stops, and the repair result of the target image in this iteration is taken as the repair result of the target image.

[0180] Based on the above, the set conditions include:

[0181] The feature differences between the repair result of the target image in this iteration and the repair result in the previous iteration are within the allowable range.

[0182] Based on the above, the device also includes:

[0183] The calculation module is used to select a target image as a reference frame after obtaining the background segmentation map of the removed instance corresponding to each target image frame, and determine the affine transformation matrix between each target image other than the reference frame and the reference frame according to the feature point matching algorithm.

[0184] An alignment module is used to align the background segmentation maps of each of the target images, excluding the reference frame, with the background segmentation map of the reference frame according to the affine transformation matrix.

[0185] The background image generation apparatus described above can execute the background image generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0186] Example 6

[0187] Figure 11 This is a schematic diagram of the background image generation apparatus in Embodiment Six of this disclosure. For details not covered in this embodiment, please refer to the above embodiments. Figure 11 As shown, the device includes:

[0188] Acquisition module 710 is used to acquire at least two frames of target images;

[0189] Image fusion module 720 is used to generate a background image based on the filling results of the regions where the removed instances are located in each of the target images;

[0190] The fusion module 730 is used to fuse instances from each of the target images into the background image to obtain a fused image.

[0191] The image fusion device in this embodiment can display the rich features of multiple target images by using the fused image. In addition, by generating a background image by integrating the filling results of each target image, the features of each target image are fully utilized to generate a high-quality background image, which also improves the quality of the fused image.

[0192] Based on the above, obtaining at least two target images includes: identifying action sequence frames in the video based on an action recognition algorithm, and using the action sequence frames as the target images.

[0193] Based on the above, obtaining at least two frames of target images includes:

[0194] The similarity between keyframes in a video is determined based on a pre-trained network.

[0195] The keyframes are divided into multiple groups based on the similarity.

[0196] The keyframes in the group with the most keyframes are used as the target image.

[0197] Based on the above, the degree of fusion between the instances in each target image and the background image decreases sequentially according to the temporal order of each target image.

[0198] Based on the above, the method for generating a background image according to the filling result of the region where the removed instance is located in each of the target images is determined according to any of the above embodiments.

[0199] The image fusion apparatus described above can execute the image fusion method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0200] Example 7

[0201] Figure 12 This is a schematic diagram of the hardware structure of the electronic device in Embodiment 7 of this disclosure. Figure 12 A schematic diagram of the structure of an electronic device 800 suitable for implementing embodiments of the present disclosure is shown. The electronic device 800 in the embodiments of the present disclosure includes, but is not limited to, devices with image processing capabilities such as computers, laptops, servers, tablets, or smartphones. Figure 12 The illustrated electronic device 800 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0202] like Figure 12 As shown, the electronic device 800 may include one or more processing devices (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The one or more processing devices 801 implement the traffic packet forwarding method provided in this disclosure. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing devices 801, ROM 802, and RAM 803 are interconnected via a bus 805. An input / output (I / O) interface 804 is also connected to the bus 805.

[0203] Typically, the following devices can be connected to I / O interface 804: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc., for storing one or more programs; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 An electronic device 800 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0204] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0205] It should be noted that the computer-readable medium described in this disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0206] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0207] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0208] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: perform instance segmentation on each of at least two target images to obtain a background segmentation map corresponding to the removed instances for each target image; for each target image, fill in the region where the removed instances are located in the target image according to the background segmentation map of a set image to obtain a filling result for the target image, wherein the set image includes target images different from the target image in the at least two target images; and generate a background image based on the filling results of each target image. Alternatively, the electronic device may: acquire at least two target images; generate a background image based on the filling results of the regions where the removed instances are located in each target image; and merge the instances in each target image into the background image to obtain a merged image.

[0209] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0210] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0211] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0212] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0213] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0214] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0215] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0216] Although this topic has been described using language specific to structural features and / or methodological logic, it should be understood that the topic defined in the accompanying example book is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the example book.

[0217] According to one or more embodiments of this disclosure, Example 1 provides a background image generation method, including:

[0218] Perform instance segmentation on each of the target images in at least two frames to obtain a background segmentation map with instances removed for each target image;

[0219] For each frame of target image, the region where the removed instance is located in the target image is filled according to the background segmentation map of the set image to obtain the filling result of the target image, wherein the set image includes target images that are different from the target image in the at least two frames of target images;

[0220] A background image is generated based on the filling results of each target image.

[0221] Example 2, according to the method described in Example 1, wherein for each frame of the target image, filling in the region where the removed instances are located based on the background segmentation map of the set image to obtain the filling result of the target image includes:

[0222] For each frame of the target image, the region where the removed instance is located in the target image is filled in sequentially according to the feature information of the corresponding region in the background segmentation map of each set image, until the filling operation based on the feature information of the corresponding region in the background segmentation map of the last set image is completed, or until the region where the removed instance is located in the target image is completely filled, to obtain the filling result of the target image.

[0223] Example 3, according to the method described in Example 1, the step of generating a background image based on the filling results of each of the target images includes:

[0224] Dilation processing is performed on the regions where instances are located in each of the target images to obtain the dilated regions corresponding to each of the target images;

[0225] For each frame of the target image, the dilated region corresponding to the target image is repaired based on the feature information of the corresponding region in the filling result of each target image, so as to obtain the repair result of the target image;

[0226] The background image is generated based on the restoration results of each target image.

[0227] Example 4, according to the method described in Example 3, involves repairing the dilated region corresponding to the target image based on the feature information of the corresponding region in the filling results of each target image to obtain the repaired target image, including:

[0228] The feature information of the region corresponding to the dilated region in the filling results of each target image is averaged, and the averaged result is filled into the dilated region corresponding to the target image to obtain the repair result of the target image.

[0229] Example 5, according to the method described in Example 3, the step of generating the background image based on the restoration results of each of the target images includes:

[0230] The background image is obtained by averaging the restoration results of each target image.

[0231] Example 6, according to the method described in Example 3, wherein for each frame of the target image, the dilated region corresponding to the target image is repaired based on the feature information of the corresponding region in the filling result of each target image to obtain the repair result of the target image, includes:

[0232] In each iteration, for each frame of the target image, the feature information of the region corresponding to the dilated region in the filling result of each target image is averaged, and the averaged result is filled into the dilated region corresponding to the target image to obtain the repair result of the target image in this iteration.

[0233] If the set conditions are not met, proceed to the next iteration.

[0234] If the set conditions are met, the iteration stops, and the repair result of the target image in this iteration is taken as the repair result of the target image.

[0235] Example 7: According to the method described in Example 6, the set conditions include:

[0236] The feature differences between the repair result of the target image in this iteration and the repair result in the previous iteration are within the allowable range.

[0237] Example 8, following the method described in Example 1, further includes, after obtaining the background segmentation map of the removed instance corresponding to each frame of the target image:

[0238] Select a target image frame as a reference frame, and determine the affine transformation matrix between each target image other than the reference frame and the reference frame according to the feature point matching algorithm;

[0239] Based on the affine transformation matrix, the background segmentation maps of each of the target images other than the reference frame are aligned with the background segmentation map of the reference frame.

[0240] According to one or more embodiments of this disclosure, Example 9 provides an image fusion method, including:

[0241] Acquire at least two frames of the target image;

[0242] A background image is generated based on the filling results of the regions where the removed instances are located in each of the target images;

[0243] The instances from each of the target images are fused into the background image to obtain a fused image.

[0244] Example 10, according to the method described in Example 9, the acquisition of at least two frames of target images includes:

[0245] Action sequence frames in a video are identified based on an action recognition algorithm, and these action sequence frames are used as the target image.

[0246] Example 11 According to the method described in Example 9, acquiring at least two frames of target images includes:

[0247] The similarity between keyframes in a video is determined based on a pre-trained network.

[0248] The keyframes are divided into multiple groups based on the similarity.

[0249] The keyframes in the group with the most keyframes are used as the target image.

[0250] Example 12: According to the method described in Example 9, the degree of fusion between the instances in each target image and the background image decreases sequentially according to the temporal order of each target image.

[0251] Example 13: The method for generating a background image based on the filling result of the region where the removed instance is located in each of the target images, according to the method described in Example 9, is determined according to any one of Examples 1-8.

[0252] According to one or more embodiments of this disclosure, Example 14 provides a background image generation apparatus, comprising:

[0253] The segmentation module is used to perform instance segmentation on each frame of the target image in at least two frames of target images to obtain a background segmentation map with instances removed for each frame of target image.

[0254] The filling module is used to fill in the region where the removed instance is located in the target image for each frame of target image according to the background segmentation map of the set image, so as to obtain the filling result of the target image, wherein the set image includes target images that are different from the target image in the at least two frames of target images;

[0255] The generation module is used to generate a background image based on the filling results of each target image.

[0256] According to one or more embodiments of this disclosure, Example 15 provides an image fusion apparatus, comprising:

[0257] The acquisition module is used to acquire at least two frames of the target image;

[0258] The background image generation module is used to generate a background image based on the filling results of the regions where the removed instances are located in each of the target images;

[0259] The fusion module is used to fuse instances from each of the target images into the background image to obtain a fused image.

[0260] According to one or more embodiments of this disclosure, Example 16 provides an electronic device comprising:

[0261] One or more processors;

[0262] Storage device for storing one or more programs;

[0263] When the one or more programs are executed by the one or more processors, the one or more processors implement the background image generation method as described in any one of Examples 1-8, or the image fusion method as described in any one of Examples 9-13.

[0264] According to one or more embodiments of the present disclosure, Example 17 provides a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements a background image generation method as described in any one of Examples 1-8, or an image fusion method as described in any one of Examples 9-13.

Claims

1. A method for generating a background image, characterized in that, include: Instance segmentation is performed on each of the target images in at least two frames to obtain a background segmentation map with instances removed for each target image. The scenes of the target images are the same but the viewpoints are different, and the positions of the same instance in the target images are different. For each frame of target image, the region where the removed instance is located is filled in the target image according to the background segmentation map of the set image to obtain the filling result of the target image. The set image includes target images that are different from the target image in the at least two frames of target images, and the number of set images is at least two. A background image is generated based on the filling results of each target image.

2. The method according to claim 1, characterized in that, For each frame of the target image, the region containing the removed instances in the target image is filled in according to the background segmentation map of the set image to obtain the filling result of the target image, including: For each frame of the target image, the region where the removed instance is located in the target image is filled in sequentially according to the feature information of the corresponding region in the background segmentation map of each set image, until the filling operation based on the feature information of the corresponding region in the background segmentation map of the last set image is completed, or until the region where the removed instance is located in the target image is completely filled, to obtain the filling result of the target image.

3. The method according to claim 1, characterized in that, The step of generating a background image based on the filling results of each of the target images includes: Dilation processing is performed on the regions where instances are located in each of the target images to obtain the dilated regions corresponding to each of the target images; For each frame of the target image, the dilated region corresponding to the target image is repaired based on the feature information of the corresponding region in the filling result of each target image, so as to obtain the repair result of the target image; The background image is generated based on the restoration results of each target image.

4. The method according to claim 3, characterized in that, The step of repairing the dilated region corresponding to the target image based on the feature information of the corresponding region in the filling result of each target image to obtain the repair result of the target image includes: The feature information of the region corresponding to the dilated region in the filling results of each target image is averaged, and the averaged result is filled into the dilated region corresponding to the target image to obtain the repair result of the target image.

5. The method according to claim 3, characterized in that, The step of generating the background image based on the restoration results of each of the target images includes: The background image is obtained by averaging the restoration results of each target image.

6. The method according to claim 3, characterized in that, For each frame of the target image, the dilated region corresponding to the target image is repaired based on the feature information of the corresponding region in the filling result of each target image, to obtain the repair result of the target image, including: In each iteration, for each frame of the target image, the feature information of the region corresponding to the dilated region in the filling result of each target image is averaged, and the averaged result is filled into the dilated region corresponding to the target image to obtain the repair result of the target image in this iteration. If the set conditions are not met, proceed to the next iteration. If the set conditions are met, the iteration stops, and the repair result of the target image in this iteration is taken as the repair result of the target image.

7. The method according to claim 6, characterized in that, The setting conditions include: The feature differences between the repair result of the target image in this iteration and the repair result in the previous iteration are within the allowable range.

8. The method according to claim 1, characterized in that, After obtaining the background segmentation map of the removed instance corresponding to each frame of the target image, the following is also included: Select a target image frame as a reference frame, and determine the affine transformation matrix between each target image other than the reference frame and the reference frame according to the feature point matching algorithm; Based on the affine transformation matrix, the background segmentation maps of each of the target images other than the reference frame are aligned with the background segmentation map of the reference frame.

9. An image fusion method, characterized in that, include: Acquire at least two frames of the target image; A background image is generated based on the filling results of the regions where the removed instances are located in each of the target images. The filling results of the regions where the removed instances are located in the target images are obtained by filling the regions where the removed instances are located in the target images with the background segmentation map of a set image. The set image includes target images that are different from the target images in the at least two target images. The number of set images is at least two. The scenes of each of the target images are the same but the viewpoints are different. The positions of the same instance in each of the target images are different. The instances in each of the target images are fused into the background image to obtain a fused image containing multiple instances.

10. The method according to claim 9, characterized in that, The acquisition of at least two frames of target images includes: Action sequence frames in a video are identified based on an action recognition algorithm, and these action sequence frames are used as the target image.

11. The method according to claim 9, characterized in that, The acquisition of at least two frames of target images includes: The similarity between keyframes in a video is determined based on a pre-trained network. The keyframes are divided into multiple groups based on the similarity. The keyframes in the group with the most keyframes are used as the target image.

12. The method according to claim 9, characterized in that, The degree of fusion between the instances in each target image and the background image decreases sequentially according to the temporal order of each target image.

13. The method according to claim 9, characterized in that, The method for generating a background image based on the filling result of the region where the removed instance is located in each of the target images is determined according to any one of claims 1-8.

14. A background image generation apparatus, characterized in that, include: The segmentation module is used to perform instance segmentation on each frame of target images in at least two frames of target images to obtain a background segmentation map with instances removed for each frame of target images. The scenes of the target images are the same but the viewpoints are different, and the positions of the same instance in the target images are different. The filling module is used to fill in the region where the removed instance is located in the target image for each frame of target image according to the background segmentation map of the set image, so as to obtain the filling result of the target image. The set image includes target images that are different from the target image in the at least two frames of target images, and the number of set images is at least two. The generation module is used to generate a background image based on the filling results of each target image.

15. An image fusion apparatus, characterized in that, include: The acquisition module is used to acquire at least two frames of the target image; The background image generation module is used to generate a background image based on the filling result of the region where the removed instance is located in each of the target images. The filling result of the region where the removed instance is located in the target image is obtained by filling the region where the removed instance is located in the target image with the background segmentation map of the set image. The set image includes target images that are different from the target image in at least two frames of target images. The number of set images is at least two. The scene of each target image is the same but the viewpoint is different. The position of the same instance in each target image is different. The fusion module is used to fuse instances from each of the target images into the background image to obtain a fused image containing multiple instances.

16. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the background image generation method as described in any one of claims 1-8, or the image fusion method as described in any one of claims 9-13.

17. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the background image generation method as described in any one of claims 1-8, or the image fusion method as described in any one of claims 9-13.

Images