Method and apparatus for processing an image

Abstract

Embodiments of this disclosure provide methods, apparatus, and computer program products for processing images. The method includes: performing style transfer on an original image based on a background image to obtain an intermediate image, wherein the intermediate image contains the same content as the original image and has the same style as the background image; cropping an object image from the intermediate image, wherein the object image contains a target object; and compositing the object image with the background image to obtain a composite image.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of image processing, and more specifically, to methods, apparatus, and computer program products for processing images. Background Technology

[0002] Image synthesis is a crucial research area in computer vision. Its purpose is to integrate different visual elements into a specific image to create a composite image. Image synthesis has wide applications not only in artistic creation and entertainment but also in assisting other computer vision tasks. However, the quality of synthesized images often falls short of expectations; for example, the synthesized image may appear abrupt, thus affecting the application prospects of image synthesis. Summary of the Invention

[0003] In view of the need for improvements to existing technologies, embodiments of this disclosure provide methods, apparatus and computer program products for processing images.

[0004] On one hand, embodiments of this disclosure provide an image processing method, including: performing style transfer on an original image based on a background image to obtain an intermediate image, wherein the intermediate image contains the same content as the original image and has the same style as the background image; cropping an object image from the intermediate image, wherein the object image contains a target object; and compositing the object image with the background image to obtain a composite image.

[0005] On the other hand, embodiments of this disclosure provide an image processing apparatus, comprising: at least one processor; and a memory coupled to the at least one processor, which stores executable instructions that, when executed by the at least one processor, cause the at least one processor to perform the method described above.

[0006] On the other hand, embodiments of this disclosure provide a computer program product that stores a computer program that, when executed, implements the above-described method.

[0007] On the other hand, embodiments of this disclosure provide a computer-readable storage medium storing executable code that, when executed, implements the above-described method. Attached Figure Description

[0008] The above and other objects, features and advantages of the embodiments of this disclosure will become more apparent from the more detailed description of the embodiments in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same elements.

[0009] Figure 1An example of a quality issue with synthesized images is shown.

[0010] Figure 2 An exemplary process for image processing according to an embodiment of this disclosure is shown.

[0011] Figure 3 An exemplary process for generating an original image according to an embodiment of this disclosure is shown.

[0012] Figure 4 An exemplary process for style transfer according to embodiments of this disclosure is illustrated.

[0013] Figure 5 An exemplary process for cropping an object image according to an embodiment of this disclosure is shown.

[0014] Figure 6 An exemplary process for obtaining a semantic graph according to an embodiment of this disclosure is shown.

[0015] Figure 7 An exemplary process for determining a target semantic region using a large language model, according to embodiments of this disclosure, is illustrated.

[0016] Figure 8 A schematic flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0017] Figure 9 A schematic block diagram of an apparatus according to an embodiment of the present disclosure is shown.

[0018] Figure 10 A schematic block diagram of an apparatus according to an embodiment of the present disclosure is shown. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to various embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims.

[0020] In recent years, image synthesis has been applied in an increasing number of scenarios. Besides everyday applications such as artistic creation and entertainment, a significant application of image synthesis is the training of image-related machine learning models. Training machine learning models typically requires a large number of training images, but often it's difficult to obtain enough training images from real-world scenarios. Synthesized images obtained through image synthesis tasks can effectively augment the training image set. However, one current challenge is that the quality of the synthesized images may be unsatisfactory; for example, the synthesized images may appear inconsistent, thus affecting their subsequent applications, such as training machine learning models.

[0021] Figure 1 An example of a quality issue with synthesized images is shown.

[0022] exist Figure 1 In the example, the original image 110 is a daytime image of a street, where the target object is a speed bump 112, and the background image 120 is an image of an underground parking lot. The image compositing task is to composite the speed bump 112 from the original image 110 into the background image 120. For example, through simple operations such as cropping, geometric transformation, and pasting, the speed bump 112 can be composited into the background image 120 to obtain the composite image 130.

[0023] However, the composite image 130 has some problems. For example... Figure 1 As shown, in the composite image 130, on the one hand, the hue of the speed bump 112 is very different from that of the background image 120, and on the other hand, the position of the speed bump 112 does not conform to reality. This makes the composite image 130 not very harmonious in style and unreasonable in semantics.

[0024] It is evident that although a synthesized image is obtained through an image synthesis task, such a synthesized image cannot be used as a valid image. Therefore, embodiments of this disclosure propose an image processing technique that can effectively improve the quality of the synthesized image, thereby promoting its subsequent applications. A detailed description will follow with specific embodiments.

[0025] Figure 2 An exemplary process for image processing according to an embodiment of this disclosure is shown.

[0026] In stage 220, style transfer can be performed on the original image 202 based on the background image 201.

[0027] For example, assuming background image 201 has a first style and original image 202 has a second style, then performing style transfer on the original image 202 can be understood as converting the original image 202 from the second style to the first style. The resulting image is referred to as the intermediate image in this paper. Figure 2 The intermediate image is labeled 203. During style transfer, the content of the original image 202 is not changed; therefore, the intermediate image 203 can contain the same content as the original image 202. For example, the original image 202 can contain the target object and its environment, and the intermediate image 203 also contains the target object and its environment. However, because the style of the original image 202 is changed to the style of the background image 201 during style transfer, the intermediate image 203 has the same style as the background image 201 (such as the first style mentioned above).

[0028] In stage 240, an object image 204 can be cropped from the intermediate image 203. The object image 204 may contain the target object; therefore, the object image 204 can be understood as an image of the target object. This cropping can be implemented in various applicable ways. For example, as mentioned earlier, the intermediate image 203 has the same content as the original image 202, and the semantic information of the original image 202 is generally known. Therefore, the semantic information of the intermediate image 203 is also known, and this semantic information may include the semantic tags of the target object. Thus, the object image 204 can be cropped from the intermediate image 203 based on the semantic tags of the target object. For example, pixels representing the target object can be extracted from the intermediate image 203 based on the semantic tags of the target object to obtain the object image 204. As another example, a bounding box or boundary polygon for enclosing the target object can be formed in the intermediate image 203 based on the semantic tags of the target object, and then the object image 204 can be extracted from the intermediate image 203 based on this bounding box or boundary polygon.

[0029] In stage 280, the object image 204 can be combined with the background image 201 to obtain the composite image 208.

[0030] In some existing methods, an object image containing the target object might be directly cropped from the original image and then composited with the background image. While such a process can yield a composite image, it can present several problems. For example, both before and after compositing, it is often necessary to perform tone or style matching between the object image and the background image. Traditionally, the mean and standard deviation of the object image can be calculated and then replaced with the mean and standard deviation of the background image. However, since object images typically have a relatively limited number of pixel samples (e.g., in...), this approach becomes problematic. Figure 1 In such scenarios, the image of a speed bump may only have pixels of two colors (e.g., in practical applications, speed bumps typically only have black and yellow), while the background image has a relatively richer pixel sample (e.g., in...). Figure 1In a scene where trees may contain green and brown, and the sky may be blue, etc., this substitution method can easily lead to an imbalance in the color distribution of the synthesized image.

[0031] In view of this, Figure 2 During the process, style transfer is performed on the entire original image. This means that pixel samples of the entire original image are considered in terms of tone or style matching, not just pixel samples of the object image. As a result, the style of the object image cropped from the intermediate image is more in line with the background image, thereby avoiding the aforementioned color distribution imbalance problem and ensuring that the synthesized image is stylistically harmonious.

[0032] Alternatively, Figure 2 The process may also include stage 260. In stage 260, the target position 207 of the background image 201 to which the target object is to be placed can be determined based on the semantic information 205 of the background image 201 and the category 206 of the target object.

[0033] Semantic information 205 can indicate the semantics of each pixel in the background image 201. For example, semantic information 205 can include a semantic map of the background image 201. The semantic map can be obtained through a semantic segmentation model.

[0034] In stage 260, by combining the semantic information 205 of the background image 201 with the category 206 of the target object, the semantically suitable location for placing the target object in the background image 201 can be effectively determined. This is crucial for improving the semantic plausibility of the synthesized image. As a negative example, in Figure 1 In the synthetic image 130, the speed bump 112 is placed in a semantically unreasonable position.

[0035] In this case, at stage 280, the object image 204 can be composited with the background image 201 based on the target location 207 to obtain a composite image 208. In the composite image 208, the target object can be located at the target location 207.

[0036] In some embodiments, the object image 204 can be pasted onto the target position 207 of the background image 201 to obtain a composite image 208.

[0037] As can be seen, since the position of the target object in the synthesized image is determined based on the semantic information of the background image and the category of the target object, it can be ensured that the synthesized image is semantically reasonable and feasible.

[0038] It can be seen that, through Figure 2This process can effectively improve the quality of synthesized images, thus making them more suitable for subsequent applications. For example, such synthesized images can efficiently expand the training image set of machine learning models, thereby greatly improving the training performance of machine learning models.

[0039] In some embodiments, synthetic images can be used as supplementary training images for an object detection model, where the category of the target object is unknown to the object detection model. In this case, the object detection model can be trained to recognize the target object of that category based on such synthetic images. Therefore, the synthetic images obtained through the embodiments of this disclosure can efficiently expand the training image set of the object detection model, thereby improving the training performance of the object detection model.

[0040] In one implementation, the object detection model can be a few-shot object detection (FSOD) model. While FSOD aims to train the model to recognize new object categories using a small number of training images, the limited number of training images still restricts the training performance of the FSOD model. Therefore, the synthetic images obtained through the embodiments of this disclosure can efficiently expand the training image set of the FSOD model, thereby improving its training performance.

[0041] The original image can be obtained in various ways, as long as it contains the target object.

[0042] In some embodiments, the original image may be taken by a camera, selected from an existing image set, etc.

[0043] In some embodiments, the original image can be generated by a model. For example, a generative model can be used to generate the original image. In some implementations, image cues can be input into the generative model, which can then output the original image. The image cues can describe the target object and its environment. As mentioned earlier, style matching using only an image of the target object may present some problems. To avoid such problems, the environment of the target object must also be considered. Therefore, when using a generative model to generate the original image, descriptions of the target object and its environment can be input into the generative model, enabling it to output an original image with rich pixels, thereby improving the subsequent style transfer effect.

[0044] Figure 3 An exemplary process for generating an original image according to an embodiment of this disclosure is shown.

[0045] exist Figure 3In the example, image cue 302 can be input into generative model 350. Generative model 350 can be any applicable model capable of generating images. For example, generative model 350 can include a diffusion model.

[0046] Image cue 302 can include two parts. One part can describe the target object, such as its category, color, texture, and other features. The other part can describe the environment in which the target object exists, such as the weather, time, and surrounding physical conditions. For example, for some generative models, prefix can indicate the beginning of the cue. In this case, image cue 302 can have the following format: Prefix target object + environment. As an example, image cue 302 could be: Prefix long speed bump across two lanes in gloomy daytime weather.

[0047] Based on the image cue 302, the generative model 350 can generate the original image 310. The original image 310 may include the target object 312 and the environment 314 in which the target object 312 is located.

[0048] As can be seen, the original image generated in this way not only contains the target object, but also the environment in which the target object is located. This rich pixel information can effectively improve the style transfer effect of the original image, especially the style transfer effect of the object image, thereby improving the style coordination of the synthesized image.

[0049] In some embodiments, a style transfer model can be used during the style transfer stage. For example, the original image and the background image can be input into the style transfer model, which can then output an intermediate image.

[0050] Figure 4 An exemplary process for style transfer according to embodiments of this disclosure is illustrated.

[0051] exist Figure 4 In the example, the original image 402 and the background image 404 can be input into the style transfer model 450. The style transfer model 450 can be any applicable model capable of performing style transfer, such as a convolutional neural network. The style transfer model 450 can convert the style of the original image 402 to the style of the background image 404, while preserving the content of the original image 402. Figure 4 As shown, the style transfer model 450 can output an intermediate image 406, which has the content of the original image 402 but the style of the background image 404.

[0052] After style transfer, the object image can be cropped from the intermediate image. (Still combined) Figure 4 Let me give an example. Figure 5 An exemplary process for cropping an object image according to an embodiment of this disclosure is shown. Figure 5 In the example, assume the target object is a speed bump. The semantic information of the original image 402 can be known, and this semantic information can include the semantic label of the speed bump. The intermediate image 406 contains the content of the original image 402, therefore the semantic information of the intermediate image 406 is consistent with the semantic information of the original image 402. Thus, based on the semantic label of the speed bump, an object image 504 containing the speed bump can be cropped from the intermediate image 406. For ease of understanding, in Figure 5 Image 502 is also shown. In image 502, pixels representing speed bumps in the intermediate image 406 are highlighted; these pixels are determined based on the semantic tags of the speed bumps. It is evident that, based on the semantic tags of the speed bumps, the object image 504 can be easily cropped from the intermediate image 406.

[0053] It should be understood that Figure 5 This is merely an example. In a concrete implementation, image 502 may not actually be formed. For instance, pixels representing the speed bump can be directly extracted from the intermediate image 406 based on its semantic tags, thus forming object image 504. Alternatively, a bounding box or boundary polygon surrounding the speed bump can be formed in the intermediate image 406 based on its semantic tags, and then object image 504 can be obtained based on that bounding box or boundary polygon. Furthermore, Figure 5 The black background in object image 504 is only for highlighting the speed bump. In actual implementation, object image 504 may not contain such a black background.

[0054] As previously stated, to ensure the semantic plausibility of the synthesized image, the target location for placing the target object in the background image can be determined based on the semantic information of the background image and the category of the target object. In some embodiments, the semantic information of the background image may include a semantic map of the background image. The semantic map may include semantic labels for each pixel in the background image. For example, the size of the semantic map may be the same as the size of the background image, and it may indicate pixels in the background image that belong to the same category (e.g., pixels of the same category may have the same color), and may also display their corresponding semantic labels.

[0055] Semantic maps can be obtained through various applicable methods. For example, a background image can be input into a semantic segmentation model, which can then output the corresponding semantic map. Semantic segmentation models can be various applicable models, such as convolutional neural networks.

[0056] Figure 6 An exemplary process for obtaining a semantic graph according to an embodiment of this disclosure is shown.

[0057] exist Figure 6 In the example, the background image 602 can be input into the semantic segmentation model 650, and the semantic style model 650 can output a semantic map 604. The semantic map 604 can include the semantic labels of each pixel in the background image 602, such as sky, tree, lane, shoulder, etc.

[0058] Continuing from the previous description, the target location can be determined based on the semantic information of the background image and the category of the target object. The background image can include multiple semantic regions obtained based on semantic information, and correspondingly, the target location can include the target semantic region within these multiple semantic regions.

[0059] In some embodiments, a target semantic region conforming to preset semantic rules can be determined based on the category of the target object. The preset semantic rules can define the placement relationship between object categories and image semantic regions. For example, preset semantic rules can define that a vehicle is placed on a lane, an animal on grass, a speed bump on a lane, and so on. In this way, it can be determined which semantic region of the background image the target object can be placed in conforms to the preset semantic rules, and the semantic region conforming to the preset semantic rules can be used as the target semantic region where the target object can be placed.

[0060] In some embodiments, a model can be used to determine the target semantic region. For example, location cues can be input into a large language model. The large language model can output the target semantic region. The location cues can describe multiple semantic regions of the background image and the category of the target object.

[0061] Figure 7 An exemplary process for determining a target semantic region using a large language model, according to embodiments of this disclosure, is illustrated.

[0062] exist Figure 7 In the example, a location cue word 704 can be constructed based on the target object's category 702 and the semantic map 604 of the background image. The location cue word 704 can describe the target object's category 702 and multiple semantic regions of the background image. For example, multiple semantic regions can be divided based on semantic labels in the semantic map 604, such as sky regions, tree regions, lane regions, shoulder regions, etc. In some implementations, for some large language models, the prefix indicates the start of the cue word. In this case, the location cue word 704 can have the following format: Which semantic region is the most suitable for placing the target object in? For example, the location cue word 704 could be: Which place is more suitable for placing [speed bump] among [sky], [shoulder], [tree], and [lane]?

[0063] The location cue word 704 can be input into the large language model 750. The large language model 750 can output region information 706. Region information 706 can indicate the target semantic region. For example, in the above example where the target object is a speed bump, region information 706 can indicate the lane.

[0064] As can be seen from the process described above in conjunction with the various figures, the embodiments of this disclosure can ensure that the synthesized image is not only stylistically consistent but also semantically reasonable, thus greatly improving the quality of the synthesized image and ensuring that the synthesized image can be effectively applied in other scenarios.

[0065] Figure 8 A schematic flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0066] At step 802, style transfer can be performed on the original image based on the background image to obtain an intermediate image. The intermediate image contains the same content as the original image and has the same style as the background image.

[0067] At step 804, the object image can be cropped from the intermediate image. The object image contains the target object.

[0068] At step 806, the object image and the background image can be combined to obtain a composite image.

[0069] In some embodiments, at step 802, the original image and the background image can be input into the style transfer model to obtain an intermediate image output by the style transfer model.

[0070] In some embodiments, Figure 8 The method may also include: determining the target location in the background image where the target object should be placed, based on the semantic information of the background image and the category of the target object. In this case, the target object can be located at the target location in the synthesized image.

[0071] In some embodiments, the background image may include multiple semantic regions obtained based on semantic information, and the target location may include the target semantic region among the multiple semantic regions.

[0072] Accordingly, in some implementations, the target semantic region that conforms to preset semantic rules can be determined based on the category of the target object. The preset semantic rules can be used to define the placement relationship between object categories and image semantic regions.

[0073] In other implementations, location cues can be input into a large language model to obtain region information output by the model. Location cues can be used to describe multiple semantic regions and the category of the target object. Region information can indicate the target semantic region.

[0074] In some embodiments, semantic information includes a semantic graph. A background image can be input into a semantic segmentation model to obtain a semantic graph output by the semantic segmentation model.

[0075] In some embodiments, prior to step 802, a generative model can be used to generate the original image. For example, image cues can be input into the generative model to obtain the original image output by the generative model. The image cues can be used to describe the target object and its environment.

[0076] In some embodiments, the generative model may include a diffusion model.

[0077] In some embodiments, synthetic images can be used as supplementary training images for an object detection model, where the category of the target object is unknown to the object detection model.

[0078] Figure 9 A schematic block diagram of an apparatus according to an embodiment of the present disclosure is shown.

[0079] like Figure 9 As shown, the device 900 may include a style transfer unit 902, a cropping unit 904, and a compositing unit 906.

[0080] Style transfer unit 902 can perform style transfer on the original image based on the background image to obtain an intermediate image. The intermediate image contains the same content as the original image and has the same style as the background image. Cropping unit 904 can crop an object image from the intermediate image. The object image contains the target object. Compositing unit 906 can composite the object image with the background image to obtain a composite image.

[0081] Each unit of the device 900 can execute the specific processes described above with respect to the method embodiments. Therefore, for the sake of brevity, the specific operations and functions of each unit of the device 900 will not be repeated here.

[0082] Figure 10 A schematic block diagram of an apparatus according to an embodiment of the present disclosure is shown.

[0083] like Figure 10 As shown, device 1000 may include processor 1002, memory 1004, input interface 1006, and output interface 1008, and these modules can be coupled together via bus 1010. However, it should be understood that... Figure 10 This is merely an example and is not intended to limit the scope of this disclosure. For example, in different application scenarios, device 1000 may include more or fewer modules, which is not limited herein.

[0084] The memory 1004 can be used to store various data, executable instructions, or code related to the functions of the device 1000. For example, the memory 1004 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), registers, hard disks, etc.

[0085] Processor 1002 can execute executable instructions or code stored in memory 1004 to implement various functions or operations of device 1000, such as the processes described in the figures above. Processor 1002 may include various suitable processors, such as general-purpose processors (e.g., central processing unit (CPU)) and special-purpose processors (e.g., digital signal processors, application-specific integrated circuits).

[0086] Input interface 1006 can receive various forms of data, such as the original image and background image mentioned above. Output interface 1008 can output various forms of data, such as the composite image mentioned above.

[0087] Embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium may store executable instructions, which, when executed, perform the processes described above in conjunction with the various figures.

[0088] For example, computer-readable storage media may include, but are not limited to, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), hard disk, flash memory, etc.

[0089] Embodiments of this disclosure also provide a computer program product. The computer program product may include a computer program. When executed, the computer program can perform the processes described above in conjunction with the various figures.

[0090] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] Not all steps and units in the above process and system structure diagrams are necessary; some steps or units can be omitted according to actual needs. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical entity, some units may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0092] The optional embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present disclosure are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present disclosure, various modifications can be made to the technical solutions of the embodiments of the present disclosure, and these modifications all fall within the protection scope of the embodiments of the present disclosure.

Claims

1. An image processing method, comprising: Style transfer is performed on the original image based on the background image to obtain an intermediate image, wherein the intermediate image contains the same content as the original image and has the same style as the background image; An object image is cropped from the intermediate image, wherein the object image contains the target object; The object image and the background image are combined to obtain a composite image.

2. The method according to claim 1, wherein, Perform style transfer on the original image based on the background image to obtain an intermediate image, including: The original image and the background image are input into the style transfer model to obtain the intermediate image output by the style transfer model.

3. The method according to claim 1, further comprising: Based on the semantic information of the background image and the category of the target object, determine the target location in the background image where the target object should be placed; In the synthesized image, the target object is located at the target location.

4. The method according to claim 3, wherein, The background image includes multiple semantic regions obtained based on the semantic information, and the target location includes the target semantic region among the multiple semantic regions.

5. The method according to claim 4, wherein, Determining the target location includes: Based on the category of the target object, the target semantic region that conforms to the preset semantic rules is determined, wherein the preset semantic rules are used to define the placement relationship between the object category and the image semantic region.

6. The method according to claim 4, wherein, Determining the target location includes: The location cue words are input into the large language model to obtain the region information output by the large language model. The location cue words are used to describe the multiple semantic regions and the category of the target object, and the region information is used to indicate the target semantic region.

7. The method according to claim 3, wherein, The semantic information includes a semantic graph; The method further includes: The background image is input into the semantic segmentation model to obtain the semantic map output by the semantic segmentation model.

8. The method according to claim 1, further comprising: The original image is generated using a generative model.

9. The method according to claim 8, wherein, The original image is generated using a generative model, including: The image prompts are input into the generative model to obtain the original image output by the generative model, wherein the image prompts are used to describe the target object and the environment in which the target object is located.

10. The method according to claim 8, wherein, The generative model includes a diffusion model.

11. The method according to claim 1, wherein, The synthesized image is used as a supplementary training image for the object detection model, and the category of the target object is unknown to the object detection model.

12. An image processing apparatus, comprising: At least one processor; A memory coupled to the at least one processor stores executable instructions that, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 11.

13. A computer program product storing a computer program that, when executed, implements the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing executable code that, when executed, implements the method according to any one of claims 1 to 11.

Images