Image synthesis, display method and device

Abstract

This application provides an image compositing and display method and apparatus. The method includes acquiring a first image to be composited, a first mask, a second image to be composited, and a second mask. The method determines whether a first target object and a second target object have overlapping pixels using the first and second masks. If so, the first and second images to be composited are combined based on the first depth information of the overlapping pixels in the first image to be composited and the second depth information of the overlapping pixels in the second image to be composited, to determine a first fused image. This approach, by using the depth information of each pixel in the first and second images to be composited, can promptly and accurately determine the occlusion relationship between the two images when they overlap, thereby making the image compositing more logical and improving the quality of the composited image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image synthesis and display method and apparatus. Background Technology

[0002] Image synthesis is an important branch of artificial intelligence image algorithms. Its main function is to combine multiple images together to form a new image, thus meeting users' personalized needs for synthesizing images from the same or different scenes. Currently, image synthesis is widely used in short video apps, live video streaming, and other applications.

[0003] Existing image synthesis methods primarily involve segmenting two images separately to obtain their respective masks. Then, based on these masks, target sub-images are extracted from their corresponding images. Finally, the two extracted target sub-images are synthesized. However, if the target sub-images overlap, occlusion and misalignment issues can sometimes occur.

[0004] In summary, there is an urgent need for an image synthesis and display method to solve the problem in existing technologies where multiple images to be synthesized overlap and it is impossible to determine which image occludes which image or which image is in front or behind. Summary of the Invention

[0005] The exemplary embodiments of this application provide an image synthesis and display method and apparatus to solve the problem in the prior art that when multiple images to be synthesized overlap, it is impossible to determine which one occludes which one or which one is in front and which one is behind.

[0006] In a first aspect, an exemplary embodiment of this application provides an image synthesis method, comprising:

[0007] A first image to be composited, a first mask image, a second image to be composited, and a second mask image are obtained; the first mask image is determined after extracting a first target object from the first image to be composited; the second mask image is determined after extracting a second target object from the second image to be composited; the first image to be composited and the second image to be composited include depth information of each pixel.

[0008] The first mask and the second mask are used to determine whether there are overlapping pixels between the first target object and the second target object;

[0009] If so, the first image to be synthesized and the second image to be synthesized are synthesized based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, to determine the first fused image.

[0010] In the above technical solution, before the first image to be synthesized and the second image to be synthesized are synthesized, it is determined whether there are overlapping pixels between the first target object and the second target object through the first mask and the second mask. When it is determined that there are overlapping pixels between the first target object and the second target object, the first image to be synthesized and the second image to be synthesized are synthesized based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, so as to determine the first fused image. Existing technologies directly synthesize multiple images without considering their depth information. This makes it impossible to determine the occlusion relationship when multiple images overlap, leading to occlusion logic errors during image synthesis and resulting in low image synthesis quality. Therefore, this solution uses the depth information of each pixel in both the first and second images to accurately and promptly determine their occlusion relationship when they overlap. This makes image synthesis more logical and improves its quality, thus solving the problem of not being able to determine which image occludes which or which image precedes the other when multiple images overlap. Furthermore, this solution directly uses a mask image to determine whether there are overlapping pixels between target objects. Compared to directly using the images to be synthesized, this reduces the computational load required for determining overlapping pixels, thereby improving the efficiency of image synthesis.

[0011] In some exemplary embodiments, the step of combining the first image to be combined and the second image to be combined based on the first depth information of the overlapping pixels in the first image to be combined and the second depth information of the overlapping pixels in the second image to be combined to determine the first fused image includes:

[0012] The first image to be synthesized is segmented using the first mask to obtain the first target object, and the second image to be synthesized is segmented using the second mask to obtain the second target object;

[0013] The first target object and the second target object are combined to generate a second fused image;

[0014] Based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, the second fused image is adjusted to determine the first fused image.

[0015] In the above technical solution, since the target object synthesized into the background image first is usually covered by the target object synthesized into the background image later, this solution synthesizes the first target object and the second target object without considering which target object is synthesized into the background image first. This saves the computational resources consumed by determining which target object is synthesized into the background image first, and improves the efficiency of data processing. In addition, based on the size relationship between the first depth information and the second depth information, the occlusion relationship between the first image to be synthesized and the second image to be synthesized can be accurately determined, so as to adjust the second fused image in a timely manner, thereby obtaining a reasonable and high-quality fused image.

[0016] In some exemplary embodiments, adjusting the second fused image based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, to determine the first fused image, includes:

[0017] When the first depth information is less than the second depth information, the pixel value of the overlapping pixel is set to the first pixel value of the overlapping pixel in the first composite image; or

[0018] When the first depth information is not less than the second depth information, the pixel value of the overlapping pixel is set to the second pixel value of the overlapping pixel in the second composite image.

[0019] In the above technical solution, the current pixel value of the overlapping pixel in the second fused image is not considered in which image to be synthesized the overlapping pixel is located. Instead, the first depth information and the second depth information are directly compared to quickly determine the size relationship between the first depth information and the second depth information. Based on the size relationship between the first depth information and the second depth information, it is possible to accurately determine which image to be synthesized the overlapping pixel needs to be adjusted to.

[0020] In some exemplary embodiments, adjusting the second fused image based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, to determine the first fused image, includes:

[0021] Determine the pixel values ​​of the overlapping pixels in the second fused image;

[0022] If the pixel value of the overlapping pixel is the first pixel value of the overlapping pixel in the first composite image, and the first depth information is not less than the second depth information, then the pixel value of the overlapping pixel is set to the second pixel value of the overlapping pixel in the second composite image.

[0023] If the pixel value of the overlapping pixel is the second pixel value of the overlapping pixel in the second composite image, and the first depth information is less than the second depth information, then the pixel value of the overlapping pixel is set to the first pixel value of the overlapping pixel in the first composite image.

[0024] In the above technical solution, the pixel value of the overlapping pixel in the second fused image is first determined to be the pixel value in which the overlapping pixel is located in the composite image. That is, the current pixel value of the overlapping pixel in the second fused image is first considered in which composite image it belongs. Then, the size relationship between the first depth information and the second depth information is determined, and the pixel value of the overlapping pixel is adjusted based on this relationship. In this way, different processing mechanisms can be set for different determination results to ensure that the pixel value of the overlapping pixel can be adjusted in a timely manner and to save the resources consumed in adjusting the pixel value of the overlapping pixel.

[0025] In some exemplary embodiments, the step of compositing the first target object and the second target object to generate a second fused image includes:

[0026] The first target object and the second target object are combined into the same background image to generate the second fused image.

[0027] In the above technical solution, the first target object and the second target object are composited into the same new background image without regard to which target object is composited into the background image first. This saves the computing resources consumed by determining which one to composite into the background image first.

[0028] In some exemplary embodiments, the first target object in the first mask image and the second target object in the second mask image have set pixel values;

[0029] Determining whether the first target object and the second target object have overlapping pixels by using the first mask and the second mask includes:

[0030] By determining whether the positions of pixels with the set pixel values ​​in the first mask and the second mask are the same, it is determined whether there are overlapping pixels between the first target object and the second target object.

[0031] In the above technical solution, by determining whether there are overlapping pixels between the first target object and the second target object based on whether the positions of pixels with set pixel values ​​in the first mask and the second mask are the same, the overlapping part between the first target object and the second target object can be determined in a timely and accurate manner, so as to provide support for subsequent adjustment of the overlapping part based on depth information.

[0032] In some exemplary embodiments, obtaining the first image to be composited, the first mask image, the second image to be composited, and the second mask image includes:

[0033] The first image to be synthesized and the first mask image are provided by the first device after real-time acquisition;

[0034] The second image to be synthesized and the second mask image are provided by the second device after real-time acquisition.

[0035] In the above technical solution, since the corresponding images to be synthesized are acquired and segmented by multiple devices to obtain the corresponding mask images, the processing pressure caused by processing multiple images to be synthesized in one device can be reduced, and the computing resources consumed by processing multiple images to be synthesized in one device can be saved, thereby improving the efficiency of image synthesis.

[0036] Secondly, an exemplary embodiment of this application provides an image display method, comprising:

[0037] In response to a user compositing instruction, in the fused image, the image corresponding to the first target object occludes the overlapping portion of the image corresponding to the second target object and the image corresponding to the first target object, wherein the distance of the first target object from the lens is less than the distance of the second target object from the lens.

[0038] Thirdly, an exemplary embodiment of this application provides an image synthesis apparatus, comprising:

[0039] The acquisition unit is used to acquire a first image to be synthesized, a first mask image, a second image to be synthesized, and a second mask image; the first mask image is determined after extracting a first target object from the first image to be synthesized; the second mask image is determined after extracting a second target object from the second image to be synthesized; the first image to be synthesized and the second image to be synthesized include depth information of each pixel;

[0040] The processing unit is configured to determine whether there are overlapping pixels between the first target object and the second target object through the first mask and the second mask; if so, based on the first depth information of the overlapping pixels in the first image to be synthesized and the second image to be synthesized, the first image to be synthesized and the second image to be synthesized are synthesized to determine the first fused image.

[0041] In some exemplary embodiments, the processing unit is specifically used for:

[0042] The first image to be synthesized is segmented using the first mask to obtain the first target object, and the second image to be synthesized is segmented using the second mask to obtain the second target object;

[0043] The first target object and the second target object are combined to generate a second fused image;

[0044] Based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, the second fused image is adjusted to determine the first fused image.

[0045] In some exemplary embodiments, the processing unit is specifically used for:

[0046] When the first depth information is less than the second depth information, the pixel value of the overlapping pixel is set to the first pixel value of the overlapping pixel in the first composite image; or

[0047] When the first depth information is not less than the second depth information, the pixel value of the overlapping pixel is set to the second pixel value of the overlapping pixel in the second composite image.

[0048] In some exemplary embodiments, the processing unit is specifically used for:

[0049] Determine the pixel values ​​of the overlapping pixels in the second fused image;

[0050] If the pixel value of the overlapping pixel is the first pixel value of the overlapping pixel in the first composite image, and the first depth information is not less than the second depth information, then the pixel value of the overlapping pixel is set to the second pixel value of the overlapping pixel in the second composite image.

[0051] If the pixel value of the overlapping pixel is the second pixel value of the overlapping pixel in the second composite image, and the first depth information is less than the second depth information, then the pixel value of the overlapping pixel is set to the first pixel value of the overlapping pixel in the first composite image.

[0052] In some exemplary embodiments, the processing unit is specifically used for:

[0053] The first target object and the second target object are combined into the same background image to generate the second fused image.

[0054] In some exemplary embodiments, the first target object in the first mask image and the second target object in the second mask image have set pixel values;

[0055] The processing unit is specifically used for:

[0056] By determining whether the positions of pixels with the set pixel values ​​in the first mask and the second mask are the same, it is determined whether there are overlapping pixels between the first target object and the second target object.

[0057] In some exemplary embodiments, the acquisition unit is specifically used for:

[0058] The first image to be synthesized and the first mask image are provided by the first device after real-time acquisition;

[0059] The second image to be synthesized and the second mask image are provided by the second device after real-time acquisition.

[0060] Fourthly, an exemplary embodiment of this application provides an image display device, comprising:

[0061] In response to a user compositing instruction, in the fused image, the image corresponding to the first target object occludes the overlapping portion of the image corresponding to the second target object and the image corresponding to the first target object, wherein the distance of the first target object from the lens is less than the distance of the second target object from the lens.

[0062] Fifthly, embodiments of this application provide a computing device, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor performs any of the image synthesis methods described in the first aspect or the image display methods described in the second aspect.

[0063] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computing device, which, when run on the computing device, causes the computing device to perform any of the image synthesis methods described in the first aspect above.

[0064] In a seventh aspect, embodiments of this application provide a display device, including at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the image display method described in the second aspect. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 A schematic diagram of an image synthesis system architecture provided for some embodiments of this application;

[0067] Figure 2 A schematic flowchart of an image synthesis method provided for some embodiments of this application;

[0068] Figure 3 This application provides a schematic diagram of a data transmission structure for some embodiments;

[0069] Figure 4 A schematic diagram illustrating an image synthesis process based on depth information, provided for some embodiments of this application;

[0070] Figure 5a This is a schematic diagram showing that a first mask and a second mask overlap, provided for some embodiments of this application;

[0071] Figure 5b This is a schematic diagram illustrating a first mask and a second mask that do not overlap, provided for some embodiments of this application.

[0072] Figure 6a An initial group photo of user A and user B is provided for some embodiments of this application;

[0073] Figure 6b A schematic diagram of a final group photo of user A and user B provided for some embodiments of this application;

[0074] Figure 7 This is a schematic diagram of the structure of an image synthesis apparatus provided in some embodiments of this application;

[0075] Figure 8 This is a schematic diagram of the structure of a computing device provided for some embodiments of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] To facilitate understanding of the embodiments of this application, firstly... Figure 1 The system structure shown is used as an example to illustrate the image synthesis system architecture applicable to embodiments of this application. This image synthesis system architecture can be applied to portrait synthesis, or to object synthesis, etc. The portrait or object can be a person or object in a video, or it can be a person or object in an image; this application embodiment does not limit this. Figure 1 As shown, the image synthesis system architecture may include a data processing device 100, a camera 200, and a display screen 300.

[0078] The data processing device 100 has information processing and forwarding functions. The data processing device 100 can be a terminal device with data processing capabilities (such as smartphones, tablets, laptops, or in-vehicle terminals). Alternatively, it can be a single server or a server cluster. For example, the data processing device 100 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, cloud communication, domain name services, security services, and big data and artificial intelligence platforms. In this embodiment, a home brain device is used as the data processing device to process the information data in this application. This home brain device is a home control center integrating data storage, data analysis, and device control functions, possessing powerful data storage and processing capabilities.

[0079] Camera 200 is a camera capable of capturing both RGB and depth information, such as an RGBD camera (i.e., a depth camera). Besides measuring the RGB information of each pixel, it also measures the depth information of each pixel. RGB represents the three color channels: Red, Green, and Blue. For each pixel in an image, its RGB information (i.e., its pixel value) needs to be recorded. Depth information is used to characterize the distance between a pixel and the camera lens in the shooting scene. For example, in the depth map of an RGBD camera, the distance between each pixel in the image and the camera lens is recorded; this distance is usually measured in millimeters.

[0080] The display screen 300 includes a display screen assembly for presenting images and a driving assembly for driving the image display. The displayed image content may be image content processed by the data processing device 100, or it may display various image content received from a network server via a network communication protocol.

[0081] Furthermore, the camera 200 and the display screen 300 are respectively connected to the data processing device 100. For example, the data processing device 100, the camera 200, and the display screen 300 can communicate via one or more networks. This network can be a wired network or a wireless network. For example, the wireless network could be a Wi-Fi (Wi-Fi) network, a mobile cellular network, or other possible networks; this embodiment does not limit the specific network used. It should be noted that the camera 200 can be configured independently or integrated into the display screen 300; this embodiment does not limit the specific camera used.

[0082] For example, taking the data processing device 100 as a home brain device, the application scenario of this application embodiment will be introduced. For instance, a user discovers that they do not have a group photo with others (or several people), or, in pursuit of personalization, the user discovers that they do not have a group photo with others (or several people) in a certain scene, or several users discover that they do not have a group photo or, in pursuit of personalization, they discover that they do not have a group photo in a certain scene. Therefore, in order to achieve multi-person image synthesis (i.e., multi-person virtual group photo), an image synthesis system is used to achieve multi-person image synthesis. This application embodiment uses two users as an example to illustrate the implementation process of multi-person image synthesis, namely user 1 and user 2. Both user 1 and user 2 have an image synthesis system. User 1 can use the RGBD camera in their image synthesis system to capture user 1's image and upload the captured user 1's image to the home brain device. The home brain device uses image segmentation algorithms (such as semantic segmentation algorithms, instance segmentation algorithms, or panoramic segmentation algorithms, etc.) to segment user 1's image and obtain the corresponding mask. Figure 1 (That is, the mask image, with the human face area in white and the non-human face area in black). Then, the image of User 1 and the corresponding mask are... Figure 1 Send to User 2's home brain device.

[0083] Similarly, User 2 will also use the RGBD camera in their own image synthesis system to capture images of User 2 in real time and upload them to the Home Brain device. The Home Brain device will then use image segmentation algorithms (such as semantic segmentation, instance segmentation, or panorama segmentation) to segment User 2's images and obtain the corresponding masks. Figure 2 Then take User 2's image and its corresponding mask. Figure 2 Send to User 1's home brain device.

[0084] After receiving the image and corresponding mask B sent by User 2, User 1's home brain device will first determine the mask. Figure 1 With mask Figure 2 If overlapping pixels exist, the image from User 1 and User 2 is composited based on the depth values ​​in the depth information of the overlapping pixels. If no overlapping pixels exist, User 1 and User 2's images are directly composited to obtain a photo of User 1 and User 2 together, which is then displayed on User 1's screen. Similarly, User 2's home brain device receives the image and corresponding mask sent by User 1. Figure 1 Then, the mask will be determined first. Figure 2 With mask Figure 1 If there are overlapping pixels, the image of user 1 and the image of user 2 are composited based on the depth value in the depth information of the overlapping pixels. If there are no overlapping pixels, the image of user 1 and the image of user 2 are composited directly to obtain a photo of user 1 and user 2, and the photo of user 1 and user 2 is displayed on user 2's screen.

[0085] It should be noted that the above Figure 1 The structure shown is merely an example, and the embodiments of this application do not limit it.

[0086] Based on the above description Figure 2 An exemplary illustration shows the flow of an image compositing method provided in an embodiment of this application, which can be executed by an image compositing apparatus.

[0087] like Figure 2 As shown, the process specifically includes:

[0088] Step 201: Obtain the first image to be synthesized, the first mask image, the second image to be synthesized, and the second mask image.

[0089] Step 202: Determine whether there are overlapping pixels between the first target object and the second target object using the first mask and the second mask.

[0090] Step 203: If it is determined that the first target object and the second target object have overlapping pixels, then based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, the first image to be synthesized and the second image to be synthesized are synthesized to determine the first fused image.

[0091] In step 201 above, the first mask is determined after extracting the first target object from the first image to be synthesized; the second mask is determined after extracting the second target object from the second image to be synthesized; both the first and second images to be synthesized include depth information for each pixel; the depth information is used to characterize the distance between the pixel and the camera lens in the shooting scene, and this depth information includes a depth value. For example, when shooting an instance to be synthesized (such as a human body, an object, etc.), the distance between each point in the instance to be synthesized and the camera lens represents the depth information of that point, which is the depth information of the pixel corresponding to that point in the captured image. Specifically, image synthesis can be completed through interaction between the first and second devices. The first image to be synthesized can be acquired in real time by the first device, and the first mask can be extracted by image segmentation of the first image to be synthesized. Similarly, the second image to be synthesized can be acquired in real time by the second device, and the second mask can be extracted by image segmentation of the second image to be synthesized. For example, after segmenting the first mask image, the first device can send the first image to be synthesized and the first mask image to the second device. Similarly, after segmenting the second mask image, the second device can send the second image to be synthesized and the second mask image to the first device. Thus, both the first and second devices will store the first image to be synthesized, the first mask image, the second image to be synthesized, and the second mask image. Then, the first and second devices can perform image synthesis separately. In this way, by having multiple devices separately acquire and segment the corresponding images to be synthesized to obtain the corresponding mask images, the processing pressure caused by centrally processing multiple images to be synthesized on a single device can be reduced, and the computational resources consumed by centrally processing multiple images to be synthesized on a single device can be saved, thereby improving the efficiency of image synthesis.

[0092] For example, the first and second devices may include a camera, a data processing device, and a display screen. The camera (e.g., an RGBD camera) can capture images or video feeds of the user in real time and store them so that the data processing device can access them promptly for image synthesis. For instance, taking the interaction between user 1 and user 2 to complete image synthesis as an example, user 1's data processing device can perform image segmentation on the captured images to obtain a mask. Figure 1 and the image and mask of user 1 Figure 1 The image is sent to User 2 so that User 2's data processing device can perform image synthesis in a timely manner. Similarly, User 2's data processing device can perform image segmentation on the acquired image to obtain a mask. Figure 2 and the image and mask of user 2. Figure 2 Send it to User 1 so that User 1's data processing device can perform image synthesis in a timely manner.

[0093] In steps 202 and 203 above, since the first target object in the first mask and the second target object in the second mask have set pixel values, it can be determined whether there are overlapping pixels between the first target object and the second target object by checking whether the positions of pixels with set pixel values ​​in the first mask and the second mask are the same. Here, the pixels with set pixel values ​​are white pixels in the mask (white pixels have a value of 255, and black pixels have a value of 0), and the position of the white pixels in the image represents the location of the target object. For example, it can be determined whether there are overlapping pixels between the first and second target objects by checking whether there are white pixels in the same position in the first and second masks. If not, the first image to be composited is segmented using the first mask to obtain the first target object, and the second image to be composited is segmented using the second mask to obtain the second target object. The first and second target objects are then directly composited into the same background image to generate the first fused image.

[0094] If overlapping pixels exist, the coordinates of the overlapping pixels are recorded, and the first image to be synthesized is segmented using a first mask to obtain the first target object. Similarly, the second image to be synthesized is segmented using a second mask to obtain the second target object. The first and second target objects are then synthesized to generate a second fused image. Based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, the second fused image is adjusted to determine the first fused image. Since the first and second target objects overlap during image synthesis, the target object synthesized first in the background image is usually covered by the target object synthesized later in the background image. Utilizing this characteristic, during image synthesis, the first and second target objects can be synthesized into the same background image without any order of priority to generate the second fused object. Therefore, since target objects synthesized into the background image first are usually covered by target objects synthesized into the background image later, this scheme synthesizes the first and second target objects without considering which target object is synthesized into the background image first. This saves the computational resources consumed by determining which target object is synthesized into the background image first, and improves the efficiency of data processing. In addition, based on the relationship between the depth values ​​in the first depth information and the depth values ​​in the second depth information, the occlusion relationship between the first and second images to be synthesized can be accurately determined, so as to adjust the second fused image in a timely manner, thereby obtaining a reasonable and high-quality fused image.

[0095] Furthermore, regarding the adjustment of the second fused image based on depth information, it was determined that the first fused image has two implementation methods. That is:

[0096] The first implementation method is as follows: This method does not consider which image the overlapping pixel in the second fused image belongs to based on its current pixel value. Instead, after determining the relationship between the depth values ​​in the first and second depth information, it determines which image the overlapping pixel needs to be adjusted to. Specifically, if the depth value in the first depth information is less than the depth value in the second depth information, the overlapping pixel's pixel value is set to its first pixel value in the first image to be merged; or, if the depth value in the first depth information is not less than the depth value in the second depth information, the overlapping pixel's pixel value is set to its second pixel value in the second image to be merged. Based on this, after adjusting the pixel values ​​of the overlapping pixels, the first fused image is obtained.

[0097] For example, this implementation does not determine which image the overlapping pixel in the second fused image contains the current pixel value. Instead, it first determines the relationship between the depth values ​​in the first depth information and the depth values ​​in the second depth information, and then adjusts the pixel values ​​of the overlapping pixels based on this relationship. If the depth value in the first depth information is less than the depth value in the second depth information, the pixel value of the overlapping pixel is adjusted to the first pixel value of the overlapping pixel in the first image to be synthesized; if the depth value in the first depth information is not less than the depth value in the second depth information, the pixel value of the overlapping pixel is adjusted to the second pixel value of the overlapping pixel in the second image to be synthesized.

[0098] For example, let's illustrate this implementation by generating a photo of User 1 and User 2 together. Assume that User 1's and User 2's portrait images overlap; then, a mask is used... Figure 1 The image of user 1 is segmented to obtain target object 1, and a mask is used to segment the image. Figure 2 The portrait image of User 2 is segmented to obtain Target Object 2. Target Object 1 and Target Object 2 are then combined to generate a second fused image. Next, the depth value of the overlapping pixel in User 1's portrait image and its depth value in User 2's portrait image are determined, and the pixel value of the overlapping pixel is adjusted according to this relationship. For example, if the depth information of the overlapping pixel in User 1's portrait image is 1.5m, and the depth information of the overlapping pixel in User 2's portrait image is 1.7m, then it can be determined that the depth value of the overlapping pixel in User 2's portrait image is greater than its depth value in User 1's portrait image, and the pixel value of the overlapping pixel is adjusted to match its depth value in User 1's portrait image. If the depth information of the overlapping pixel in User 1's portrait image is 1.7m, and the depth information of the overlapping pixel in User 2's portrait image is 1.5m, then it can be determined that the depth value of the overlapping pixel in User 1's portrait image is greater than the depth value of the overlapping pixel in User 2's portrait image. The pixel value of the overlapping pixel is then adjusted to match the pixel value of the overlapping pixel in User 2's portrait image.

[0099] The second implementation method is as follows: This implementation method first needs to determine which image to be synthesized contains the pixel value of the overlapping pixel in the second fused image. That is, it first considers which image to be synthesized contains the current pixel value of the overlapping pixel in the second fused image. Then, it determines the relationship between the depth values ​​in the first depth information and the depth values ​​in the second depth information, and adjusts the pixel value of the overlapping pixel based on the relationship. That is, first, the pixel value of the overlapping pixel in the second fused image is determined, and then it is determined which image to be synthesized contains the pixel value of the overlapping pixel. If the pixel value of the overlapping pixel is the first pixel value of the overlapping pixel in the first image to be synthesized, and the depth value in the first depth information is not less than the depth value in the second depth information, the pixel value of the overlapping pixel is set to the second pixel value of the overlapping pixel in the second image to be synthesized. If the pixel value of the overlapping pixel is the second pixel value of the overlapping pixel in the second image to be synthesized, and the depth value in the first depth information is less than the depth value in the second depth information, the pixel value of the overlapping pixel is set to the first pixel value of the overlapping pixel in the first image to be synthesized. Based on this, after adjusting the pixel values ​​of the overlapping pixels, the first fused image can be obtained.

[0100] For example, this implementation first determines which composite image the current pixel value of the overlapping pixel in the second fused image belongs to. If the pixel value of the overlapping pixel is the first pixel value of the overlapping pixel in the first composite image, and the first depth information is less than the second depth information, then no adjustment is needed to the pixel value of the overlapping pixel. If the pixel value of the overlapping pixel is the first pixel value of the overlapping pixel in the first composite image, and the first depth information is not less than the second depth information, then the pixel value of the overlapping pixel needs to be adjusted. Alternatively, if the pixel value of the overlapping pixel is the second pixel value of the overlapping pixel in the second composite image, and the first depth information is less than the second depth information, then the pixel value of the overlapping pixel needs to be adjusted. If the pixel value of the overlapping pixel is the second pixel value of the overlapping pixel in the second composite image, and the first depth information is not less than the second depth information, then no adjustment is needed to the pixel value of the overlapping pixel. In this way, different processing mechanisms can be set for different determination results to ensure that the pixel values ​​of overlapping pixels can be adjusted in a timely manner and to save the resources consumed in adjusting the pixel values ​​of overlapping pixels.

[0101] For example, taking the generation of a group photo of User 1 and User 2 as an example to illustrate this implementation method, firstly, the pixel values ​​of overlapping pixels in the second fused images of User 1 and User 2 are determined. It is assumed that the pixel value of this overlapping pixel is the same as the pixel value of the overlapping pixel in User 1's portrait image. Next, the relationship between the depth value of the overlapping pixel in User 1's portrait image and the depth value of the overlapping pixel in User 2's portrait image is determined, and then the pixel value of the overlapping pixel is adjusted according to this relationship. For example, if the depth value of the overlapping pixel in User 1's portrait image is 1.9m, and the depth value of the overlapping pixel in User 2's portrait image is 1.7m, then it can be determined that the depth value of the overlapping pixel in User 1's portrait image is greater than the depth value of the overlapping pixel in User 2's portrait image, and the pixel value of the overlapping pixel is adjusted to match the pixel value of the overlapping pixel in User 2's portrait image. If the depth value of the overlapping pixel in User 1's portrait image is 1.7m, and the depth value of the overlapping pixel in User 2's portrait image is 1.9m, then it can be determined that the depth value of the overlapping pixel in User 2's portrait image is greater than the depth value of the overlapping pixel in User 1's portrait image, and there is no need to adjust the pixel value of the overlapping pixel.

[0102] It should be noted that the above image compositing method can be reflected in the user interface. That is, in one embodiment of image display, the image compositing system responds to the compositing command issued by the user, and then performs compositing processing on the first image to be composited and the second image to be composited. If it is determined that the distance of the first target object from the lens is less than the distance of the second target object from the lens, then a fused image with logical and accurate occlusion relationship is displayed on the screen. In the fused image, the image corresponding to the first target object occludes the overlapping part of the image corresponding to the second target object and the image corresponding to the first target object.

[0103] In view of this, the following description uses User A's image synthesis system as an example to illustrate the implementation process of the image synthesis method in this application. It should be noted that, in order to achieve multi-user image synthesis (i.e., a virtual group photo), multiple users will participate in the image synthesis process, such as User A and User B. Both User A and User B will use their own image synthesis systems to perform image synthesis, and User A and User B will also interact during the image synthesis process.

[0104] Step 1: Each user uses a camera to capture images and performs image segmentation on the captured images to obtain a mask image.

[0105] For any given user, based on the user's shooting command, the user's camera (such as an RGBD camera) captures an image or a video stream from the user. Each frame of the image or video stream includes the RGB information (i.e., the pixel value of each pixel) and depth information for each pixel. The image or video stream is then sent to the user's data processing device (such as a home intelligence device) so that the data processing device can segment the image or video stream to obtain the corresponding mask image.

[0106] For example, user A can use the RGBD camera in the image compositing system to capture a portrait image (i.e., an RGBD image) of user A, and transmit the captured portrait image to their home brain device. The home brain device uses an image segmentation algorithm to segment the portrait image to obtain a first mask image. Alternatively, the RGBD camera in the image compositing system can also use the RGBD camera to capture one video image (e.g., a portrait video of user A) in real time, and transmit the captured video image to the home brain device in the image compositing system. The home brain device uses an image segmentation algorithm to segment each frame of the video image to obtain the corresponding mask image A.

[0107] Similarly, User B can use the RGBD camera in the image synthesis system to capture a portrait image of User B (i.e., an RGBD image), and transmit the captured image to their home brain device. The home brain device then uses an image segmentation algorithm to segment the portrait image and obtain a second mask image. Alternatively, the RGBD camera in the image synthesis system can also capture one video stream of User B in real time (e.g., a portrait video of User A), and transmit this video stream to their home brain device. The home brain device then uses an image segmentation algorithm to segment each frame of the video stream and obtain the corresponding mask image B.

[0108] Step 2: Each user transmits the captured image and its mask to the corresponding user's home brain device.

[0109] For any user, the user can transmit the captured image and the mask of the image to the data processing device (such as a home brain device) of the corresponding user who needs to take a group photo, so that the corresponding user's data processing device can perform image synthesis based on the user's image and the mask of the image.

[0110] refer to Figure 3 , Figure 3 This is a schematic diagram of a data transmission structure provided in an embodiment of this application. Based on... Figure 3Taking user A's data processing device as an example of a home brain device, after user A's home brain device uses an image segmentation algorithm to segment user A's portrait image to obtain a first mask image, it stores the first mask image in a database that also stores user A's portrait image. Then, it sends user A's portrait image and the first mask image to user B's home brain device via the cloud. Alternatively, user A can encrypt user A's portrait image and the first mask image, thus ensuring the security and privacy of user A's portrait image and the first mask image during transmission to prevent them from being stolen by criminals. The encrypted user A's portrait image and the first mask image are then sent to user B's home brain device. For example, user A encrypts user A's portrait image and the first mask image using a private key, and user B's home brain device then decrypts the encrypted user A's portrait image and the first mask image using the public key corresponding to the private key, obtaining the decrypted user A's portrait image and the first mask image.

[0111] Similarly, after user B uses an image segmentation algorithm to segment their portrait image to obtain a second mask, they store the second mask in a database that also stores user B's portrait image. Then, they send user B's portrait image and the second mask to user A's home brain device via the cloud. Alternatively, user B can encrypt their portrait image and the first mask, ensuring the security and privacy of these images and masks during transmission to prevent them from being stolen by malicious actors. The encrypted image and mask are then sent to user A's home brain device. For example, user B can encrypt their image and mask using their private key, and user A's home brain device can then decrypt them using the public key corresponding to that private key, obtaining the decrypted image and mask.

[0112] Step 3: Each user combines their own image with the image of the corresponding user based on the depth information of each pixel in the image to obtain a photo of the two users together.

[0113] For any given user, their data processing device (such as a home intelligence device) stores a background image, their own image and a mask, and the image and mask of the user they want to take a group photo with. The user's data processing device first determines if there are any overlapping pixels between their own mask and the mask of the user they want to take a group photo with. If so, it records the position information of the overlapping pixels and composites their own image and the image of the user they want to take a group photo with into the same background image, forming an initial group photo of the two users. Then, based on the coordinate information of the overlapping pixels, the distance information of the overlapping pixels in their own image, and the distance information of the overlapping pixels in the image of the user they want to take a group photo with, the initial group photo of the two users is adjusted to form the final group photo of the two users. If no overlapping pixels are found, the user's own image and the image of the user they want to take a group photo with are directly composited into the same background image to form the group photo of the two users.

[0114] refer to Figure 4 , Figure 4 This is a schematic diagram illustrating an image synthesis process based on depth information, provided as an embodiment of this application. Figure 4 Taking user A's data processing device as a home brain device as an example, user A's home brain device first determines whether user A's portrait image overlaps with user B's portrait image based on user A's first mask and user B's second mask. That is, since user A's portrait image and user B's portrait image are the same size, and the white pixels in the mask (white pixels have a value of 255, black pixels have a value of 0) represent the location of the portrait in the image, it is only necessary to determine whether the first mask and the second mask have white pixels at the same position. If so, it means that the first mask and the second mask overlap at that pixel, that is, user A's portrait image overlaps with user B's portrait image at that pixel, and the position coordinates of that pixel are recorded. This allows the determination of the position coordinates of each overlapping pixel. For example... Figure 5a The diagram shown illustrates an embodiment of this application where a first mask and a second mask overlap. If they do not overlap, it indicates that the first mask and the second mask do not overlap, meaning that user A's portrait image and user B's portrait image do not overlap. For example... Figure 5b The diagram shown is a schematic representation of an embodiment of this application where the first mask and the second mask do not overlap.

[0115] After User A's home brain device determines whether User A's portrait image overlaps with User B's portrait image, it can then composite the two images. That is, during image compositing, if User A's and User B's portrait images overlap, the portrait image composited first in the background image will usually be covered by the one composited later. Utilizing this characteristic, during the final image compositing, the two portrait images can be composited into the same background image without any order of priority, resulting in an image like... Figure 6a The image shown is an initial composite photo of user A's portrait image and user B's portrait image. Specifically, user A's home brain device segments user A's portrait image using a first mask to obtain a target sub-image within user A's image. Similarly, it segments user B's portrait image using a second mask to obtain a target sub-image within user B's image. The target sub-images of user A and user B are then combined into the same background image without any order, forming the initial composite photo. Based on the coordinates of each overlapping pixel, the overlapping pixels are identified from the initial composite photo, and the depth information (i.e., the depth value of the overlapping pixels) of each overlapping pixel in user A's portrait image and user B's portrait image in the initial composite photo are obtained. The first implementation method can be used to adjust overlapping pixels in the initial group photo based on depth information. This method does not consider which user's portrait image the current pixel value of the overlapping pixel in the initial group photo belongs to. Instead, after determining the relationship between the depth values ​​in the first depth information and the second depth information, it determines which user's portrait image the overlapping pixel needs to be adjusted to. That is, for each overlapping pixel, the depth value of the overlapping pixel in user A's portrait image in the initial group photo is compared with the depth value of the overlapping pixel in user B's portrait image. If the depth value of the overlapping pixel in user A's portrait image is greater than that in user B's portrait image, then the RGB information of the overlapping pixel in user A's portrait image is replaced with the RGB information of the overlapping pixel in user B's portrait image. In other words, the RGB information of the overlapping pixel uses the RGB information of the overlapping pixel in user B's portrait image. In other words, the overlapping area is covered by pixels closer to the camera lens that cover pixels farther away from the camera lens.

[0116] Alternatively, a second implementation method can be adopted. This method first considers which user's portrait image the current pixel value of the overlapping pixel in the initial group photo corresponds to. Then, it determines the relationship between the first depth information and the second depth information, and adjusts the pixel value of the overlapping pixel based on this relationship. Specifically, it first determines the pixel value of each overlapping pixel in the initial group photo. For each overlapping pixel, if its pixel value corresponds to the pixel value of that overlapping pixel in user A's portrait image, it then compares the depth value of that overlapping pixel in user A's portrait image in the initial group photo with the depth value of that overlapping pixel in user B's portrait image in the initial group photo. If the depth value of the overlapping pixel in User A's portrait image is less than the depth value of the overlapping pixel in User B's portrait image, then no adjustment is needed to the pixel value of the overlapping pixel. If the depth value of the overlapping pixel in User A's portrait image is not less than the depth value of the overlapping pixel in User B's portrait image, then the RGB information of the overlapping pixel in User A's portrait image is replaced with the RGB information of the overlapping pixel in User B's portrait image. Alternatively, if the pixel value of the overlapping pixel is the pixel value of the overlapping pixel in User B's portrait image, then the depth value of the overlapping pixel in User A's portrait image in the initial group photo is compared with the depth value of the overlapping pixel in User B's portrait image. If the depth value of the overlapping pixel in User A's portrait image is less than the depth value of the overlapping pixel in User B's portrait image, then the RGB information of the overlapping pixel in User B's portrait image is replaced with the RGB information of the overlapping pixel in User A's portrait image. If the depth value of the overlapping pixel in user A's portrait image is not less than the depth value of the overlapping pixel in user B's portrait image, then there is no need to adjust the pixel value of the overlapping pixel.

[0117] For example, the process of adjusting overlapping pixels in an initial group photo based on depth information is described using the first implementation method. Specifically, if the depth value of a certain overlapping pixel in user A's portrait image is 2.1m (meaning the distance between the overlapping pixel in user A's portrait image and the lens of user A's RGBD camera is 2.1m), and the depth value of the same overlapping pixel in user B's portrait image is 2m (meaning the distance between the overlapping pixel in user B's portrait image and the lens of user B's RGBD camera is 2m), then it can be determined that the depth value of the overlapping pixel in user A's portrait image is greater than the depth value of the same overlapping pixel in user B's portrait image (2.1m is greater than 2m). Based on this, the RGB information of the overlapping pixel in user A's portrait image can be replaced with the RGB information of the overlapping pixel in user B's portrait image. Thus, by adjusting the RGB information of the overlapping pixels in User A's portrait image, the final photo of User A and User B (i.e., a composite of User A's portrait image and User B's portrait image) can be obtained.

[0118] If the depth value of the overlapping pixel in user A's portrait image is less than the depth value of the overlapping pixel in user B's portrait image, then the RGB information of the overlapping pixel in user B's portrait image will be replaced with the RGB information of the overlapping pixel in user A's portrait image. That is, the RGB information of the overlapping pixel will use the RGB information of the overlapping pixel in user A's portrait image. For example, a certain overlapping pixel has a depth value of 1.8m in user A's portrait image (i.e., the distance between this overlapping pixel and the lens of user A's RGBD camera is 1.8m), and the same overlapping pixel has a depth value of 2m in user B's portrait image (i.e., the distance between this overlapping pixel and the lens of user B's RGBD camera is 2m). Therefore, it can be determined that the depth value of this overlapping pixel in user A's portrait image is less than the depth value of this overlapping pixel in user B's portrait image (1.8m is less than 2m). Based on this, the RGB information of this overlapping pixel in user B's portrait image can be replaced with the RGB information of this overlapping pixel in user A's portrait image. Based on this, after adjusting the RGB information of this overlapping pixel in user B's portrait image, the following can be obtained: Figure 6b The final photo shown is of user A and user B together.

[0119] Furthermore, the process of adjusting overlapping pixels in the initial group photo based on depth information is described using a second implementation method. Specifically, if the pixel value of a determined overlapping pixel is the pixel value of that overlapping pixel in user A's portrait image, and if the depth value of that overlapping pixel in user A's portrait image is 2.1m and the depth value of that overlapping pixel in user B's portrait image is also 2m, then it can be determined that the depth value of that overlapping pixel in user A's portrait image is greater than the depth value of that overlapping pixel in user B's portrait image (2.1m is greater than 2m). Based on this, the RGB information of that overlapping pixel in user A's portrait image can be replaced with the RGB information of that overlapping pixel in user B's portrait image. If the depth value of the overlapping pixel in user A's portrait image is 2m and the depth value of the overlapping pixel in user B's portrait image is 2.1m, it can be determined that the depth value of the overlapping pixel in user A's portrait image is less than the depth value of the overlapping pixel in user B's portrait image (2m is less than 2.1m), and therefore no adjustment is needed to the pixel value of the overlapping pixel.

[0120] If the pixel value of a certain overlapping pixel is determined to be the pixel value of that overlapping pixel in user B's portrait image, and if the depth value of that overlapping pixel in user A's portrait image is 1.8m and the depth value of that overlapping pixel in user B's portrait image is 2m, then it can be determined that the depth value of that overlapping pixel in user A's portrait image is less than the depth value of that overlapping pixel in user B's portrait image (1.8m is less than 2m). Based on this, the RGB information of that overlapping pixel in user B's portrait image can be replaced with the RGB information of that overlapping pixel in user A's portrait image. If the depth value of that overlapping pixel in user A's portrait image is 2m and the depth value of that overlapping pixel in user B's portrait image is 1.8m, then it can be determined that the depth value of that overlapping pixel in user A's portrait image is less than the depth value of that overlapping pixel in user B's portrait image (2m is greater than 1.8m), and therefore, no adjustment to the pixel value of the overlapping pixel is needed.

[0121] If user A's portrait image and user B's portrait image do not overlap, user A's home brain device can segment user A's portrait image using a first mask to obtain the target sub-image within user A's portrait image, and then segment user B's portrait image using a second mask to obtain the target sub-image within user B's portrait image. Finally, the target sub-images of user A and user B are directly composited into the same background image to form the final photo of user A and user B together.

[0122] It should be noted that the above embodiment, based on the interaction between two users (i.e., user A and user B), performs image synthesis, which can alleviate the processing pressure caused by centrally processing portrait image data from multiple users on a single home brain device and improve the efficiency of image synthesis. For example, after user A performs image segmentation on the captured portrait image locally to obtain a mask image, user A sends the captured portrait image and the mask image to user B. When user B performs image synthesis on user A's portrait image and user B's portrait image, there is no need to perform image segmentation on user A's portrait image. Similarly, user A will also receive the mask image obtained by user B from image segmentation on the captured portrait image locally. When user A performs image synthesis on user A's portrait image and user B's portrait image, there is no need to perform image segmentation on user B's portrait image. In this way, the processing pressure caused by centrally processing portrait image data from multiple users on a single home brain device can be reduced, thereby saving the computing resources consumed when processing portrait image data from multiple users on a single home brain device.

[0123] It is understood that the above is merely an exemplary description, and the listed embodiments are only for the purpose of illustrating this solution and do not constitute a limitation on this solution. In the specific implementation of this application, the image synthesis system of user B can also perform image synthesis in the same way as user A to form the final photo of user A and user B, which will not be elaborated here.

[0124] Furthermore, in specific implementation processes, image synthesis can be performed based on three users (or four or more users, etc.). Taking three users as an example, the implementation process of a multi-user virtual group photo (i.e., multi-user image synthesis) will be explained. For instance, there are three users: User A, User B, and User C. This application embodiment uses User A's image synthesis system as an example to describe the virtual group photo process for three users. Of course, User B and User C can also perform image synthesis in the same way as User A to form the final group photo of User A, User B, and User C. Specifically, after User A uses their own data processing device (such as a home brain device) to perform image segmentation on their portrait image to obtain mask A, they will also simultaneously obtain User B's portrait image, mask B, and User C's portrait image and mask C sent by User B and User C. User A's data processing device first determines whether there are overlapping pixels among mask A, mask B, and mask C. The result of this determination may vary, and adjustments are made to the overlapping pixels in the image based on these different scenarios to obtain the final group photo of User A, User B, and User C. The implementation process for each of these possible scenarios is described below. Specifically, the process of adjusting the overlapping pixels in the initial group photo based on depth information is described using the first implementation method as an example.

[0125] Case 1: There is no overlap between mask A, mask B, and mask C.

[0126] To address this situation, User A's home brain device segments User A's portrait image using mask A to obtain the target sub-image within User A's image. It then segments User B's portrait image using mask B to obtain the target sub-image within User B's image, and finally segments User C's portrait image using mask C to obtain the target sub-image within User C's image. These target sub-images are then directly composited into the same background image to form the final group photo of Users A, B, and C.

[0127] Scenario 2: There is overlap between mask A, mask B, and mask C.

[0128] To address this situation, User A's home brain device segments User A's portrait image using mask A to obtain the target sub-image. It then segments User B's portrait image using mask B to obtain the target sub-image, and finally segments User C's portrait image using mask C to obtain the target sub-image. These target sub-images are then combined into a single background image without any specific order, forming an initial group photo. Based on the coordinates of each overlapping pixel, the device identifies these overlapping pixels from the initial group photo and obtains the depth values ​​of each overlapping pixel for User A, User B, and User C.

[0129] For each overlapping pixel in the portrait images of User A and User B, the depth values ​​of that overlapping pixel in User A and User B in the initial group photo are compared. If the depth value of the overlapping pixel in User A is greater than that in User B, then the RGB information of the overlapping pixel in User A is replaced with the RGB information of the overlapping pixel in User B. If the depth value of the overlapping pixel in User A is less than that in User B, then the RGB information of the overlapping pixel in User B is replaced with the RGB information of the overlapping pixel in User A.

[0130] For each overlapping pixel in the portrait images of User A and User C, the depth values ​​of that overlapping pixel in User A and User C in the initial group photo are compared. If the depth value of the overlapping pixel in User A is greater than that in User C, then the RGB information of the overlapping pixel in User A is replaced with the RGB information of the overlapping pixel in User C. If the depth value of the overlapping pixel in User A is less than that in User C, then the RGB information of the overlapping pixel in User C is replaced with the RGB information of the overlapping pixel in User A.

[0131] For each overlapping pixel in the portrait images of User B and User C, the depth values ​​of that overlapping pixel in User B and User C in the initial group photo are compared. If the depth value of the overlapping pixel in User B is greater than that in User C, then the RGB information of the overlapping pixel in User B is replaced with the RGB information of the overlapping pixel in User C. If the depth value of the overlapping pixel in User B is less than that in User C, then the RGB information of the overlapping pixel in User C is replaced with the RGB information of the overlapping pixel in User B.

[0132] Scenario 3: Mask A and mask B overlap, mask A and mask C do not overlap, and mask B and mask C do not overlap.

[0133] To address this situation, User A's home brain device segments User A's portrait image using mask A to obtain the target sub-image. It then segments User B's portrait image using mask B to obtain the target sub-image, and finally segments User C's portrait image using mask C to obtain the target sub-image. These target sub-images are then combined into a single background image without any specific order, forming the initial group photo. Since only mask A and mask B overlap, meaning there's an overlap between User A's and User B's portrait images in the initial group photo, only the overlapping portion needs adjustment.

[0134] Specifically, based on the coordinate positions of each overlapping pixel, the overlapping pixels are determined from the initial group photo, and the depth values ​​of each overlapping pixel for user A and user B in the initial group photo are obtained. For each overlapping pixel, the depth values ​​of that overlapping pixel for user A and user B in the initial group photo are compared. If the depth value of the overlapping pixel for user A is greater than that for user B, then the RGB information of the overlapping pixel for user A is replaced with the RGB information of the overlapping pixel for user B. If the depth value of the overlapping pixel for user A is less than that for user B, then the RGB information of the overlapping pixel for user B is replaced with the RGB information of the overlapping pixel for user A.

[0135] Scenario 4: Mask A overlaps with mask B, mask A overlaps with mask C, and mask B does not overlap with mask C.

[0136] To address this situation, User A's home brain device segments User A's portrait image using mask A to obtain the target sub-image. It then segments User B's portrait image using mask B to obtain the target sub-image, and finally segments User C's portrait image using mask C to obtain the target sub-image. These target sub-images are then composited into a single background image without any specific order, forming an initial group photo. Since only mask A and mask B, and mask A and mask C overlap, adjustments are only needed to the overlapping portions of User A's and User B's portrait images, and the overlapping portions of User A's and User C's portrait images in the initial group photo.

[0137] Specifically, based on the coordinate positions of each overlapping pixel, the overlapping pixels are determined from the initial group photo, and the depth values ​​of each overlapping pixel for user A, user B, and user C in the initial group photo are obtained. For each overlapping pixel between user A's and user B's portrait images, the depth values ​​of that overlapping pixel in user A's initial group photo and user B's initial group photo are compared. If the depth value of the overlapping pixel in user A is greater than that in user B's initial group photo, the RGB information of that overlapping pixel in user A is replaced with the RGB information of that overlapping pixel in user B. If the depth value of the overlapping pixel in user A is less than that in user B's initial group photo, the RGB information of that overlapping pixel in user B is replaced with the RGB information of that overlapping pixel in user A's initial group photo. For each overlapping pixel in the portrait images of User A and User C, the depth values ​​of that overlapping pixel in User A and User C in the initial group photo are compared. If the depth value of the overlapping pixel in User A is greater than that in User C, then the RGB information of the overlapping pixel in User A is replaced with the RGB information of the overlapping pixel in User C. If the depth value of the overlapping pixel in User A is less than that in User C, then the RGB information of the overlapping pixel in User C is replaced with the RGB information of the overlapping pixel in User A.

[0138] Case 5: Mask A and mask B overlap, mask A and mask C do not overlap, and mask B and mask C overlap.

[0139] To address this situation, User A's home brain device segments User A's portrait image using mask A to obtain the target sub-image. It then segments User B's portrait image using mask B to obtain the target sub-image, and finally segments User C's portrait image using mask C to obtain the target sub-image. These target sub-images are then composited into a single background image without any specific order, forming an initial group photo. Since only mask A overlaps with mask B, and mask B overlaps with mask C, adjustments are only needed to the overlapping portions of User A's and User B's portrait images, and the overlapping portions of User B's and User C's portrait images in the initial group photo.

[0140] Specifically, based on the coordinate positions of each overlapping pixel, the overlapping pixels are determined from the initial group photo, and the depth values ​​of each overlapping pixel for user A, user B, and user C in the initial group photo are obtained. For each overlapping pixel between user A's and user B's portrait images, the depth values ​​of that overlapping pixel in user A's initial group photo and user B's initial group photo are compared. If the depth value of the overlapping pixel in user A is greater than that in user B's initial group photo, the RGB information of that overlapping pixel in user A is replaced with the RGB information of that overlapping pixel in user B. If the depth value of the overlapping pixel in user A is less than that in user B's initial group photo, the RGB information of that overlapping pixel in user B is replaced with the RGB information of that overlapping pixel in user A's initial group photo. For each overlapping pixel in the portrait images of User B and User C, the depth values ​​of that overlapping pixel in User B and User C in the initial group photo are compared. If the depth value of the overlapping pixel in User B is greater than that in User C, then the RGB information of the overlapping pixel in User B is replaced with the RGB information of the overlapping pixel in User C. If the depth value of the overlapping pixel in User B is less than that in User C, then the RGB information of the overlapping pixel in User C is replaced with the RGB information of the overlapping pixel in User B.

[0141] Case 6: Mask A and mask B do not overlap, mask A and mask C do not overlap, and mask B and mask C overlap.

[0142] To address this situation, User A's home brain device segments User A's image using mask A to obtain the target sub-image. It then segments User B's image using mask B to obtain the target sub-image, and finally segments User C's image using mask C to obtain the target sub-image. These target sub-images are then combined into a single background image without any specific order, forming the initial group photo. Since only mask B and mask C overlap, meaning there's an overlap between User B's and User C's images in the initial group photo, only the overlapping portion needs adjustment.

[0143] Specifically, based on the coordinate positions of each overlapping pixel, the overlapping pixels are determined from the initial group photo, and the depth values ​​of each overlapping pixel for user B and user C in the initial group photo are obtained. For each overlapping pixel, the depth values ​​of that overlapping pixel for user B and user C in the initial group photo are compared. If the depth value of the overlapping pixel for user B is greater than that for user C, then the RGB information of the overlapping pixel for user B is replaced with the RGB information of the overlapping pixel for user C. If the depth value of the overlapping pixel for user B is less than that for user C, then the RGB information of the overlapping pixel for user C is replaced with the RGB information of the overlapping pixel for user B.

[0144] Case 7: Mask A and mask B do not overlap, mask A and mask C overlap, and mask B and mask C do not overlap.

[0145] To address this situation, User A's home brain device segments User A's portrait image using mask A to obtain the target sub-image. It then segments User B's portrait image using mask B to obtain the target sub-image, and finally segments User C's portrait image using mask C to obtain the target sub-image. These target sub-images are then combined into a single background image without any specific order, forming the initial group photo. Since only mask A and mask C overlap—meaning there's an overlap between User A's and User C's portrait images in the initial group photo—only the overlapping portion needs adjustment.

[0146] Specifically, based on the coordinate positions of each overlapping pixel, the overlapping pixels are determined from the initial group photo, and the depth values ​​of each overlapping pixel for user A and user C in the initial group photo are obtained. For each overlapping pixel, the depth values ​​of that overlapping pixel for user A and user C in the initial group photo are compared. If the depth value of the overlapping pixel for user A is greater than that for user C, then the RGB information of the overlapping pixel for user A is replaced with the RGB information of the overlapping pixel for user C. If the depth value of the overlapping pixel for user A is less than that for user C, then the RGB information of the overlapping pixel for user C is replaced with the RGB information of the overlapping pixel for user A.

[0147] Case 8: Mask A and mask B do not overlap, mask A and mask C overlap, and mask B and mask C overlap.

[0148] To address this situation, User A's home brain device segments User A's portrait image using mask A to obtain the target sub-image. It then segments User B's portrait image using mask B to obtain the target sub-image, and finally segments User C's portrait image using mask C to obtain the target sub-image. These target sub-images are then composited into a single background image without any specific order, forming an initial group photo. Since only mask A and mask C overlap, and mask B and mask C overlap, adjustments are only needed to the overlapping portions of User A's and User C's portrait images in the initial group photo.

[0149] Specifically, based on the coordinate positions of each overlapping pixel, the overlapping pixels are determined from the initial group photo, and the depth values ​​of each overlapping pixel for user A, user B, and user C in the initial group photo are obtained. For each overlapping pixel between user A's and user C's portrait images, the depth values ​​of that overlapping pixel in user A's initial group photo and user C's initial group photo are compared. If the depth value of the overlapping pixel in user A is greater than that in user C, the RGB information of that overlapping pixel in user A is replaced with the RGB information of that overlapping pixel in user C. If the depth value of the overlapping pixel in user A is less than that in user C, the RGB information of that overlapping pixel in user C is replaced with the RGB information of that overlapping pixel in user A. For each overlapping pixel in the portrait images of User B and User C, the depth values ​​of that overlapping pixel in User B and User C in the initial group photo are compared. If the depth value of the overlapping pixel in User B is greater than that in User C, then the RGB information of the overlapping pixel in User B is replaced with the RGB information of the overlapping pixel in User C. If the depth value of the overlapping pixel in User B is less than that in User C, then the RGB information of the overlapping pixel in User C is replaced with the RGB information of the overlapping pixel in User B.

[0150] Furthermore, it should be understood that the embodiments of this application can also centrally process portrait images of multiple users (such as two users, three users, or more than three users) on a single user's home brain device. That is, a home brain device first collects portrait images of multiple users, then sequentially segments these images to obtain portrait masks for multiple users. It then determines whether there are overlapping pixels between the segmented portrait masks. If so, the corresponding target sub-images for each user are extracted from the portrait masks and combined into a single background image to form an initial group photo of the multiple users. The RGB information of the overlapping pixels is then adjusted according to the processing method used by user A to obtain the final group photo of the multiple users. If no overlapping pixels are found, the corresponding target sub-images for each user are directly extracted from the portrait masks and combined into a single background image to form the final group photo of the multiple users.

[0151] The above embodiments show that before the first image to be synthesized and the second image to be synthesized are synthesized, it is determined whether there are overlapping pixels between the first target object and the second target object through the first mask and the second mask. When it is determined that there are overlapping pixels between the first target object and the second target object, the first image to be synthesized and the second image to be synthesized are synthesized based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, so as to determine the first fused image. Existing technologies directly synthesize multiple images without considering their depth information. This makes it impossible to determine the occlusion relationship when multiple images overlap, leading to occlusion logic errors during image synthesis and resulting in low image synthesis quality. Therefore, this solution uses the depth information of each pixel in both the first and second images to accurately and promptly determine their occlusion relationship when they overlap. This makes image synthesis more logical and improves its quality, thus solving the problem of not being able to determine which image occludes which or which image precedes the other when multiple images overlap. Furthermore, this solution directly uses a mask image to determine whether there are overlapping pixels between target objects. Compared to directly using the images to be synthesized, this reduces the computational load required for determining overlapping pixels, thereby improving the efficiency of image synthesis.

[0152] Based on the same technological concept Figure 7 An image compositing apparatus provided in an embodiment of this application is illustrated by way of example, which can execute the process of an image compositing method.

[0153] like Figure 7 As shown, the device includes:

[0154] The acquisition unit 701 is used to acquire a first image to be synthesized, a first mask image, a second image to be synthesized, and a second mask image; the first mask image is determined after extracting a first target object from the first image to be synthesized; the second mask image is determined after extracting a second target object from the second image to be synthesized; the first image to be synthesized and the second image to be synthesized include depth information of each pixel;

[0155] Processing unit 702 is configured to determine whether there are overlapping pixels between the first target object and the second target object through the first mask and the second mask; if so, based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, the first image to be synthesized and the second image to be synthesized are synthesized to determine the first fused image.

[0156] In some exemplary embodiments, the processing unit 702 is specifically used for:

[0157] The first image to be synthesized is segmented using the first mask to obtain the first target object, and the second image to be synthesized is segmented using the second mask to obtain the second target object;

[0158] The first target object and the second target object are combined to generate a second fused image;

[0159] Based on the first depth information of the overlapping pixels in the first image to be synthesized and the second depth information of the overlapping pixels in the second image to be synthesized, the second fused image is adjusted to determine the first fused image.

[0160] In some exemplary embodiments, the processing unit 702 is specifically used for:

[0161] When the first depth information is less than the second depth information, the pixel value of the overlapping pixel is set to the first pixel value of the overlapping pixel in the first composite image; or

[0162] When the first depth information is not less than the second depth information, the pixel value of the overlapping pixel is set to the second pixel value of the overlapping pixel in the second composite image.

[0163] In some exemplary embodiments, the processing unit 702 is specifically used for:

[0164] Determine the pixel values ​​of the overlapping pixels in the second fused image;

[0165] If the pixel value of the overlapping pixel is the first pixel value of the overlapping pixel in the first composite image, and the first depth information is not less than the second depth information, then the pixel value of the overlapping pixel is set to the second pixel value of the overlapping pixel in the second composite image.

[0166] If the pixel value of the overlapping pixel is the second pixel value of the overlapping pixel in the second composite image, and the first depth information is less than the second depth information, then the pixel value of the overlapping pixel is set to the first pixel value of the overlapping pixel in the first composite image.

[0167] In some exemplary embodiments, the processing unit 702 is specifically used for:

[0168] The first target object and the second target object are combined into the same background image to generate the second fused image.

[0169] In some exemplary embodiments, the first target object in the first mask image and the second target object in the second mask image have set pixel values;

[0170] The processing unit 702 is specifically used for:

[0171] By determining whether the positions of pixels with the set pixel values ​​in the first mask and the second mask are the same, it is determined whether there are overlapping pixels between the first target object and the second target object.

[0172] In some exemplary embodiments, the acquisition unit 701 is specifically used for:

[0173] The first image to be synthesized and the first mask image are provided by the first device after real-time acquisition;

[0174] The second image to be synthesized and the second mask image are provided by the second device after real-time acquisition.

[0175] Based on the same technical concept, embodiments of this application also provide a computing device, such as... Figure 8 As shown, it includes at least one processor 801 and a memory 802 connected to at least one processor. In this embodiment, the specific connection medium between the processor 801 and the memory 802 is not limited. Figure 8 Taking the connection between the processor 801 and the memory 802 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0176] In this embodiment of the application, the memory 802 stores instructions that can be executed by at least one processor 801. By executing the instructions stored in the memory 802, at least one processor 801 can perform the steps included in the aforementioned image synthesis method or image display method.

[0177] The processor 801 is the control center of the computing device. It can connect to various parts of the computing device using various interfaces and lines, and performs data processing by running or executing instructions stored in the memory 802 and calling data stored in the memory 802. Optionally, the processor 801 may include one or more processing units. The processor 801 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles issuing instructions. It is understood that the modem processor may not be integrated into the processor 801. In some embodiments, the processor 801 and the memory 802 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0178] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the image synthesis method or image display method can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0179] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 802 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 802 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0180] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer program executable by a computing device, which, when run on the computing device, causes the computing device to perform the steps of the above-described image synthesis method.

[0181] Based on the same technical concept, embodiments of this application also provide a display device, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor performs the image display method described in the second aspect.

[0182] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0186] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0187] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image synthesis method, characterized in that, include: Obtain the first image to be synthesized, the first mask image, the second image to be synthesized, and the second mask image; The first mask image is determined after extracting the first target object from the first image to be synthesized; The second mask image is determined after extracting the second target object from the second image to be synthesized; the first image to be synthesized and the second image to be synthesized include the depth information of each pixel. The first mask and the second mask are used to determine whether there are overlapping pixels between the first target object and the second target object; If so, the first image to be synthesized is segmented using the first mask to obtain the first target object, and the second image to be synthesized is segmented using the second mask to obtain the second target object; Wherein, the first depth information is the depth information of the overlapping pixels in the first image to be synthesized; the second depth information is the depth information of the overlapping pixels in the second image to be synthesized. When the first depth information is less than the second depth information, the pixel value of the overlapping pixel is set to the first pixel value of the overlapping pixel in the first composite image; or when the first depth information is not less than the second depth information, the pixel value of the overlapping pixel is set to the second pixel value of the overlapping pixel in the second composite image. or; The first target object and the second target object are combined to generate a second fused image; Determine the pixel value of the overlapping pixel in the second fused image; if the pixel value of the overlapping pixel is the first pixel value of the overlapping pixel in the first image to be synthesized, and the first depth information is not less than the second depth information, set the pixel value of the overlapping pixel to the second pixel value of the overlapping pixel in the second image to be synthesized; if the pixel value of the overlapping pixel is the second pixel value of the overlapping pixel in the second image to be synthesized, and the first depth information is less than the second depth information, set the pixel value of the overlapping pixel to the first pixel value of the overlapping pixel in the first image to be synthesized.

2. The method as described in claim 1, characterized in that, The step of merging the first target object and the second target object to generate a second fused image includes: The first target object and the second target object are combined into the same background image to generate the second fused image.

3. The method as described in claim 1, characterized in that, The first target object in the first mask image and the second target object in the second mask image have set pixel values; Determining whether the first target object and the second target object have overlapping pixels by using the first mask and the second mask includes: By determining whether the positions of pixels with the set pixel values ​​in the first mask and the second mask are the same, it is determined whether there are overlapping pixels between the first target object and the second target object.

4. The method according to any one of claims 1 to 3, characterized in that, Obtaining the first image to be composited, the first mask image, the second image to be composited, and the second mask image includes: The first image to be synthesized and the first mask image are provided by the first device after real-time acquisition; The second image to be synthesized and the second mask image are provided by the second device after real-time acquisition.

5. A computing device, characterized in that, The method includes at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 4.

Images