Cloud photo taking method, device, computer equipment and storage medium

Abstract

The present application relates to a cloud group photo method, device, computer equipment and storage medium. The method includes: in the process of cloud group photo, obtaining the object preview image of this terminal in real time; the terminal is one of the terminals participating in the cloud group photo; the object preview image is the image obtained by the terminal performing real-time image acquisition on the corresponding cloud group photo object; the corresponding cloud group photo object is the object of the cloud group photo taken using this terminal; the object preview image obtained by this terminal is subjected to object cutout processing to obtain a cutout result; the group photo preview image is displayed in real time; the group photo preview image is obtained by synthesizing the cutout results obtained by each terminal participating in the cloud group photo; after receiving the photo taking instruction, the generated group photo is displayed; the group photo is generated based on the current real-time group photo preview image. The use of this method can improve the effect of cloud group photos.

Description

Technical Field

[0001] The present application relates to computer technology and image processing technology, and in particular to a cloud group photo method, apparatus, computer equipment, and storage medium. Background Art

[0002] Photography technology is now very popular, but when people need to take photos, they sometimes face the situation of being separated from others by many places. For example, when a group photo is needed during an event or a party, some people are unable to arrive at the scene. In this case, cloud photo taking is a good solution, which can enable multiple parties to take photos remotely.

[0003] Traditional cloud group photo methods typically involve users on each device taking their own photos and uploading them. The server then performs a cutout of the portraits from each uploaded user's photos, then simply overlays the cutouts onto a common background to create a composite photo. This photo is then sent to each device for display. Therefore, traditional methods simply overlay the portraits from multiple photos onto a common background, resulting in poor cloud group photo quality. Summary of the Invention

[0004] Based on this, it is necessary to provide a cloud photo taking method, device, computer equipment, computer-readable storage medium and computer program product that can improve the effect of cloud photo taking in response to the above technical problems.

[0005] In a first aspect, the present application provides a cloud photo taking method. The method comprises:

[0006] During a cloud group photo, a preview image of an object of the terminal is obtained in real time; the terminal is one of the terminals participating in the cloud group photo; the preview image of the object is an image obtained by the terminal performing real-time image acquisition of the corresponding cloud group photo object; the corresponding cloud group photo object is the object of the cloud group photo taken using the terminal;

[0007] Performing object cutout processing on the object preview image obtained by the terminal to obtain a cutout result;

[0008] Displaying a preview image of the group photo in real time; the preview image of the group photo is synthesized based on the cutout results obtained by each terminal participating in the cloud group photo;

[0009] After receiving the photo-taking instruction, the generated group photo is displayed; the group photo is generated based on the current real-time preview image of the group photo.

[0010] In a second aspect, the present application further provides a cloud photo taking device. The device includes:

[0011] An image acquisition module is configured to acquire, in real time, a preview image of an object from the terminal during a cloud photo session; the terminal being one of the terminals participating in the cloud photo session; the preview image of the object being acquired by the terminal through real-time image acquisition of the corresponding cloud photo subject; the corresponding cloud photo subject being the subject of the cloud photo session using the terminal;

[0012] A cutout module is used to perform object cutout processing on the object preview image obtained by the terminal to obtain a cutout result;

[0013] The display module is used to display a preview image of the group photo in real time; the preview image of the group photo is obtained by synthesizing the image based on the cutout results obtained by each terminal participating in the cloud group photo; after receiving the photo-taking instruction, the generated group photo is displayed; the group photo is generated based on the current real-time preview image of the group photo.

[0014] In one embodiment, the display module is further used to enter the cloud group photo room based on the room credential of the cloud group photo room; in the object area of ​​the cloud group photo room, the object preview image corresponding to each terminal participating in the cloud group photo is displayed in real time; in the photo preview area of ​​the cloud group photo room, the photo preview image is displayed in real time.

[0015] In one embodiment, the apparatus further comprises:

[0016] a posture evaluation module, configured to evaluate the posture rationality of the objects in the group photo preview image to obtain a posture rationality evaluation result;

[0017] The display module is further configured to display posture adjustment prompt information based on the posture rationality evaluation result; the posture adjustment prompt information is configured to prompt the cloud photo subject to adjust the posture; and to display a real-time preview image of the photo after the posture is adjusted in real time.

[0018] In one embodiment, the posture evaluation module is further configured to perform posture rationality evaluation on the object in the group photo preview image using a pre-trained posture rationality evaluation model to obtain a posture rationality evaluation result;

[0019] The pose rationality assessment model is obtained by pre-training the model based on positive samples and negative samples; the positive samples are real group photos; and the negative samples are synthesized group photos.

[0020] In one embodiment, the group photo is a harmonious group photo; the device further comprises:

[0021] a harmonization module, configured to, after receiving a photo-taking instruction, harmonize the foreground and background of the group photo preview image through a beauty server to obtain a harmonized group photo;

[0022] The display module is further configured to display the harmonized group photo obtained from the beautification server.

[0023] In one embodiment, the harmonization process is implemented by a harmonization network model; the harmonization network model is obtained in advance through a model training process; the loss function used in the model training process includes at least one of a perceptual loss, a style loss, and a total variation loss;

[0024] The perceptual loss is used to characterize the difference in feature information between the predicted group photo obtained during model training and the target group photo in the training sample in the feature space;

[0025] The style loss is used to characterize the style difference between the predicted group photo and the target group photo;

[0026] The total change loss is used to represent the difference between the pixel values ​​of adjacent pixels in the group prediction image.

[0027] In one embodiment, the harmonized group photo obtained through the harmonization process is an initial harmonized group photo; the harmonization module is further configured to perform color migration on the initial harmonized group photo through the beautification server to align color attributes of facial regions of various subjects in the initial harmonized group photo to obtain a final harmonized group photo;

[0028] The display module is further configured to display the final harmonized group photo obtained from the beautification server.

[0029] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the cloud group photo method described in each embodiment of the present application.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the steps of the cloud group photo taking method described in each embodiment of the present application.

[0031] In a fifth aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, causes the processor to execute the steps of the cloud group photo taking method described in each embodiment of the present application.

[0032] The above-mentioned cloud group photo method, device, computer equipment, storage medium and computer program product, during the cloud group photo process, obtains the object preview image of the terminal in real time, performs object cutout processing on the object preview image obtained by the terminal, obtains the cutout result, and displays the group photo preview image in real time. The group photo preview image is obtained by image synthesis based on the cutout results obtained by each terminal participating in the cloud group photo. Therefore, when multiple terminals take cloud group photos at the same time, the group photo preview effect can be displayed in real time through the group photo preview image, which facilitates the objects at each terminal participating in the cloud group photo to adjust their own expressions, postures and positions in real time according to the real-time displayed group photo preview image. After receiving the photo taking instruction, the group photo generated based on the current real-time group photo preview image is displayed, thereby improving the cloud group photo effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a diagram of an application environment of a cloud photo taking method in one embodiment;

[0034] Figure 2 A diagram illustrating an application environment of a cloud photo taking method in another embodiment;

[0035] Figure 3 1. A schematic diagram of a process of a cloud photo taking method according to an embodiment;

[0036] Figure 4 Schematic diagram of the overall process of a cloud photo taking method in one embodiment;

[0037] Figure 5 Schematic diagram of positive samples and negative samples for training a pose rationality assessment model in one embodiment;

[0038] Figure 6 A comparison diagram before and after harmonization in one embodiment;

[0039] Figure 7 This is a structural block diagram of a cloud photo taking device in one embodiment;

[0040] Figure 8 is a structural block diagram of a cloud photo taking device in another embodiment;

[0041] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] In one embodiment, the cloud photo taking method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, each terminal 102 participating in a cloud group photo communicates with a cloud server 104 via a network. There are at least two terminals 102. During the cloud group photo process, each terminal 102 can obtain a preview image of its own object in real time, perform object cutout processing on the preview image obtained by the terminal, obtain a cutout result, and send the cutout result to the cloud server 104. The cloud server 104 can perform image synthesis based on the cutout results obtained by each terminal 102 participating in the cloud group photo to obtain a group photo preview image, and send the group photo preview image to each terminal 102. Each terminal 102 can display the group photo preview image in real time. After receiving a photo capture instruction, each terminal 102 can display a group photo generated based on the current real-time group photo preview image. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, camera devices, IoT devices, and portable wearable devices. Camera devices can be cameras or camcorders, etc. IoT devices can be smart TVs or smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The cloud server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0044] In another embodiment, the cloud photo taking method provided in the embodiment of the present application can also be applied to Figure 2In the application environment shown, each terminal 102 participating in a cloud group photo communicates with a cloud server 104 via a network, and the cloud server 104 communicates with a beauty server 106 via a network. There are at least two terminals 102. During the cloud group photo process, each terminal 102 participating in the cloud group photo can obtain an object preview image of its own terminal in real time, perform object cutout processing on the object preview image obtained by its own terminal, obtain a cutout result, and send the cutout result to the cloud server 104. The cloud server 104 can perform image synthesis based on the cutout results obtained by each terminal 102 participating in the cloud group photo to obtain a group photo preview image, and send the group photo preview image to each terminal 102. Each terminal 102 can display the group photo preview image in real time. After at least one terminal 102 receives a photo taking instruction, the cloud server 104 can send the group photo preview image to the beauty server 106. The beauty server 106 can perform harmonization processing on the group photo preview image to obtain a harmonized group photo, which each terminal 102 can display. Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, cameras, IoT devices, and portable wearable devices. Cameras may include cameras or video cameras, and IoT devices may include smart TVs or smart car-mounted devices. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, etc. Cloud server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers. Beauty server 106 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0045] In one embodiment, Figure 3 As shown, a cloud photo taking method is provided, comprising the following steps:

[0046] Step 302: During the cloud photo process, obtain the object preview image of this terminal in real time; this terminal is one of the terminals participating in the cloud photo; the object preview image is the image obtained by this terminal performing real-time image acquisition of the corresponding cloud photo object; the corresponding cloud photo object is the object taking the cloud photo using this terminal.

[0047] Among them, cloud group photo refers to a group photo method in which users of multiple terminals use their respective terminals to take photos remotely, and obtain a group photo that includes the users corresponding to each terminal.

[0048] In one embodiment, the number of terminals participating in the cloud photo is at least two.

[0049] In one embodiment, the cloud photo subject can be a person, or an animal such as a cat or a dog, without limitation.

[0050] Specifically, during the cloud group photo process, each terminal participating in the cloud group photo can respectively obtain the object preview image of the terminal in real time.

[0051] It can be understood that real-time acquisition refers to continuously acquiring the object preview image of the terminal frame by frame.

[0052] Step 304: perform object cutout processing on the object preview image obtained by the terminal to obtain a cutout result.

[0053] The object cutout processing refers to the processing of segmenting the image content corresponding to the cloud photo object from the object preview image.

[0054] Specifically, each terminal participating in the cloud group photo can respectively perform object cutout processing on the object preview image obtained by the terminal in real time to obtain the cutout result of each terminal.

[0055] like Figure 4 As shown, terminal 1 and terminal 2 perform object cutout processing on their respective object preview images to obtain respective cutout results. The figure only schematically shows the cutout result of terminal 2. In actual application, each terminal performs cutout processing to obtain its own cutout result.

[0056] In one embodiment, the terminal may perform object cutout processing on the object preview image obtained by the terminal through a lightweight object cutout model deployed on the terminal to obtain a cutout result.

[0057] The object cutout model is a model used to perform object cutout processing. In one embodiment, the object cutout model can be a machine learning model. In one embodiment, the object cutout model can be a deep learning model.

[0058] In one embodiment, each terminal participating in the cloud photo can upload its own cutout results to the cloud server in real time. Figure 4 As shown, the cutout results of each terminal can be uploaded to the cloud server through the streaming server.

[0059] In one embodiment, the uploaded cutout result may include a preview image of the object and an object mask obtained by the object cutout process. The object mask is a binary image corresponding to the object in the cloud photo. In another embodiment, the uploaded cutout result may include a color cutout result obtained by the object cutout process.

[0060] Step 306 , displaying a preview image of the group photo in real time; the preview image of the group photo is obtained by synthesizing the cutout results obtained by each terminal participating in the cloud group photo.

[0061] Image compositing refers to the process of combining the various cutout results into a single background image. The group photo preview image is an image containing all the cloud photo objects.

[0062] In one embodiment, the cloud server can synthesize images based on the cutout results obtained by each terminal participating in the cloud group photo to obtain a group photo preview image. Figure 4 , a preview image of the group photo obtained by image synthesis by the cloud server is shown in FIG.

[0063] In one embodiment, the cloud server may send the group photo preview image to each terminal participating in the cloud group photo, and each terminal may display the group photo preview image in real time.

[0064] In one embodiment, Figure 4 As shown, the group photo preview image obtained by the cloud server can be sent to each terminal participating in the cloud group photo through the streaming server.

[0065] It is understood that both the object preview image and the group photo preview image are real-time images. The terminal can display the group photo preview image frame by frame in real time.

[0066] In one embodiment, the cloud photo subject can autonomously adjust at least one of his / her own expression, position or posture according to the real-time preview image of the photo.

[0067] In another embodiment, the terminal may display prompt information determined based on the preview image of the group photo, and the cloud photo subject may adjust at least one of their expression, position, or posture based on the prompt information displayed by the terminal. The prompt information is used to prompt the cloud photo subject to adjust at least one of their expression, position, or posture.

[0068] In other embodiments, the cloud photo object can also operate on the group photo preview image displayed on the terminal to adjust the position of the cloud photo object in the group photo preview image. For example, by dragging the cloud photo object in the group photo preview image, the position of the cloud photo object in the group photo preview image can be adjusted.

[0069] In one embodiment, the cloud photo subject can also adjust the auxiliary information in the photo preview image through the terminal. Auxiliary information refers to image information in the photo preview image other than the cloud photo subject. For example, the auxiliary information can include at least one of a background image or stickers. For example, the cloud photo subject can change the background image or stickers in the photo preview image, and can also adjust the sticker position.

[0070] Step 308: After receiving the photo taking instruction, the generated group photo is displayed; the group photo is generated based on the current real-time group photo preview image.

[0071] The photo taking instruction is an instruction for triggering a photo taking, and the group photo is a photo obtained through the cloud group photo that includes all the cloud group photo objects.

[0072] In one embodiment, the photographing instruction may be a triggering operation for a photographing identifier. In one embodiment, the photographing identifier may be an option displayed on a display screen of the terminal for triggering photographing, or may be a hardware button provided on the terminal for triggering photographing.

[0073] In one embodiment, after at least one terminal receives a photo-taking instruction, each terminal participating in the cloud group photo may display the generated group photo.

[0074] In one embodiment, after receiving the photo-taking instruction, each terminal may generate and display a group photo based on the current real-time group photo preview image.

[0075] In another embodiment, after receiving the photo-taking instruction, the group photo preview image may be harmonized to obtain a harmonized group photo, and each terminal may display the harmonized group photo.

[0076] In one embodiment, after receiving the photo-taking instruction, the group photo preview image can be sent to the beautification server. The beautification server can harmonize the group photo preview image to obtain a harmonized group photo. Each terminal can obtain the harmonized group photo obtained by the beautification server and display it.

[0077] The above-mentioned cloud group photo method obtains the object preview image of this terminal in real time during the cloud group photo process, performs object cutout processing on the object preview image obtained by this terminal, obtains the cutout result, and displays the group photo preview image in real time. The group photo preview image is obtained by synthesizing the images based on the cutout results obtained by each terminal participating in the cloud group photo. Therefore, when multiple terminals take cloud group photos at the same time, the group photo preview effect can be displayed in real time through the group photo preview image, which makes it convenient for the objects at each terminal participating in the cloud group photo to adjust their own expressions, postures and positions in real time according to the real-time displayed group photo preview image, thereby improving the real-time performance. After receiving the photo-taking instruction, the group photo generated according to the current real-time group photo preview image is displayed, thereby improving the cloud group photo effect and obtaining a more realistic group photo.

[0078] In one embodiment, before obtaining the object preview image of this terminal in real time during the cloud group photo process, the method also includes: entering the cloud group photo room according to the room credential of the cloud group photo room; displaying the object preview image corresponding to each terminal participating in the cloud group photo in real time in the object area of ​​the cloud group photo room; real-time display of the group photo preview image includes: real-time display of the group photo preview image in the group photo preview area of ​​the cloud group photo room.

[0079] The room credential is the credential for entering the cloud photo room. The cloud photo room is a virtual room used for multiple cloud photo subjects to take cloud photos together.

[0080] In one embodiment, the room credential may be any one of a room number, a room link, a room QR code, and the like.

[0081] In one embodiment, each terminal participating in a cloud group photo can enter the same cloud group photo room using a common room credential. Each terminal participating in the cloud group photo can display a preview image of the object corresponding to each terminal participating in the cloud group photo in real time in the object area of ​​the cloud group photo room, and display a preview image of the group photo in real time in the group photo preview area of ​​the cloud group photo room.

[0082] In one embodiment, each terminal may only display the object preview image of the terminal itself in the object area. In another embodiment, each terminal may display the object preview images of various terminals in the object area.

[0083] It is understood that the object preview image and the group photo preview image are real-time images. The terminal can display the object preview image and the group photo preview image frame by frame in real time.

[0084] In one embodiment, each cloud photo partner can make a voice call in the cloud photo room through a terminal.

[0085] In one embodiment, the beautification server can create a cloud group photo room. In one embodiment, after receiving a photo taking instruction, the beautification server can first authenticate the cloud group photo room, and after the authentication is passed, perform harmonization on the group photo preview image to obtain a harmonized group photo.

[0086] In the above embodiment, each terminal enters the cloud photo room according to the room certificate of the cloud photo room, and displays the object preview image corresponding to each terminal in real time in the object area of ​​the cloud photo room, and displays the photo preview image in real time in the photo preview area of ​​the cloud photo room, so that each cloud photo object can take a cloud photo together in the same cloud photo room, and can preview the photo effect in real time, thereby improving the cloud photo effect.

[0087] In one embodiment, after receiving a photo-taking instruction and before displaying the generated group photo, the method further includes: performing a posture rationality assessment on the objects in the group photo preview image to obtain a posture rationality assessment result; displaying posture adjustment prompt information based on the posture rationality assessment result; the posture adjustment prompt information is used to prompt the cloud group photo objects to adjust their posture; and displaying the real-time group photo preview image after the posture adjustment in real time.

[0088] The posture rationality assessment is a process of evaluating the rationality of at least one of the position and posture of the object in the group photo preview image. The posture may include at least one of the position and posture.

[0089] In one embodiment, the cloud server can perform a pose rationality assessment on the subject in the group photo preview image, obtain a pose rationality assessment result, and send the pose rationality assessment result to each terminal. Each terminal can display pose adjustment prompt information based on the pose rationality assessment result to prompt the cloud group photo subject to adjust the pose, and display the real-time group photo preview image after the pose adjustment in real time.

[0090] In one embodiment, the pose rationality assessment result may be a pose rationality score. In other embodiments, the pose rationality assessment result may also be a pose rationality assessment level, which is not limited.

[0091] In one embodiment, the terminal may directly display the posture rationality assessment result as posture adjustment prompt information, for example, the terminal directly displays the posture rationality score or posture rationality assessment level.

[0092] In another embodiment, the terminal may generate a posture adjustment prompt based on the posture rationality assessment result for display. For example, based on the posture rationality score or posture rationality assessment level, a posture adjustment prompt may be generated, for example, the posture adjustment prompt may be "The current position is unreasonable, please adjust."

[0093] It can be understood that since the object preview image and the group photo preview image are both real-time, after the cloud photo object adjusts its posture, the terminal can perform real-time object cutout processing on the real-time object preview image after the adjusted posture, and display the real-time group photo preview image after the adjusted posture obtained by image synthesis based on the real-time cutout result in real time.

[0094] In the above embodiment, the posture rationality of the objects in the group photo preview image is evaluated to obtain a posture rationality evaluation result. Based on the posture rationality evaluation result, posture adjustment prompt information is displayed, so that the posture rationality of the cloud group photo objects can be automatically evaluated, and then the posture adjustment prompt information is given, which can more intelligently prompt the cloud group photo objects to adjust their posture. By adjusting the posture, a more realistic and natural group photo can be obtained, avoiding the problem that the positions and postures of the objects in the group photo are too stiff and do not look like a real group photo, thereby improving the effect of the cloud group photo.

[0095] In one embodiment, performing pose rationality assessment on objects in a group photo preview image to obtain a pose rationality assessment result includes: performing pose rationality assessment on the objects in the group photo preview image using a pre-trained pose rationality assessment model to obtain the pose rationality assessment result. The pose rationality assessment model is pre-trained based on positive and negative samples. The positive samples are real group photos, and the negative samples are synthesized group photos.

[0096] like Figure 5As shown in the figure, positive samples and negative samples are shown respectively. It can be seen that since the positive sample is a real group photo, the posture relationship of each object in the positive sample is reasonable, while the negative sample is a synthetic group photo, so the posture relationship of each object in the negative sample does not look real enough.

[0097] In one embodiment, the pose rationality assessment model may be a machine learning model. In one embodiment, the pose rationality assessment model may be a deep learning model. In one embodiment, the pose rationality assessment model may be a deep residual network (ResNet) model. In one embodiment, the pose rationality assessment model may be a ResNet18 (a deep residual network) model.

[0098] Specifically, the cloud server can input the group photo preview image into the pre-trained pose rationality evaluation model, perform pose rationality evaluation on the objects in the group photo preview image through the pose rationality evaluation model, and output the pose rationality evaluation result. Figure 4 In the figure, it is shown that the cloud server evaluates the position rationality of the group photo preview image through the pose rationality evaluation model.

[0099] In one embodiment, the pose rationality assessment model is obtained in advance through a model training step. The model training step of the pose rationality assessment model includes: inputting pose training samples into the pose rationality assessment model to be trained, outputting pose rationality prediction results, and then iteratively adjusting model parameters of the pose rationality assessment model to be trained based on the difference between the pose rationality prediction results and the rationality labels of the pose training samples until an iteration stopping condition is met, thereby obtaining a trained pose rationality assessment model.

[0100] In one embodiment, the pose training samples include positive samples and negative samples. The rationality label of the positive sample is a positive label, and the rationality label of the negative sample is a negative label. For example, the rationality label of the positive sample can be 1, and the rationality label of the negative sample can be 0.

[0101] In one embodiment, the pose rationality assessment model can be trained using the ImageNet dataset (a large visualization database used for visual object recognition software research) as pose training samples to obtain an initial pose rationality assessment model. The initial pose rationality assessment model is then trained using self-made positive and negative samples as pose training samples to fine-tune the model parameters of the pose rationality assessment model, resulting in a trained pose rationality assessment model.

[0102] In the above embodiment, the posture rationality assessment model is trained by using real group photos as positive samples and synthesized group photos as negative samples, so that the posture rationality assessment model can have the ability to accurately assess the posture rationality of objects in the group photo preview image, thereby more intelligently and accurately prompting cloud group photo objects to adjust their posture. By adjusting the posture, a more realistic and natural group photo can be obtained, avoiding the problem that the positions and postures of the objects in the group photo are too stiff and do not look like real photos, thereby improving the effect of the cloud group photo.

[0103] In one embodiment, the group photo is a harmonized group photo; after receiving a photo-taking instruction, displaying the generated group photo includes: after receiving the photo-taking instruction, harmonizing the foreground and background in the group photo preview image through a beautification server to obtain a harmonized group photo; and displaying the harmonized group photo obtained from the beautification server.

[0104] The foreground refers to the image content corresponding to the cloud photo object in the group photo preview image, and the background refers to the image content other than the cloud photo object in the group photo preview image.

[0105] Specifically, after at least one terminal receives a photo-taking instruction, the beautification server can harmonize the foreground and background in the current real-time group photo preview image to obtain a harmonized group photo. The terminal can display the harmonized group photo obtained from the beautification server as a cloud group photo result.

[0106] In one embodiment, the beautification server may harmonize the color attribute information of the foreground and background in the group photo preview image to obtain a harmonized group photo.

[0107] In one embodiment, the color attribute information may include at least one of light, brightness, or contrast.

[0108] In the above embodiment, after receiving the photo-taking instruction, the foreground and background in the group photo preview image are harmonized by the beautification server to obtain a harmonized group photo, and the harmonized group photo obtained from the beautification server is displayed, so that the light, brightness or contrast and other information between the foreground and background in the displayed harmonized group photo are more harmonious, avoiding the problem that the color attribute difference between the foreground and background in the directly synthesized group photo is too large, resulting in an unrealistic appearance, making the cloud group photo result more harmonious, real and natural, and improving the cloud group photo effect.

[0109] In one embodiment, the harmonization process is implemented through a harmonization network model; the harmonization network model is obtained in advance through a model training process; the loss function used in the model training process includes at least one of perceptual loss, style loss and total variation loss.

[0110] The perceptual loss is used to characterize the difference in feature space between the predicted group photo obtained during model training and the target group photo in the training sample. The style loss is used to characterize the style difference between the predicted group photo and the target group photo. The total variation loss (TV loss) is used to characterize the difference in pixel values ​​between adjacent pixels in the predicted group photo.

[0111] In one embodiment, the harmonized network model may be a deep learning model. In one embodiment, the harmonized network model may be a RainNet network (a deep learning network) model.

[0112] Specifically, the beautification server may input the group photo preview image into a pre-trained harmonization network model, harmonize the foreground and background in the group photo preview image through the harmonization network model, and output a harmonized group photo.

[0113] In one embodiment, the model training process of the harmonized network model includes: inputting the sample image and the target photo as training samples into the harmonized network model to be trained, obtaining a photo prediction map based on the sample image through the harmonized network model, iteratively adjusting the model parameters of the harmonized network model according to the difference between the target photo and the photo prediction map until the iteration stop condition is met, thereby obtaining a trained harmonized network model.

[0114] In one embodiment, the difference between the target photo and the predicted photo can be represented by a loss function. In one embodiment, the loss function can include at least one of perceptual loss, style loss, and total variation loss. Perceptual loss, style loss, and total variation loss are loss functions in the fields of image filling and style transfer. The loss function in this embodiment can be expressed as:

[0115] Target loss=perpetual loss*lambda+style loss*beta+Tv loss*gamma;

[0116] Target loss is the loss function, perpetual loss is the perceptual loss, style loss is the style loss, and Tv loss is the total change loss. lambda, beta, and gamma are hyperparameters.

[0117] In a well-thought-out approach, one can use a small dataset to perform a hyperparameter search to determine lambda, beta, and gamma.

[0118] In one embodiment, the loss function may include adversarial loss (Gan loss) in addition to perceptual loss, style loss, and total variation loss. The loss function in this embodiment can be expressed as:

[0119] Total loss=Gan loss+Target loss;

[0120] Among them, Total loss is the final loss function, Gan loss is the adversarial loss, and Target loss is the loss function that includes perceptual loss, style loss, and total change loss.

[0121] In this embodiment, the loss functions in the fields of image filling and style transfer, such as perceptual loss, style loss, and total variation loss, are transferred to the harmonization process, which can effectively avoid the problem of artifacts in the harmonized photo and obtain a better harmonized photo.

[0122] In one embodiment, the harmonized group photo obtained through the harmonization process is the initial harmonized group photo; after receiving the photo-taking instruction, the foreground and background in the group photo preview image are harmonized by the beautification server to obtain the harmonized group photo, and the method also includes: performing color migration on the initial harmonized group photo by the beautification server to align the color attributes of the facial areas of each object in the initial harmonized group photo to obtain the final harmonized group photo; displaying the harmonized group photo obtained from the beautification server includes: displaying the final harmonized group photo obtained from the beautification server.

[0123] Color migration refers to the process of modifying the color attributes of one object based on the color attributes of another object.

[0124] Specifically, after at least one terminal receives a photo-taking instruction, the beautification server can harmonize the foreground and background in the group photo preview image to obtain an initial harmonized group photo. It can then perform color migration on the initial harmonized group photo to align the color attributes of the facial regions of the subjects in the initial harmonized group photo to obtain a final harmonized group photo. Each terminal can display the final harmonized group photo obtained from the beautification server as the cloud group photo result.

[0125] In one embodiment, the color attribute may include at least one of hue, saturation, or brightness.

[0126] In one embodiment, the beautification server may perform color migration on the facial regions of the objects in the initial harmonized group photo, that is, modify the color attributes of the facial regions of the other objects according to the color attributes of the facial region of one object, to obtain the final harmonized group photo.

[0127] like Figure 4As shown in the figure, the beautification server can perform harmonization processing through the harmonization network model and face alignment through color migration, thereby producing a harmonized group photo. As can be seen from the figure, the lighting between the foreground and background in the harmonized group photo is more harmonious, and the colors of each facial area are more consistent.

[0128] like Figure 6 As shown, the group photo before and after harmonization is shown. It can be seen that in the group photo before harmonization, the light between the foreground and the background is not harmonious, and the colors of the two faces are not consistent, one is darker and the other is brighter. In the group photo after harmonization (i.e., the final harmonized group photo), the light between the foreground and the background is more harmonious, and the colors of the two faces are more consistent.

[0129] In the above embodiment, through color migration, the hue, saturation, brightness and other information of the facial areas of each object in the group photo are aligned, ensuring the consistency of the facial areas of each object in the final harmonious group photo, making the cloud group photo result more realistic and improving the cloud group photo effect.

[0130] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0131] Based on the same inventive concept, embodiments of the present application also provide a cloud group photo device for implementing the aforementioned cloud group photo method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more cloud group photo device embodiments provided below can be found in the above-mentioned limitations of the cloud group photo method and will not be further elaborated here.

[0132] In one embodiment, Figure 7 As shown, a cloud photo taking device 700 is provided, comprising: an image acquisition module 702, a cutout module 704 and a display module 706, wherein:

[0133] Image acquisition module 702 is used to obtain the object preview image of this terminal in real time during the cloud photo process; this terminal is one of the terminals participating in the cloud photo; the object preview image is the image obtained by this terminal performing real-time image acquisition of the corresponding cloud photo object; the corresponding cloud photo object is the object used to take the cloud photo using this terminal.

[0134] The cutout module 704 is configured to perform object cutout processing on the object preview image obtained by the terminal to obtain a cutout result.

[0135] Display module 706 is used to display the group photo preview image in real time; the group photo preview image is obtained by synthesizing the image based on the cutout results obtained by each terminal participating in the cloud group photo; after receiving the photo taking instruction, the generated group photo is displayed; the group photo is generated based on the current real-time group photo preview image.

[0136] In one embodiment, the display module 706 is also used to enter the cloud group photo room according to the room certificate of the cloud group photo room; in the object area of ​​the cloud group photo room, the object preview images corresponding to each terminal participating in the cloud group photo are displayed in real time; in the photo preview area of ​​the cloud group photo room, the photo preview image is displayed in real time.

[0137] In one embodiment, Figure 8 As shown, the device 700 further includes:

[0138] The posture evaluation module 708 is used to evaluate the posture rationality of the objects in the group photo preview image to obtain a posture rationality evaluation result.

[0139] The display module 706 is also used to display posture adjustment prompt information based on the posture rationality evaluation result; the posture adjustment prompt information is used to prompt the cloud photo subject to adjust the posture; and to display the real-time photo preview image after the posture is adjusted in real time.

[0140] In one embodiment, the pose assessment module 708 is further configured to perform pose rationality assessment on the objects in the group photo preview image using a pre-trained pose rationality assessment model to obtain a pose rationality assessment result. The pose rationality assessment model is pre-trained using positive and negative samples; the positive samples are real group photos, and the negative samples are synthesized group photos.

[0141] In one embodiment, the group photo is a harmonious group photo. Figure 8 As shown, the apparatus 700 further includes:

[0142] The harmonization module 710 is used to, after receiving a photo-taking instruction, harmonize the foreground and background in the group photo preview image through the beauty server to obtain a harmonized group photo.

[0143] The display module 706 is further configured to display the harmonized group photo obtained from the beautification server.

[0144] In one embodiment, the harmonization processing is implemented through a harmonization network model; the harmonization network model is obtained in advance through a model training process; the loss function used in the model training process includes at least one of perceptual loss, style loss and total change loss; wherein, perceptual loss is used to characterize the difference between the feature information in the feature space between the group photo prediction image obtained during the model training process and the target group photo in the training sample; style loss is used to characterize the style difference between the group photo prediction image and the target group photo; total change loss is used to characterize the difference between the pixel values ​​of adjacent pixels in the group photo prediction image.

[0145] In one embodiment, the harmonized group photo obtained through the harmonization process is the initial harmonized group photo; the harmonization module 710 is further used to perform color migration on the initial harmonized group photo through the beautification server to align the color attributes of the facial areas of each object in the initial harmonized group photo to obtain the final harmonized group photo; the display module 706 is further used to display the final harmonized group photo obtained from the beautification server.

[0146] The above-mentioned cloud group photo device obtains the object preview image of this terminal in real time during the cloud group photo process, performs object cutout processing on the object preview image obtained by this terminal, obtains the cutout result, and displays the group photo preview image in real time. The group photo preview image is obtained by synthesizing the images based on the cutout results obtained by each terminal participating in the cloud group photo. Therefore, when multiple terminals take cloud group photos at the same time, the group photo preview effect can be displayed in real time through the group photo preview image, which facilitates the objects at each terminal participating in the cloud group photo to adjust their own expressions, postures and positions in real time according to the real-time displayed group photo preview image. After receiving the photo-taking instruction, the group photo generated based on the current real-time group photo preview image is displayed, thereby improving the cloud group photo effect.

[0147] Each module in the aforementioned cloud photo-taking device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0148] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a cloud photo taking method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0149] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0150] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0152] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0154] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0155] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A cloud photo taking method, characterized in that: The method comprises: Enter the cloud photo room according to the room voucher of the cloud photo room; In the object area of ​​the cloud photo room, preview images of the objects corresponding to the terminals participating in the cloud photo are displayed in real time; During a cloud group photo, a preview image of an object of the terminal is obtained in real time; the terminal is one of the terminals participating in the cloud group photo; the preview image of the object is an image obtained by the terminal performing real-time image acquisition of the corresponding cloud group photo object; the corresponding cloud group photo object is the object of the cloud group photo taken using the terminal; Performing object cutout processing on the object preview image obtained by the terminal to obtain a cutout result; A preview image of the group photo is displayed in real time in the group photo preview area of ​​the cloud group photo room; the group photo preview image is obtained by synthesizing the image based on the cutout results obtained by each terminal participating in the cloud group photo; after receiving the photo-taking instruction, the beauty server is triggered to harmonize the group photo preview image through the harmonization network model to obtain a harmonized group photo; the loss function used by the harmonization network model in the training process is the sum of the adversarial loss and the loss function including the perceptual loss, style loss and total change loss; the loss function including the perceptual loss, style loss and total change loss is the sum of the products of the perceptual loss, the style loss and the total change loss multiplied by the corresponding hyperparameters respectively; the perceptual loss is used to characterize the difference between the feature information in the feature space between the group photo prediction image obtained during the model training process and the target group photo in the training sample; the style loss is used to characterize the style difference between the group photo prediction image and the target group photo; the total change loss is used to characterize the difference between the pixel values ​​of adjacent pixels in the group photo prediction image; The generated harmonized photo is displayed.

2. The method according to claim 1, characterized in that The object cutout processing refers to a process of segmenting the image content corresponding to the cloud photo object from the object preview image.

3. The method according to claim 1, characterized in that After receiving the photo-taking instruction and before displaying the generated group photo, the method further includes: Performing a posture rationality assessment on the objects in the group photo preview image to obtain a posture rationality assessment result; Displaying posture adjustment prompt information according to the posture rationality evaluation result; the posture adjustment prompt information is used to prompt the cloud photo subject to adjust the posture; The real-time preview image of the group photo after the posture adjustment is displayed in real time.

4. The method according to claim 3, characterized in that The performing pose rationality assessment on the objects in the group photo preview image to obtain a pose rationality assessment result includes: Performing pose rationality evaluation on the object in the group photo preview image using a pre-trained pose rationality evaluation model to obtain a pose rationality evaluation result; The pose rationality assessment model is obtained by pre-training the model based on positive samples and negative samples; the positive samples are real group photos; and the negative samples are synthesized group photos.

5. The method according to claim 1, wherein The group photo is a harmonized group photo; after receiving the photo-taking instruction, triggering the beautification server to perform harmonization processing on the group photo preview image through the harmonization network model to obtain the harmonized group photo, including: After receiving the photo-taking instruction, the beautification server is triggered to perform harmonization processing on the foreground and background of the group photo preview image through the harmonization network model to obtain a harmonized group photo; The harmonious group photo generated by displaying includes: The harmonized group photo obtained from the beautification server is displayed.

6. The method according to claim 1, characterized in that The object preview image and the group photo preview image are both real-time images.

7. The method according to claim 1, characterized in that The harmonized group photo obtained by the harmonization process is an initial harmonized group photo; the method further includes: performing color migration on the initial harmonized group photo by the beautification server to align color attributes of facial regions of respective objects in the initial harmonized group photo to obtain a final harmonized group photo; The harmonious group photo generated by displaying includes: The final harmonized group photo obtained from the beautification server is displayed.

8. A cloud photo taking device, characterized in that: The device comprises: A display module is used to enter the cloud photo room according to the room credential of the cloud photo room; and to display in real time in the object area of ​​the cloud photo room the preview images of the objects corresponding to the terminals participating in the cloud photo; An image acquisition module is configured to acquire, in real time, a preview image of an object from the terminal during a cloud photo session; the terminal being one of the terminals participating in the cloud photo session; the preview image of the object being acquired by the terminal through real-time image acquisition of the corresponding cloud photo subject; the corresponding cloud photo subject being the subject of the cloud photo session using the terminal; A cutout module is used to perform object cutout processing on the object preview image obtained by the terminal to obtain a cutout result; The display module is further configured to display a preview image of the group photo in real time in the group photo preview area of ​​the cloud group photo room; the group photo preview image is obtained by synthesizing images based on the cutout results obtained by each terminal participating in the cloud group photo; after receiving a photo taking instruction, the beauty server is triggered to harmonize the group photo preview image through a harmonization network model to obtain a harmonized group photo; the loss function used by the harmonization network model during training is the sum of an adversarial loss and a loss function including a perceptual loss, a style loss, and a total change loss; the loss function including the perceptual loss, the style loss, and the total change loss is the sum of the products of the perceptual loss, the style loss, and the total change loss multiplied by corresponding hyperparameters respectively; the perceptual loss is used to characterize the difference in feature information in the feature space between the group photo prediction image obtained during model training and the target group photo in the training sample; the style loss is used to characterize the style difference between the group photo prediction image and the target group photo; the total change loss is used to characterize the difference between the pixel values ​​of adjacent pixels in the group photo prediction image; and the generated harmonized group photo is displayed.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images