A co-production method, electronic device and storage medium

By capturing depth images with a depth camera and performing fusion processing on the server side, the problem of unnatural composite photos in remote collaborative shooting is solved, generating realistic and natural collaborative images and improving the user experience.

CN117221712BActive Publication Date: 2024-11-08HONOR DEVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311472969.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-07
Publication Date
2024-11-08
Estimated Expiration
2043-11-07

AI Technical Summary

Technical Problem

Existing image compositing techniques struggle to generate natural and harmonious composite photos, especially in remote collaborative shooting scenarios where photos of users together often appear unnatural.

Method used

By using a depth camera to capture depth images and performing depth information fusion on the server side, a realistic and natural composite image is generated, including pose cues and background image processing.

Benefits of technology

It enables the generation of more realistic and natural composite photos in remote duet shooting scenarios, with user poses and backgrounds blending more harmoniously, thus improving the duet shooting experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117221712B_ABST
    Figure CN117221712B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a co-shooting method, an electronic device and a storage medium, which are applied to the field of terminal technology and can obtain a more realistic and natural co-shooting image by fusing the three-dimensional information of the captured image. The co-shooting method is applied to a first device, and the first device includes at least one depth camera. The method includes: after the first device establishes a remote co-shooting connection with at least one second device, a first interface is displayed, and the first interface is a shooting preview interface; in response to a first operation of the co-shooting initiator, a first captured image is sent to a server, and the first captured image is a depth image captured by at least one depth camera, and the first captured image includes a background image and a first portrait of the co-shooting initiator; a second interface is displayed, and the second interface includes a co-shooting image, and the co-shooting image is obtained by the server performing deep fusion of the depth information of the background image, the first portrait and the second portrait in the second captured image captured by at least one second device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to a synchronization method, an electronic device, and a storage medium. Background Art

[0002] At present, the camera function is a common function provided by electronic devices. Users can use electronic devices to make video calls, take photos or videos, etc. In daily life, friends, family or colleagues often need to take photos to commemorate special moments or share important moments. However, due to time, geographical distance or other restrictions, it is difficult for users to take photos with their relatives and friends at the same time and place, and it can often only be achieved through post-image synthesis technology.

[0003] However, existing image synthesis technologies (such as video call or photo synthesis technology) have many limitations, which make the characters in the synthesized photos stiff and unnatural, making it difficult to synthesize a smooth photo. Summary of the invention

[0004] The embodiments of the present application provide a combined shooting method, an electronic device and a storage medium. By fusing the depth information of the depth image, the obtained combined shot image is more realistic and natural.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a co-shooting method, which is applied to a first device, where the first device includes at least one depth camera, and the method includes:

[0007] After the first device establishes a remote co-shooting connection with at least one second device, a first interface is displayed, where the first interface is a shooting preview interface; in response to a first operation of the co-shooting initiator, a first captured image is sent to the server, where the first captured image is a depth image captured by at least one depth camera, and the first captured image includes a background image and a first portrait of the co-shooting initiator; a second interface is displayed, where the second interface includes a co-shooting image, where the co-shooting image is obtained by the server performing depth fusion of depth information of the background image, the first portrait, and the second portrait in a second captured image captured by at least one second device.

[0008] Exemplarily, the first device may be a device used by the co-shooting initiator, and the second device may be a device used by the co-shooting invitee. The second device may be one or more. For example, the co-shooting initiator may invite one friend to perform remote co-shooting, or may invite multiple friends to perform remote co-shooting at the same time.

[0009] Since the first captured image and the second captured image are depth images, each user in the combined image obtained by performing depth fusion of the first captured image and the second captured image appears more realistic and natural.

[0010] Optionally, in response to the first operation of the co-shooting initiator, before sending the first captured image to the server, the co-shooting method may further include:

[0011] A first depth image is sent to the server, where the first depth image is a preview image captured by at least one depth camera; after receiving posture prompt information from the server, a third interface is displayed.

[0012] The posture prompt information is used to prompt the photo pose of the initiator or the invitee of the photo shoot, and the posture prompt information is generated by the server according to the first depth image and the depth information of the second depth image collected by at least one second device.

[0013] It can be understood that when the first device and the second device are shooting a preview, they can send the preview image to the server, so that the server can predict the shooting posture of each co-shooting user based on the depth information of the preview image, so as to prompt each co-shooting user to shoot the image at the predicted position, so that the posture of each co-shooting user in the subsequent composite image is more harmonious and natural.

[0014] Optionally, before receiving the posture prompt information from the server, the above-mentioned co-shooting method may further include:

[0015] In response to the second operation of the co-production initiator, a co-production template request is sent to the server.

[0016] The combined template request is used by the server to generate posture prompt information according to the combined template, the first depth image and the second depth image corresponding to the combined template request.

[0017] It can be understood that the postures and positions of the users in different co-shooting templates are different, and the co-shooting initiator can choose the co-shooting template he wants. The server can generate posture prompt information based on the postures, positions and other information of each user in the co-shooting template and the depth information of the first depth image and the second depth image.

[0018] Optionally, after receiving the posture prompt information from the server and displaying the third interface, the above-mentioned co-shooting method may further include:

[0019] Sending a third depth image to the server, where the third depth image is a preview image captured by at least one depth camera after the photo-taking initiator adjusts the posture according to the posture prompt information;

[0020] A fourth interface is displayed, where the fourth interface includes a pre-captured image, where the pre-captured image is obtained by fusing the third depth image and a fourth depth image captured by at least one second device by the server.

[0021] It can be understood that before capturing the shot image, the first device can receive the pre-shot image generated by the server based on the preview image. If the initiator of the shot is not satisfied with the pre-shot image displayed in the first device, the initiator of the shot triggers the shooting operation after adjusting the shooting posture, position, expression, makeup, etc., thereby capturing a shot image that the user is more satisfied with.

[0022] Optionally, the background image of the combined image is the background of the first shot image or the background of the second shot image.

[0023] That is to say, the co-photographing image may be the background of the shooting environment where the co-photographing initiator is located, or may be the background of the shooting environment where the co-photographing invitee is located, and is not limited here.

[0024] Optionally, before responding to the first operation of the co-production initiator, the co-production method may further include:

[0025] In response to a third operation on the first control, a fourth interface is displayed, the first control is used to switch the background of the combined image, and the background of the preview image displayed in the fourth interface is different from the background in the first interface.

[0026] That is, a control for switching the background of the co-photographed image is displayed on the shooting interface of the first device, and the first device can switch the background of the co-photographed image in response to the operation of the first control by the co-photographing initiator.

[0027] Optionally, the camera parameters carried in the first captured image and the second captured image are the same, and the camera parameters include at least one of sensitivity, brightness, color, white balance parameter or exposure parameter.

[0028] It can be understood that the parameters of the cameras in the first device and the second device are the same, so that the styles of the co-shot users in the captured co-shot images are unified.

[0029] Optionally, the combined image is an image that has been beautified, and the beautification processing includes at least one of skin smoothing, face slimming, acne removal, enlarging eyes, beauty makeup, body shaping, whitening, wrinkle removal, hairdressing, and dark circle removal.

[0030] In one embodiment, the server may perform beautification processing on the combined image obtained by fusion to obtain a beautified combined image.

[0031] In another embodiment, after receiving the combined image returned by the server, the first device and / or the second device may first beautify the combined image and then save the beautified combined image.

[0032] In a second aspect, an embodiment of the present application provides another co-production method, which is applied to a server. The co-production method may include:

[0033] Receive a first captured image of a first device and a second captured image of at least one second device, wherein both the first device and the second device include at least one depth camera, the first captured image includes a background image and a first portrait of a co-photographing initiator, and the second captured image includes the background image and a second portrait of a co-photographing invitee; fuse the depth information corresponding to the background image, the first portrait, and the second portrait, respectively, to obtain a co-photographing image; and send the co-photographing image to the first device and the at least one second device.

[0034] It can be understood that the first captured image captured by the first device and the second captured image captured by the second device are both depth images. When the server fuses the depth images, it can adjust the posture, position, expression, etc. of the co-photographed user in combination with the depth information of the depth images to obtain a more realistic and natural co-photographed image.

[0035] Optionally, before receiving the first captured image of the first device and the second captured image of at least one second device, the combined capture method may further include:

[0036] Receiving a first depth image sent by a first device and a second depth image sent by at least one second device;

[0037] Generating posture prompt information according to the depth information of the first depth image and the depth information of the second depth image, where the posture prompt information is used to prompt the co-shooting initiator and the co-shooting invitee with shooting postures;

[0038] The gesture prompt information is sent to the first device or at least one second device.

[0039] It can be understood that the server can generate posture prompt information based on the depth information of the preview image to prompt each co-shooting user to shoot an image at the predicted position, so that the posture of each co-shooting user in the subsequent composite image is more harmonious and natural.

[0040] Optionally, generating posture prompt information according to the depth information of the first depth image and the depth information of the second depth image includes:

[0041] Receiving a synchronization template request sent by the first device or at least one second device;

[0042] The posture prompt information is generated according to the combined template, the first depth image and the second depth image corresponding to the combined template request.

[0043] It can be understood that the co-shooting user can first select a co-shooting template, and the server can generate posture prompt information based on the posture, position and other information of each user in the co-shooting template, and the depth information of the first depth image and the second depth image.

[0044] Optionally, before fusing the depth information corresponding to the background image, the first portrait, and the second portrait, the above-mentioned combined shooting method may further include:

[0045] The pose estimation algorithm is used to estimate the poses of the first and second portraits, and target poses and / or target positions of the first and second portraits are determined.

[0046] It can be understood that before fusing the combined image, the server can also adjust the posture, position, etc. of the combined user to make the combined image more natural.

[0047] Optionally, fusing the depth information corresponding to the background image, the first portrait, and the second portrait, respectively, includes:

[0048] Based on the target posture and / or target position of the first portrait and the second portrait, the depth information corresponding to the background image, the first portrait and the second portrait is fused.

[0049] Optionally, before receiving the first captured image of the first device and the second captured image of at least one second device, the method further includes:

[0050] Receiving a third depth image sent by the first device and a fourth depth image sent by at least one second device;

[0051] fusing the third depth image and the fourth depth image to obtain a pre-combined image;

[0052] The pre-captured image is sent to the first device and at least one second device.

[0053] It can be understood that the server generates a pre-shot image based on the preview image. If the initiator of the shot is not satisfied with the pre-shot image displayed in the first device, the initiator triggers the shooting operation after adjusting the shooting posture, position, expression, makeup, etc., thereby capturing a pre-shot image that the user is more satisfied with.

[0054] Optionally, the background image of the combined image is the background of the first shot image or the background of the second shot image.

[0055] Optionally, before fusing the depth information corresponding to the background image, the first portrait, and the second portrait to obtain the combined image, the method further includes:

[0056] Receive background switching request;

[0057] Switch the background image based on the background switching request.

[0058] Optionally, the camera parameters carried in the first captured image and the second captured image are the same, and the camera parameters include at least one of sensitivity, brightness, color, white balance parameter or exposure parameter.

[0059] Optionally, before sending the combined image to the first device and the at least one second device, the method further includes:

[0060] The coordinated images are beautified, and the beautification includes at least one of skin smoothing, face slimming, acne removal, eye enlargement, makeup, body shaping, whitening, wrinkle removal, hairdressing, and dark circle removal.

[0061] In a third aspect, the present application provides an electronic device comprising: at least one depth camera and a display screen; one or more processors; and a memory; wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform a combination shooting method as described in any one of the first aspects above.

[0062] In a fourth aspect, the present application provides a server comprising: one or more processors; a memory; wherein one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the processor, enable the server to execute the co-production method as described in any one of the second aspects above.

[0063] In a fifth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on an electronic device, the electronic device executes the co-production method as described in any one of the first aspects; when the instructions are executed on a server, the server executes the co-production method as described in any one of the second aspects.

[0064] In a sixth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are run on an electronic device, the electronic device executes the co-production method as described in any one of the first aspects; when the instructions are run on a server, the server executes the co-production method as described in any one of the second aspects.

[0065] It can be understood that the electronic device described in the third aspect, the server described in the fourth aspect, the computer storage medium described in the fifth aspect, and the computer program product described in the sixth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is an example diagram of a remote co-production of the related technology;

[0067] Figure 2 A schematic diagram of an application scenario of the co-production method provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0070] Figure 5 A system architecture diagram provided for an embodiment of the present application;

[0071] Figure 6 A schematic diagram of a process of identity authentication and joint establishment provided in an embodiment of the present application;

[0072] Figure 7 An example diagram of a co-production application login provided in an embodiment of the present application;

[0073] Figure 8 An example diagram of a co-production establishment provided in an embodiment of the present application;

[0074] Fig. 9 An example diagram of a timing prompt provided in an embodiment of the present application;

[0075] Fig.10 A schematic diagram of a process flow of a synchronization method provided in an embodiment of the present application;

[0076] Fig.11 An example diagram of a synchronized background switching provided in an embodiment of the present application;

[0077] Fig.12 Another example diagram of background switching provided in an embodiment of the present application;

[0078] Fig.13 This is another example diagram of a synchronized background switching provided in an embodiment of the present application;

[0079] Fig.14 An example diagram of selecting a matching template provided in an embodiment of the present application;

[0080] Fig.15 An example diagram showing a combined image provided by an embodiment of the present application;

[0081] Fig.16 Another example diagram showing a combined image provided by an embodiment of the present application;

[0082] Fig.17 A flowchart of another synchronization method provided in an embodiment of the present application;

[0083] Fig.18 An example diagram of a combination provided in an embodiment of the present application;

[0084] Fig.19 Another example diagram of a combination provided in an embodiment of the present application;

[0085] Fig. 20 An example diagram of a co-produced video provided in an embodiment of the present application;

[0086] Fig.21 An example diagram of switching cameras provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] The technical solution in the embodiment of the present application will be described below in conjunction with the drawings in the embodiment of the present application. In the description of the embodiment of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0088] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0089] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0090] The electronic device in the embodiment of the present application has one or more cameras and is installed with an application with a co-shooting function (referred to as a co-shooting application for short), which can realize functions such as remote co-shooting of photos, videos or live broadcasts with other users.

[0091] In a possible scenario, when multiple users use the co-shooting application in different electronic devices at different geographical locations to capture co-shooting images, each electronic device detects the triggering operation (e.g., click operation or touch operation) of the corresponding user on the shooting control, and in response to the triggering operation, each electronic device performs a photo-taking operation to capture a frame of image, and each electronic device sends the captured image to the server. The server synthesizes the images captured by the multiple electronic devices to obtain the co-shooting image, and then sends the co-shooting image to the electronic devices corresponding to the multiple users, so that the co-shooting image is displayed on the display interface of the co-shooting application of each electronic device.

[0092] Figure 1 This is an example diagram of remote co-production of the related technology. Figure 1 (a) shows that user 1 uses mobile phone 1 to capture image 1. Figure 1 (b) in FIG. 1 shows that user 2 uses mobile phone 2 to capture image 2. Mobile phone 1 and mobile phone 2 upload image 1 and image 2 to the server respectively. After receiving image 1 and image 2, the server synthesizes image 1 and image 2 to obtain a combined image, as shown in FIG. Figure 1 (c) shows the combined image.

[0093] Depend on Figure 1 As can be seen from the group-shot image shown in (c), after the server receives image 1 sent by mobile phone 1 and image 2 sent by mobile phone 2, it only performs a simple image stitching on image 1 and image 2, resulting in problems such as unnatural group photos of multiple users and a single shooting angle in the obtained group-shot image, which makes the experience of multiple users using the group-shot application to shoot group-shot images poor.

[0094] To this end, an embodiment of the present application provides a co-shooting method, in which a first device or a second device equipped with at least one depth camera collects a depth image and then performs depth fusion on the depth image. During image fusion, the posture, position, etc. of the co-shooting user in the image are adjusted to make the fused co-shooting image more realistic and natural.

[0095] Figure 2A schematic diagram of an application scenario of the co-shooting method provided in an embodiment of the present application, in which the application scenario includes: a first device 210, a second device 220, a third device 230, a fourth device 240 and a server 250. The first device 210 to the fourth device 240 can all communicate with the server 250. Among them, the owner of the first device 210 is user 1, the owner of the second device 220 is user 2, the owner of the third device 230 is user 3, and the owner of the fourth device 240 is user 4. The first device 210 to the fourth device 240 all have a shooting function, for example, the first device 210 to the fourth device 240 are all installed with a co-shooting application with a co-shooting function, or the first device 210 to the fourth device 240 can load a small program through an already installed application to call a co-shooting-related component, or the first device 210 to the fourth device 240 can realize remote co-shooting through a web version application of an application with a co-shooting function. Among them, the co-shooting application can be a system application of an electronic device, or it can be a third-party application downloaded by a user in a third-party application market, and this application is not specifically limited here.

[0096] Any device from the first device 210 to the fourth device 240 can be used as the initiator of remote co-shooting. For example, the first device 210 is the device that initiates remote co-shooting. The first device 210 responds to the operation of user 1 and determines other users to shoot with user 1. For example, user 1 can choose to shoot with user 2 and user 3. The first device 210 sends a co-shooting request to the second device 220 and the third device 230. After the second device 220 and the third device 230 receive the co-shooting request sent by the first device 210, the second device 220 and the third device 230 determine to shoot remotely with the first device 210. The first device 210, the second device 220 and the third device 230 respectively respond to the user's shooting operation and collect corresponding co-shooting image data.

[0097] It should be noted that the specific form of the co-shot image data is not limited in the embodiments of the present application. If the co-shot initiator initiates a co-shot photo request, then the co-shot image data is a co-shot image; if the co-shot initiator initiates a co-shot short video request, then the co-shot image data is a co-shot short video; if the co-shot initiator initiates a co-shot live broadcast request, then the co-shot image data is a co-shot live broadcast.

[0098] In addition, at least one of a time of flight (TOF) camera, a structured light camera or a binocular camera is disposed in each of the first device 210 to the fourth device 240. The image collected by the first device 210 to the fourth device 240 through at least one of the TOF camera, the structured light camera or the binocular camera is a depth image.

[0099] A depth image, also called a distance image, refers to an image that uses the distance from the image collector to each point in the scene as the pixel value.

[0100] The server 250 has an image processing function. For example, the server 250 can perform image segmentation processing on the images received from the first device 210 to the fourth device 240 to separate the users from the background image. The server 250 can also estimate the postures of users 1 to 5 and the layout of the group photo based on the location information and posture information of each user. The server 250 can also merge the foreground image of each user with the selected background image to obtain a group photo image.

[0101] by Figure 2 For example, user 1 is the initiator of the joint photo shoot, and users 2 to 4 are the invitees of the joint photo shoot. After users 1 to 4 upload the preview images to the server, the server can extract the background and foreground of each preview image to obtain the background image and the portraits of users 1 to 4. The server determines the layout of users 1 to 4 in the joint photo shoot template in response to the user's joint photo shoot template selection operation. The server can also determine the camera parameters of the second device, the third device, and the fourth device according to the camera parameters of the first device, and ensure that the camera parameters of the second device to the fourth device are consistent with the camera parameters of the first device, so as to ensure that the shooting style of each user in the joint photo shoot image remains unified.

[0102] After the first to fourth devices respond to the shooting requests of users 1 to 4, they upload the captured images 1 to 4 to the server. After receiving images 1 to 4, the server estimates the postures of the users in images 1 to 4 and performs image fusion to obtain the captured images.

[0103] It should be noted that, in addition, the above Figure 2 The number and type of electronic devices in the video are only used as an example, and the number of users who take photos together and the number and type of electronic devices are not limited in the embodiments of the present application. For example, when three users take photos together remotely, the three users can all use mobile phones, or the three users can all use phone watches, etc.

[0104] Figure 3 A schematic diagram of the structure of a server provided in an embodiment of the present application, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. This application does not limit this. The server is described in detail below.

[0105] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the server. Figure 3More or fewer components, or combining some components, or splitting some components, or different component arrangements. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware.

[0106] like Figure 3 As shown, the server may include a processor 310, a memory 320 and a communication module 330. The processor 310 may be used to read and execute computer-readable instructions. Specifically, the processor 310 may include a controller, an arithmetic unit and a register. Among them, the controller is mainly responsible for decoding instructions and issuing control signals for operations corresponding to the instructions. The arithmetic unit is mainly responsible for saving register operands and intermediate operation results temporarily stored during the execution of instructions. The register is a high-speed storage component with limited storage capacity, which can be used to temporarily store instructions, data and addresses.

[0107] The processor 310 may further include an image processing module 311. The image processing module 311 may be used to perform foreground segmentation processing, posture estimation, image fusion processing, etc. on the received depth image.

[0108] In a specific implementation, the hardware architecture of the processor 310 may be an application specific integrated circuit (ASIC) architecture, a microprocessor without interlocked piped stages (MIPS) architecture, an ARM (advanced RISC machines) architecture, or a network processor (NP) architecture, etc.

[0109] The memory 320 is coupled to the processor 310 and is used to store various software programs and / or multiple groups of instructions. In the embodiment of the present application, the synchronization method of the electronic device can be integrated into a processor of the server for implementation, or it can be stored in the memory of the server in the form of program code, and a processor of the server calls the code stored in the memory of the server to execute the above method. In a specific implementation, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 can store operating systems, such as uCOS, VxWorks, RTLinux and other embedded operating systems.

[0110] The communication module 330 can be used to establish communication between the server and other communication terminals (such as Figure 2The server can establish a communication connection between multiple devices in the network and send and receive data through the network. For example, when the electronic device is connected to the network, the server can establish a connection with the electronic device through the communication module 330 to facilitate the subsequent transmission of images. For example, when the electronic device captures an image, the server can receive the image reported by the electronic device.

[0111] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the server. In other embodiments, the server may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0112] Among them, the above-mentioned electronic device can be a mobile phone, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a smart watch, a netbook, a wearable electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a vehicle-mounted device, a smart car, and other electronic devices with a remote shooting function, and the embodiments of the present application do not impose any restrictions on this.

[0113] In addition, the above electronic device is provided with a three-dimensional camera to acquire depth image information. For example, the electronic device may be provided with a time of flight (TOF) camera or a binocular vision imaging camera, etc. The electronic device is also installed with an application with a co-shooting function (referred to as a co-shooting application), which can realize functions such as remote co-shooting of photos, videos or live broadcasts with other users.

[0114] like Figure 4 As shown, Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0115] The electronic device 400 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, and an air pressure sensor.

[0116] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 400. In other embodiments of the present application, the electronic device 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0117] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0118] The controller may be the nerve center and command center of the electronic device 400. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0119] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0120] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0121] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 400. In other embodiments of the present application, the electronic device 400 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0122] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0123] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0124] The wireless communication function of the electronic device 400 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0125] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 400 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0126] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 400. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0127] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 400. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0128] In some embodiments, the antenna 1 of the electronic device 400 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 400 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0129] The electronic device 400 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0130] Display screen 194 is used to display images, videos, etc.

[0131] The electronic device 400 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0132] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0133] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 400 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0134] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 400 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0135] Video codecs are used to compress or decompress digital videos. The electronic device 400 may support one or more video codecs. Thus, the electronic device 400 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0136] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of electronic device 400, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0137] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 400. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0138] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 400 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 400 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0139] The electronic device 400 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0140] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 400 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 400 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 400 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 400 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 400 and cannot be separated from the electronic device 400.

[0141] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the layered architecture as an example to exemplify the software structure of the electronic device.

[0142] Figure 5 A system architecture diagram provided for an embodiment of the present application.

[0143] It is understandable that the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. The software system of the first device and the second device may include an application layer (abbreviated as application layer), an application framework layer ( Figure 5 not shown), system libraries, and kernel layer.

[0144] like Figure 5 As shown, the application layer includes coordinated applications.

[0145] Among them, the co-shooting application is used to respond to the operation of the co-shooting user and realize the remote co-shooting function.

[0146] The system library includes image acquisition module and image transmission module.

[0147] The image acquisition module is used to acquire the image captured by the camera, and the image transmission module is used to upload the image captured by the camera to the server.

[0148] The kernel layer includes camera driver and display driver.

[0149] The camera driver is used to drive the camera to collect images in response to a user's trigger operation, and the display driver is used to drive the display screen to display a preview image or a combined image.

[0150] The server includes an image processing module, a posture estimation module and a parameter acquisition module.

[0151] Among them, the image processing module is used to extract the background of the images uploaded by the first device and the second device, and after obtaining the background image and the portrait of the co-photographed user, fuse the background image and the portrait of the co-photographed user to obtain the co-photographed image.

[0152] The posture estimation module is used to estimate the posture, position or angle of the user.

[0153] The parameter acquisition module is used to acquire the camera parameters of the device used to initiate the joint shooting, and then send the camera parameters of the device initiating the joint shooting to the device invited to the joint shooting.

[0154] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 4 Taking the electronic device of the structure shown as an example, the electronic device of the co-shooting initiator is taken as the first device, and the electronic device of the co-shooting invitee is taken as the second device to illustrate the co-shooting process. In the actual co-shooting scenario, the number of first devices and the number of second devices are not limited.

[0155] The first device and the second device may be devices of the same type or devices of different types. For example, the first device and the second device may both be mobile phones, or the first device may be a mobile phone and the second device may be a smart watch, and so on.

[0156] In the embodiment of the present application, before at least two users perform a remote joint shoot, both the joint shoot initiator and the joint shoot invitee need to log in to the joint shoot application and establish a joint shoot connection.

[0157] Figure 6 A flowchart of an identity verification and co-creation process provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the method may include steps 601 to 613.

[0158] S601, the first device sends login information to the server in response to a user's login operation.

[0159] For example, Figure 7 (a) in FIG. 1 shows a graphical user interface (GUI) of a mobile phone, which is a desktop 701 of the mobile phone. When the mobile phone detects that a user clicks an icon 702 of a co-production application on the desktop 701, the co-production application can be started and a display such as Figure 7 Another GUI shown in (b) of FIG. 700 may be referred to as a login interface 703. The login interface 703 displays input items of "user name" and "password", and the user may enter the registered login information (eg, user name and login password) in the input items of "user name" and "password".

[0160] It should be noted that Figure 7 The method of realizing the remote co-shooting function by logging into the co-shooting application already installed in the mobile phone is only an example. The mobile phone can also load the applet through the installed application to call the co-shooting related components to realize the remote co-shooting function, or the mobile phone can also realize the remote co-shooting function through the web version of the application with the co-shooting function. The specific form of the software for realizing the remote co-shooting function is not limited in the embodiments of the present application.

[0161] In the embodiment of the present application, after the mobile phone detects the user's trigger operation on the login control 704, in response to the trigger operation, the mobile phone sends the user's login information to the server.

[0162] S602: The server performs identity authentication based on the login information.

[0163] In an embodiment of the present application, after receiving the login information sent by the first device, the server matches the login information with the login information pre-stored in the server to determine the identity authentication result based on the matching result.

[0164] If the login information sent by the first device received by the server matches the pre-stored login information, the server determines the identity authentication result as successful authentication. If the login information sent by the first device received by the server does not match the pre-stored login information, the server determines the identity authentication result as failed authentication.

[0165] S603: The server returns an identity authentication result to the first device.

[0166] S604: The first device determines whether the login is successful based on the identity authentication result.

[0167] S605: The first device prompts the user that the login is successful.

[0168] S606: The first device prompts the user that the login has failed.

[0169] For example, when the server returns the authentication result to the first device as successful authentication, a prompt message may be displayed in the interface of the first device to prompt the user that the login is successful. Figure 7 As shown in (c) in the figure, the mobile phone interface displays a prompt message "Login successful".

[0170] When the server returns the authentication result to the first device as authentication failure, a prompt message may also be displayed in the interface of the first device to prompt the user that the login has failed. Figure 7 As shown in (d) in the figure, the mobile phone interface displays a prompt message "Login failed, please log in again".

[0171] It needs to be explained that Figure 7 The prompt information shown in (c) and (d) is only for exemplary description. The first device may also prompt the user of the identity authentication result by issuing a voice prompt, or the first device may also prompt the user of the identity authentication result by displaying other prompt information, which is not limited here.

[0172] It should be noted that the above-mentioned process of S601 to S606 for login verification of the co-shooting application is not only applicable to the first device of the co-shooting initiator, but also to the second device of the co-shooting invitee, and even other electronic devices installed with the co-shooting application. In the embodiment of the present application, the process of login verification of the second device of the co-shooting invitee is not described in detail.

[0173] S607: The first device sends a co-shooting request to the server in response to the selection operation of the co-shooting initiator.

[0174] The selection operation refers to the operation in which the co-shooting initiator selects the co-shooting invitee.

[0175] After the first device successfully logs in to the Hepai APP, the first device can add a Hepai contact in response to the selection operation of the Hepai initiator. For example, the first device can search for the registered user name of the contact to be added (for example, user A) in response to the operation of the Hepai initiator. After searching for the user name of the contact to be added, the first device sends an add request to the second device of user A through the server. After receiving the add request, the second device of user A sends an indication of consent to add to the first device through the server in response to the operation of user A. After the first device receives the indication of consent to add sent by the server, user A is successfully added as a Hepai friend. After that, user A is displayed in the Hepai contact list of the first device.

[0176] Then, the co-shooting initiator selects user A in the co-shooting contact list, and the first device sends a co-shooting request to the server in response to the selection operation of the co-shooting initiator.

[0177] The above is an introduction based on user A agreeing to add a compatible friend. Of course, user A can also refuse to agree to add a friend. In this way, the second device responds to the operation of user A and sends a refusal to add instruction to the first device through the server. After the first device receives the refusal to add instruction sent by the server, a prompt message can be displayed in the display interface of the first device to prompt user A to refuse the request to add a friend.

[0178] S608: The server sends a synchronization request to the second device.

[0179] S609: The second device sends a synchronization response message to the server.

[0180] After the server receives the co-shooting request sent by the first device, the server sends the co-shooting request to the second device. After the second device receives the co-shooting request, it can display prompt information on the display interface to prompt the co-shooting invitee whether to agree to the co-shooting.

[0181] For example, when the co-production initiator clicks Figure 8 After clicking the co-shooting control shown in (a) in the figure, in response to the click operation of the co-shooting initiator, the mobile phone can display the co-shooting contact list. Figure 8 (b) in the figure shows a list of co-photographing contacts of co-photographing friends added by the co-photographing initiator.

[0182] After the co-shooting initiator selects user D as the co-shooting invitee, a prompt message may be displayed on user D's device to prompt the user whether to agree to the co-shooting. Figure 8 As shown in (c) in FIG. 1 , it is assumed that the device of user D is a smart watch. After the smart watch receives the request for taking a photo together sent by the server, the display interface of the smart watch displays a reminder message "Reminder: ** invites you to take a photo together, do you agree?" If user D agrees to take a photo together, the smart watch responds to the response operation of user D and enters the photo-taking interface.

[0183] It should be noted that the embodiment of the present application does not limit the number of users for the co-shooting. For example, the co-shooting initiator can select only one user in the co-shooting contact list as the co-shooting invitee, or can select multiple users as the co-shooting invitees at the same time. Figure 8 As shown in (b) in the figure, the co-shooting initiator can select only user D as the co-shooting invitee. Figure 8 As shown in (d) in the figure, the co-shooting initiator can also select user C and user D as co-shooting invitees at the same time.

[0184] S610: The server returns synchronization response information to the first device.

[0185] S611: The first device determines, according to the co-shooting response information, whether the co-shooting invitee accepts the co-shooting request.

[0186] S612: The first device prompts that the photo-taking invitation is successful.

[0187] S613: The first device prompts that the photo-taking invitation fails.

[0188] In an embodiment of the present application, the second device sends a co-shooting response message to the server in response to the operation of the co-shooting invitee. After receiving the co-shooting response message, the server returns the co-shooting response message to the first device.

[0189] If the first device receives a response message indicating that the invitee agrees to the photo shoot, the first device prompts that the photo shoot invitation is successful. If the first device receives a response message indicating that the invitee refuses to the photo shoot, the first device prompts that the photo shoot invitation is failed.

[0190] For example, Figure 8 As shown in (c) of the figure, suppose user D clicks the "Yes" control. In response to user D's operation, the mobile phone can display Fig. 9 The prompt message shown in (a) in the figure is “Prompt: User D agrees to take a photo together, and the photo shoot is initiated successfully.” Assuming that user D clicks the “No” control, in response to user D’s operation, the mobile phone can display Fig. 9 The prompt message shown in (b) is “Prompt: User D refuses to take a photo together, and the photo shoot failed.”

[0191] In one scenario, the first device receives a co-shooting response message from the server to indicate that it agrees to co-shooting, and the interface of the first device can display the first preview interface. After the second device logs in to the co-shooting application, the second device displays the second preview interface. In the process of the first device and the second device displaying the preview interface, after the first device and the second device capture corresponding preview images, they upload the captured preview images to the server respectively. After the server receives the preview images captured by the first device and the second device, it fuses the preview images captured by the first device and the second device to obtain a fused co-shooting preview image. After the first device and the second device upload the captured images to the server, the server fuses the images captured by the first device and the second device to obtain the co-shooting image, and then returns the co-shooting image to the first device and the second device, so that the first device and the second device obtain the co-shooting image. The following takes the first device as the device initiating the co-shooting and the second device as the device invited to the co-shooting as an example, combined with Fig.10 The above-mentioned co-production process is introduced in detail. Fig.10 As shown, the synchronization method may include the following steps:

[0192] S1001: A first device sends a first depth image to a server in response to a shooting preview operation of a co-shooting initiator.

[0193] S1002: The second device sends a second depth image to the server in response to a shooting preview operation of the invitee.

[0194] The depth image refers to an image that uses the distance (depth) from the camera to each point in the scene as the pixel value.

[0195] Here, the first device and the second device are devices located in different geographical locations, such as location X of the first device and location Y of the second device.

[0196] After the first device displays the first preview interface, the first device sends the first depth image displayed in the first preview interface to the server. Similarly, after the second device displays the second preview interface, the second device sends the second depth image displayed in the second preview interface to the server.

[0197] For example, after the first device displays the first preview interface, the first device uses the front camera to capture the first depth image and sends the captured first depth image to the server. After the second device displays the second preview interface, the second device uses the rear camera to capture the second depth image and sends the captured second depth image to the server.

[0198] In the embodiment of the present application, when the first device and the second device display the preview interface, the preview interface of the second device may also be displayed in the first device, and the preview interface of the first device may also be displayed in the second device. That is, the preview interface can be shared between the first device and the second device in real time.

[0199] In some embodiments, after the first device and the second device establish a remote connection, the background image in the preview interface of the first device and the second device can be a background image captured by the camera of the first device, or a background image captured by the camera of the second device, so as to meet the shooting needs of the co-shooting user.

[0200] For example, Fig.11 As shown in (a) of FIG. 1 , the background image displayed in the first preview interface of the first device is the background image captured by the camera of the first device, such as Fig.11 As shown in (b), the background image displayed in the second preview interface of the second device is also the background image captured by the camera of the first device.

[0201] Furthermore, both the first device and the second device may switch the background of the combined image in response to a switching operation (one example of the third operation) by the user.

[0202] In the first case, the co-shooting initiator can switch the background image of the co-shooting image in the first preview interface of the first device. The first preview interface of the first device displays the selfie image of the co-shooting initiator, and the first device can switch the background image to the current background of any co-shooting invitee in response to the operation of the co-shooting initiator. When the first device switches the background image in the first preview interface to the background image captured by the camera of the second device in response to the operation of the co-shooting initiator, the background image in the second preview interface of the second device is also switched to the background image captured by the camera of the second device.

[0203] For example, Fig.12 As shown in (a) of FIG. 1 , a first preview interface of the co-shooting initiator is displayed in the first device. The first device can display a co-shooting contact list in response to the co-shooting initiator's operation on the "background switching" control, such as Fig.12 In response to the user's selection operation on user D, the first device can switch the shooting background to the background image captured by the camera of the second device corresponding to user D. The background image in the first preview interface of the first device is Fig.12 The background image shown in (a) is switched to Fig.12 The background image in the second preview interface of the second device is shown in (c). Fig.11 The background image shown in (b) is switched to Fig.12 The background image shown in (d).

[0204] In the second case, the invitee of the co-shooting can also switch the background image of the co-shooting image in the second preview interface of the second device. When the background image captured by the device of the co-shooting initiator is displayed in the second preview interface of the second device, the second device can switch the shooting background to the background image captured by the camera of the second device in response to the operation of the invitee of the co-shooting. Similarly, the first device also switches the background image in the first preview interface to the background image captured by the camera of the second device in response to the operation of the invitee of the co-shooting.

[0205] For example, Fig.13 As shown in (a) of FIG. 1 , the second device displays a selfie image of the co-photographing invitee and a background image captured by the camera of the first device. The second device can switch the background image to the background image captured by the camera of the second device in response to the co-photographing invitee's operation on the "background switching" control. The interface after the background switching is as shown in FIG. Fig.13 Similarly, in response to the operation of the "background switching" control by the invitee, the background image of the first device is changed to Fig.13 (c) in the switch to Fig.13 The background image shown in (d).

[0206] It needs to be explained that the above Fig.12 and Fig.13 The method of switching the shooting background is only described as an example. The specific background switching method can be determined according to the actual shooting situation and is not limited here. In addition, the background image displayed on the preview interface of the first device and the second device changes in real time and is not a fixed picture. The background image changes with the change of the position of the co-shooting user (co-shooting initiator or co-shooting invitee) or the movement of the electronic device of the co-shooting user.

[0207] When the first device and the second device collect the co-shot image, the first device and the second device may use the front camera to collect the image, or may use the rear camera to collect the image, or may use the wide-angle camera to collect the image, etc. In the embodiment of the present application, there is no limitation on the cameras used by the first device and the second device. For example, the first device uses the front camera to collect the image, and the second device uses the wide-angle camera to collect the image.

[0208] It should be noted that the camera used to capture images in the first device and the second device can be a TOF camera, a binocular vision camera or a structured light camera. In the embodiment of the present application, there is no limitation on the type of cameras in the first device and the second device. A camera that can capture depth information of an image can meet the solution of the present application.

[0209] S1003: The server performs background extraction on the first depth image to obtain a first background frame and a first subject image.

[0210] S1004: The server performs foreground extraction on the second depth image to obtain a second main image.

[0211] In an embodiment of the present application, after the server receives the first depth image sent by the first device and the second depth image sent by the second device, the server extracts the background of the first depth image to obtain a first background frame and a first subject image, and the server extracts the foreground of the second depth image to obtain a second background frame and a second subject image.

[0212] Optionally, the server may extract the first depth image and the second depth image using an image segmentation algorithm to obtain a corresponding background frame and a subject image. The image segmentation algorithm may be a semantic segmentation algorithm or an instance segmentation algorithm, and the like.

[0213] Among them, U-Net is a commonly used semantic segmentation algorithm with an encoder-decoder structure. It gradually reduces the image resolution and extracts features, and then gradually restores the resolution and performs segmentation prediction. U-Net has shown good performance in speech segmentation tasks. Mask R-CNN is an instance segmentation algorithm that further provides accurate segmentation of each instance based on object detection. Mask R-CNN combines a region proposal network and a fully convolutional network to accurately extract the foreground frame of each user in the image.

[0214] Optionally, the server may also input the first depth image into a pre-trained semantic segmentation model, and the semantic segmentation model outputs a first background frame and a first subject image. The semantic segmentation model is trained using a large number of training samples and has the ability to extract background information and foreground information of an image. Similarly, the server may also input the second depth image into a pre-trained instance segmentation model, and the instance segmentation model outputs a second background frame and a second subject image.

[0215] The method for processing the first depth image and the second depth image by the above-mentioned server is only described as an example. Other methods for extracting the foreground or background of the depth image are also applicable to the embodiments of the present application and are not introduced one by one here.

[0216] S1005: The first device sends a request for a co-production template in response to an operation of the co-production initiator.

[0217] In an embodiment of the present application, a plurality of synchronization templates are pre-stored in the server, and the first device may send a synchronization template request to the server in response to a synchronization template selection operation (an example of the second operation) of the synchronization initiator.

[0218] For example, Fig.14As shown in (a) of FIG. 1 , a “collaboration template” control is displayed in the first preview interface of the first device. In response to the triggering operation of the “collaboration template” control by the collaborator, the first device displays a collaborating gesture, such as Fig.14 As shown in (b) in FIG. 1 , the first device sends a request for a joint shooting template to the server in response to the operation of the "standing side by side" control by the joint shooting initiator. The joint shooting postures in the joint shooting template may include hugging, holding hands, kissing, standing side by side, etc.

[0219] S1006: The server obtains depth information of the first main image and the second main image.

[0220] The depth information refers to the three-dimensional coordinate value of each pixel in the image.

[0221] In an embodiment of the present application, the first depth image captured by the camera of the first device includes depth information. The second depth image captured by the camera of the second device also includes depth information. The server can obtain the depth information of the first subject image and the second subject image.

[0222] It should be noted that in the embodiment of the present application, there is no limitation on the execution order of the above-mentioned step 1005 and step 1006. Step 1005 may be executed first and then step 1006, or step 1006 may be executed first and then step 1005, or step 1005 and step 1006 may be executed simultaneously.

[0223] S1007: The server estimates the posture of the user based on the matching template request and the depth information, and determines the posture frame of each user.

[0224] In the embodiment of the present application, after receiving the synch template request sent by the first device, the server determines the target synch template according to the synch template request, wherein the synch template may include the predicted posture of the synch user.

[0225] In some embodiments, the server may input the first subject image and the second subject image into a trained posture estimation model, and determine posture estimation results of each user to be photographed according to the output of the posture estimation model. The posture estimation model is trained with a large number of samples and has the ability to predict user postures.

[0226] In other embodiments, the server may also directly input the first depth image and the second depth image into a trained posture estimation model, respectively, and determine the posture estimation results of each user to be photographed according to the output of the posture estimation model.

[0227] Common pose estimation algorithms include OpenPose, PoseNet, etc. Among them, OpenPose is a commonly used pose estimation algorithm that can detect the key points of the human body, including the head, arms, torso and other parts, and estimate the spatial structure of the pose. OpenPose has shown good performance in multi-person pose estimation and complex backgrounds.

[0228] PoseNet is a lightweight pose estimation algorithm suitable for mobile devices and real-time applications. It uses a convolutional neural network to process the input image and output the coordinates of key points. PoseNet can estimate the user's pose in real time and is suitable for the real-time needs of remote group photo shooting.

[0229] In the embodiment of the present application, after the server determines the posture estimation result of the user to be photographed, it can determine the corresponding posture frame according to the posture estimation result of the user to be photographed.

[0230] S1008: The server sends a gesture frame of the initiator of the photo-taking to the first device.

[0231] S1009: The server sends a pose frame of the invitee to the second device.

[0232] Exemplarily, after the server determines the predicted posture of the co-shooting user according to the target co-shooting template, it sends the posture frame of the co-shooting initiator to the first device and sends the posture frame of the co-shooting invitee to the second device. Fig.14 The dotted frame shown in (c) is the pose frame of the initiator of the photo-taking. Fig.14 The dotted frame shown in (d) is the pose frame of the invited person.

[0233] After the first device receives the pose frame of the co-shooting initiator sent by the server, the pose frame is displayed in the preview interface of the first device. The co-shooting initiator can adjust the pose and / or position of the photo according to the pose frame displayed in the first device.

[0234] Similarly, after the second device receives the pose frame of the invitee sent by the server, the pose frame is displayed in the preview interface of the second device. The invitee can adjust the pose and / or position of the photo according to the pose frame displayed in the second device.

[0235] In some embodiments, when the server sends a pose frame to the first device or the second device, the server may also send prompt information to the first device and the second device to prompt the user to take a photo with a pose or a position. Fig.14 As shown in (d) in the figure, when the server sends the pose frame of the invitee to the second device, it also sends the prompt information "turn sideways and clasp your fists". After the second device receives the pose frame of the invitee and the prompt information sent by the server, it can adjust the photo pose and photo position.

[0236] It needs to be explained that the above Fig.14 In the description, the server sending a prompt message to the second device is used as an exemplary introduction. The server can also send a voice prompt message to the first device or the second device to prompt the initiator of the co-photographing or the invitee of the co-photographing to adjust the photo posture and photo position.

[0237] S1010: The first device sends a first captured image to a server in response to a capturing operation.

[0238] S1011: The second device sends a second captured image to the server in response to a capturing operation.

[0239] In an embodiment of the present application, after the initiator of the co-shoot determines the photo posture and photo position, the initiator of the co-shoot triggers the shooting control. The first device responds to the triggering operation of the shooting control by the initiator of the co-shoot (an example of the first operation), and after collecting the first shot image, the first device sends the first shot image to the server.

[0240] Similarly, after the invitee determines the photo pose and location, the invitee triggers the shooting control. The second device responds to the triggering operation of the shooting control by the invitee, and after collecting the second shot image, the second device sends the second shot image to the server.

[0241] In some embodiments, in order to improve the beautification of the co-photographed image, before the co-photographing initiator triggers the shooting control, the co-photographing initiator may enable the intelligent optimization function so that the first device can capture the beautified first shot image. Fig.14 As shown in (c) in FIG. 1 , after the co-shooting initiator determines the photo pose and photo position, the co-shooting initiator first triggers the intelligent optimization control 1410 and then triggers the shooting control to acquire the beautified first shot image. Fig.14 As shown in (d) in FIG. 1 , after the invited person determines the photo pose and photo position, the invited person first triggers the intelligent optimization control 1420 and then triggers the shooting control to acquire the beautified second shot image.

[0242] In an embodiment of the present application, in order to ensure that the shooting styles of each user in the co-shot image are unified, before the first device and the second device capture images, the co-shot user can adjust the camera parameters of the first device to be consistent with the camera parameters of the second device. Optionally, the server can determine the camera parameters of the first device based on the preview stream received from the first device, and determine the camera parameters of the second device based on the preview stream of the second device. The server determines whether the camera parameters of the first device are the same as the camera parameters of the second device to determine whether to adjust the camera parameters of the first device or the second device.

[0243] In one case, assuming that the background image captured by the camera of the first device is the background of the co-photographed image, if the server determines that the camera parameters of the first device are different from the camera parameters of the second device, the server can send the camera parameters of the first device to the second device. After the second device receives the camera parameters of the first device sent by the server, the second device adjusts the camera parameters of the second device according to the camera parameters of the first device in response to the adjustment operation of the co-photographed invitee, so that the adjusted camera parameters of the second device are consistent with the camera parameters of the first device.

[0244] The above-mentioned camera parameters include but are not limited to sensitivity, brightness, color, white balance parameters, exposure parameters, etc. Therefore, by adjusting the camera parameters of the second device, the imaging effect of the image captured by the second device is consistent with the imaging effect of the image captured by the first device, ensuring the uniformity of the styles of each user in the combined image captured by the first device and the second device.

[0245] For example, it is assumed that the server determines that the sensitivity value of the camera parameters of the first device is 200 and the brightness value is 100 according to the preview stream of the first device, and the server determines that the sensitivity value of the camera parameters of the second device is 180 and the brightness value is 120. The server can send the camera parameters of the first device to the second device, and after the second device receives the camera parameters of the first device, in response to the user's adjustment operation, the sensitivity value of the camera parameters of the second device is adjusted from 180 to 200, and the brightness value is adjusted from 120 to 100.

[0246] It should be explained that the above is an exemplary explanation of adjusting the camera parameters of the second device according to the camera parameters of the first device. Assuming that the background image captured by the camera of the second device is the background of the combined image, when the server determines that the camera parameters of the first device are different from the camera parameters of the second device, the camera parameters of the proposed device can be adjusted according to the camera parameters of the second device, so that the adjusted camera parameters of the first device are consistent with the camera parameters of the second device.

[0247] S1012: The server performs image fusion on the first captured image and the second captured image to obtain a combined image.

[0248] In an embodiment of the present application, after the server receives the first captured image sent by the first device and the second captured image sent by the second device, the server performs image fusion on the first captured image and the second captured image to obtain a fused combined image.

[0249] In one embodiment, the server can perform foreground extraction on the first captured image and the second captured image to obtain a background image and two foreground images including a portrait. The server inputs the background image and the foreground image into a trained deep fusion network, and the deep fusion network outputs a fused combined image.

[0250] The process of the server training the deep fusion network is as follows:

[0251] (1) Training set preparation. The server collects pairs of images, which include a foreground image and a corresponding background image. At the same time, the server generates a mask image corresponding to each foreground image to mark the foreground area.

[0252] (2) Construct a deep fusion network. The server can use a network structure such as a convolutional neural network or a generative adversarial network as the network architecture of the deep fusion network. The deep fusion network accepts a foreground image, a background image, and a mask image as input and generates a fused composite image as output.

[0253] (3) Training process. The server uses the foreground image, background image, and mask image in the training set as input and the real matching image as the target to train the deep fusion network. During the training process, the network parameters are optimized to minimize the difference between the generated image and the real matching image.

[0254] The above-mentioned server uses a trained deep fusion network to fuse the first captured image and the second captured image only as an exemplary description. The server can also use other image fusion methods to fuse the first captured image and the second captured image, which is not limited here.

[0255] In another embodiment, when the server fuses the first captured image and the second captured image, the server may first decompose the first captured image and the second captured image into a basic part and detailed content; then, the basic part and the detailed content are fused separately, that is, the basic part is fused by weighted averaging, and for the detailed content, a deep learning network is used to extract multi-layer features, and the extracted features are used to generate multiple candidates for fused detailed content using the L1 norm and weighted averaging strategy. Once these candidate objects are obtained, the maximum selection strategy is used to obtain the final fused detailed content; finally, the fused basic part is combined with the detailed content, and the image is reconstructed to obtain a combined image.

[0256] When there are multiple users invited to take a photo together, when the server fuses the first captured image and the multiple second captured images, it can first fuse the first captured image with any one of the second captured images, and then continue to fuse the fused image with one of the second captured images until the first captured image is completely fused with the multiple second captured images to obtain a fused combined image.

[0257] In some embodiments, before fusing the first captured image and the second captured image, the server may also estimate the posture of the co-shooting user to obtain the posture estimation results of each co-shooting user. The posture estimation results include the posture, relative position and angle of the co-shooting user. Then, the server adjusts the posture, position or angle of each co-shooting user according to the posture estimation results of each co-shooting user. Thus, by adjusting the posture, relative position or angle of the co-shooting user, the adjusted co-shooting user becomes more natural.

[0258] Exemplarily, assuming that there are two users taking the photo together, the server can identify the users to determine information such as the gender and age of the users. Assuming that the server determines that the two users taking the photo together are male and female respectively, the server can estimate that the posture estimation result of the two users taking the photo together is holding hands.

[0259] As another example, suppose a father and son are taking a photo together remotely. Fig.15 As shown in (a) of FIG. 1 , after the first device acquires the first captured image of the father, the first device uploads the first captured image to the server. Fig.15 As shown in (c) in the figure, after the second device acquires the second image of the son, the second device uploads the second image to the server. After the server receives the first image and the second image, when fusing the first image and the second image, the server adjusts the posture and angle of the son, making the posture of the father and son more natural. Fig.15 The first device in (b) shows a photo taken of a father and a son. Fig.15 The second device in (d) also shows a joint image of the father and the son.

[0260] For example, assuming that there are four users taking a photo together, and the server recognizes that the gestures of three of the users taking a photo together are "V-sign", and the gesture of one user taking a photo together is not "V-sign". The server can perform gesture estimation on the gestures of the users taking a photo together, and adjust the gesture of the users taking a photo together to "V-sign" according to the gesture estimation result.

[0261] In some embodiments, the server may perform beautification processing on the combined image obtained by fusion. For example, the server may perform beautification processing on the combined user in the combined image, such as skin smoothing, face thinning, acne removal, eye enlargement, makeup, body shaping, whitening, wrinkle removal, hairdressing, and dark circle removal. Thus, the combined image taken by the combined user is more beautiful.

[0262] Optionally, after obtaining the fused composite image, the server can identify the user information of the composite users, so as to perform beautification operations on each composite user according to the identified user information. For example, the composite users identified by the server are an elderly person, a middle-aged male, and an infant. When the server beautifies each composite user, the beautification operations on the elderly person, the middle-aged male, and the infant are different. The server performs beautification operations such as wrinkle removal and skin smoothing on the elderly person, and performs beautification operations such as body shaping and whitening on the middle-aged male, but does not need to perform beautification operations on the infant.

[0263] Optionally, the server may also perform beautification operations on the co-photographing users according to the scene information of the co-photographing background image. For example, if the server recognizes that the scene information is dusk, the server may perform facial lighting processing on each co-photographing user so that each co-photographing user and the background in the processed co-photographing image are more naturally integrated.

[0264] Optionally, the server may also perform a three-dimensional reconstruction of the human body of the co-photographing user. For example, assuming that the co-photographing user A is a person with a leg disability (missing the left leg), the server may reconstruct the left leg of the co-photographing user A during the process of fusing the captured images of the co-photographing user, so that the co-photographing user A appears to be a normal person in the fused co-photographing image. Thus, the co-photographing image can better meet the user's photography needs.

[0265] Optionally, the server may also adjust the facial features of the co-photographed user in the co-photographed image so that the facial features of the co-photographed user after the adjustment are more natural. For example, assuming that the co-photographed user 1 is in a closed-eye state in the captured image 1 acquired by the device, when the server fuses the images, the eyes of the co-photographed user 1 may be adjusted so that the eyes of the co-photographed user 1 in the fused co-photographed image are in a normal open-eye state. For another example, the server may adjust the expressions of each co-photographed user in the co-photographed image so that the expressions of each co-photographed user after the adjustment are more natural.

[0266] S1013: The server returns the combined image to the first device.

[0267] S1014: The server returns the combined image to the second device.

[0268] For example, Fig.14 As shown in (c) and (d) in FIG. 1 , after the first device sends the first captured image to the server and the second device sends the second captured image to the server, the server can fuse the foreground image and the background image of the first captured image and the second captured image to obtain a fused composite image. The server can return the composite image to the first device and the second device respectively. After the first device receives the composite image, it displays the composite image on the display interface of the first device, such as Fig.16Similarly, after receiving the combined image, the second device displays the combined image on the display interface of the second device, as shown in (a) of FIG. Fig.16 As shown in (b) in .

[0269] After the first device receives the group-shooting image, in response to the operation of the group-shooting initiator, the first device can edit, share or save the group-shooting image. Similarly, after the second device receives the group-shooting image, in response to the operation of the group-shooting invitee, the second device can save, share or edit the group-shooting image.

[0270] In one embodiment, before the first device saves the co-shooting image in response to the save operation of the co-shooting initiator, the first device may also edit the co-shooting image in response to the edit operation of the co-shooting initiator, and after the editing is completed, save the edited co-shooting image in response to the save operation of the co-shooting initiator.

[0271] Optionally, after the first device receives the co-taken image sent by the server, the co-taken initiator can edit the co-taken image. The editing operation includes size adjustment, position adjustment, beautification processing, etc. Then, the first device saves the edited co-taken image in response to the save operation of the co-taken initiator.

[0272] For example, assuming that after the first device receives the group-shot image, the first device can respond to the operation of the group-shot initiator and adjust the expression and body shape of the group-shot initiator to obtain an adjusted group-shot image, so that the adjusted group-shot image better meets the requirements of the group-shot initiator.

[0273] It should be noted that after the first device and the second device receive the co-shot image sent by the server, the first device and the second device may both edit the co-shot image, or the first device edits the co-shot image and the second device does not edit the co-shot image, or the second device edits the co-shot image and the first device does not edit the co-shot image. In other words, the co-shooting initiator and the co-shooting invitee can determine whether to edit the co-shot image based on their own needs. In another remote co-shooting method, the server can fuse the first background frame, the first subject image and the second subject image, and then return the fused pre-shot image to the first device and the second device. If the co-shooting initiator or the co-shooting invitee is not satisfied with the fused pre-shot image, the co-shooting initiator and the co-shooting invitee can adjust the shooting posture or angle and then trigger the shooting operation to obtain a co-shot image that better meets the user's requirements. The following is combined with Fig.17 The above remote co-production process is introduced in detail. Fig.17 As shown, the remote co-production process may include the following steps:

[0274] S1701: The first device sends a first depth image to a server in response to a shooting preview operation of a co-shooting initiator.

[0275] S1702: The second device sends a second depth image to the server in response to a shooting preview operation of the invitee.

[0276] S1703: The server performs background extraction on the first depth image to obtain a first background frame and a first subject image.

[0277] S1704: The server performs foreground extraction on the second depth image to obtain a second main image.

[0278] S1705: The first device sends a request for a co-production template in response to the operation of the co-production initiator.

[0279] S1706: The server obtains depth information of the first subject image and the second subject image.

[0280] S1707: The server estimates the posture of the user based on the matching template request and the depth information, and determines the posture frame of each user.

[0281] S1708: The server sends a gesture frame of the initiator of the photo-taking to the first device.

[0282] S1709: The server sends a pose frame of the invitee to the second device.

[0283] The implementation process of the above S1701 to S1709 can refer to the implementation process of the above S1001 to S1009, which will not be repeated here.

[0284] S1710: The first device sends a third depth image to the server. Correspondingly, the server receives the third depth image.

[0285] S1711: The second device sends a fourth depth image to the server. Correspondingly, the server receives the fourth depth image.

[0286] In an embodiment of the present application, after the server sends the pose frame of the co-shooting initiator to the first device, and the co-shooting initiator adjusts the shooting pose according to the pose frame, the first device acquires the third depth image and sends the third depth image to the server.

[0287] Similarly, after the server sends the pose frame of the invitee to the second device, and the invitee adjusts the shooting pose according to the pose frame, the second device acquires the fourth depth image and sends the fourth depth image to the server.

[0288] S1712: The server performs background extraction on the third depth image to obtain a third background frame and a third main image.

[0289] S1713: The server performs background extraction on the fourth depth image to obtain a fourth main image.

[0290] S1714: The server fuses the third background frame, the third subject image, and the fourth subject image to obtain a pre-shot image.

[0291] The method for the server to perform background extraction on the third depth image and the fourth depth image here refers to the process of the server performing background extraction on the first depth image and the second depth image, which will not be repeated here. The process of the server performing image fusion to obtain the pre-shot image also refers to the above-mentioned image fusion process, which will not be repeated here.

[0292] S1715: The server sends the pre-captured image to the first device.

[0293] S1716: The server sends the pre-captured image to the second device.

[0294] It can be understood that the server performs image fusion on the preview images captured by the first device and the second device to obtain a pre-shot image, and the server sends the pre-shot image to the first device and the second device.

[0295] If the initiator of the co-shooting is not satisfied with the pre-co-shooting image received by the first device, the initiator of the co-shooting can adjust the shooting posture, shooting angle, expression, etc. Similarly, if the invitee of the co-shooting is not satisfied with the pre-co-shooting image received by the second device, the invitee of the co-shooting can also adjust the shooting posture, shooting angle, expression, etc.

[0296] For example, Fig.18 An example diagram of a father and son remotely co-produced is shown. Fig.18 As shown in (a) of FIG. 1 , the first device collects the third depth image and sends the third depth image to the server. The server fuses the preview images of the first device and the second device to obtain the pre-combined image, and then returns the pre-combined image to the first device and the second device. Fig.18 In (b), the shooting preview interface of the first device displays a pre-shot image. The father is not satisfied with the pre-shot image, and adjusts the shooting position or angle, for example, the father adjusts the side face position, the relative position with his son, and the curvature of the mouth corners.

[0297] S1717: The first device sends a first captured image to the server in response to the capturing operation.

[0298] S1718: The second device sends a second captured image to the server in response to the capturing operation.

[0299] After the co-shooting initiator adjusts his / her shooting posture, angle or expression, the co-shooting initiator triggers the shooting function, and the first device responds to the shooting operation, collects the first shot image, and then sends the first shot image to the server.

[0300] Similarly, after the invitee adjusts his / her shooting posture, angle or expression, the invitee triggers the shooting function, and the second device responds to the shooting operation, collects the second shot image, and sends the second shot image to the server.

[0301] S1719: The server performs image fusion on the first captured image and the second captured image to obtain a combined image.

[0302] S1720: The server returns the combined image to the first device.

[0303] S1721: The server returns the combined image to the second device.

[0304] Continue with Fig.18 As an example, after the father adjusts his posture, a shooting operation is triggered. In response to the shooting operation, the first device sends the first shot image to the server. Similarly, the second device sends the second shot image to the server. The server fuses the first shot image and the second shot image to obtain a combined image. The server returns the combined image to the first device and the second device. Fig.18 In (c), the first device displays a synchronized image. Fig.18 In (d), the second device displays a combined image.

[0305] By comparing Fig.18 As can be seen from (b) and (c) in the figure, after the father adjusted the shooting angle and posture, the composite image fused by the server became more natural. The server also adjusted the position of the composite image relative to the camera, making the fusion of the adjusted father-son photo and the background image more realistic and natural.

[0306] The implementation process of the above S1717 to S1721 can refer to the implementation process of the above S1010 to S1014, which will not be repeated here.

[0307] Thus, the server deeply fuses the preview images of the first device and the second device, and returns the fused pre-taken image to the first device and the second device, so that the taker can see the pre-taken image in the preview interface. If the taker is not satisfied with the pre-taken image, he can adjust the shooting angle, shooting posture, etc., so that the final taken image meets the user's needs. In addition, the taker can collect a real and natural taken image through a single shooting operation.

[0308] For example, Fig.19 As shown, Fig.19 A family of four was filmed remotely. Fig.19 In (a), the father is the initiator of the joint photo shoot, that is, the first device is the joint photo shoot initiator, and the mother, sister and brother are the invited persons of the joint photo shoot. The background captured by the first device is the joint photo shoot background. The preview image captured by the first device is shown in FIG. Fig.19 As shown in (a) in FIG. 1 , the preview images collected by the three second devices are respectively as follows: Fig.19 After the first device and the three second devices send the preview images to the server, the server extracts the foreground of the received preview images and obtains the portraits of the four users and the background image. Then, the server fuses the portraits of the four users and the background image to obtain the pre-taken image. The server returns the pre-taken image to the first device and the three second devices. Fig.19 (e) in FIG. 1 shows the pre-taken image received by any device. Assuming that the taker is not satisfied with the pre-taken image, the taker can reselect the take template. After the taker re-determines the take template, the first device and the three second devices respectively respond to the take operation of the taker and upload the taken images to the server.

[0309] After receiving the captured images uploaded by the four devices, the server estimates the posture of each co-photographing user, determines the posture estimation result of each co-photographing user, and then fuses the captured images to obtain the co-photographing image. The server returns the co-photographing image to the first device and the three second devices. Fig.19 (f) in the figure shows the combined image received by any one of the devices.

[0310] In addition, it should be noted that after the server receives the captured images sent by the first device and the three second devices, the server identifies the co-photographing user to determine the co-photographing user's gender, height, body shape and other user information. The server can estimate the co-photographing user's position, posture and the like based on the user information, such as Fig.19 As shown in (f) in the figure, the server determines that the father and mother are located behind the daughter and son. In addition, the server also adjusts the positions of the users and the background image. For example, the server moves the positions of the four users away from the seaside to make the fused image more realistic and natural.

[0311] It needs to be explained that Fig.19 In the figure, three co-shooting invitees are used as an example for explanation, and the number of co-shooting invitees is not limited in the embodiments of the present application.

[0312] In the above embodiment, the co-shooting method of the present application is illustrated by the co-shooting users taking photos together, but the co-shooting users can also take videos together, shoot live broadcasts together, etc., which is not limited in the embodiments of the present application.

[0313] For example, Fig. 20 As shown, the co-produced user can receive the co-produced video returned by the server.

[0314] In the embodiment of the present application, the first device and the second device can use a front camera, a rear camera or a wide-angle camera to shoot, so as to acquire images at different viewing angles.

[0315] For example, Fig.21 (a) in the figure is an image captured by the front camera of the first device. Fig.21 (b) in the figure is an image captured by the wide-angle camera of the first device.

[0316] It is understandable that, in order to realize the above functions, the above-mentioned electronic devices, etc. include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present invention.

[0317] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device etc. according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0318] In the case of dividing each functional module according to each function, a possible composition diagram of the electronic device involved in the above embodiment, the electronic device may include: a display unit, a transmission unit and a processing unit, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0319] The embodiment of the present application also provides an electronic device, including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the above-mentioned related method steps to implement the synchronization method in the above-mentioned embodiment.

[0320] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the synchronization method in the above-mentioned embodiment.

[0321] An embodiment of the present application further provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the synchronization method in the above-mentioned embodiment.

[0322] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the device executes the synchronization method executed by the electronic device in the above-mentioned method embodiments.

[0323] Among them, the electronic device, computer-readable storage medium, computer program product or device provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0324] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0325] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0326] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0327] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A synchronization method, characterized in that: Applied to a first device, the first device comprising at least one depth camera, the method comprising: After the first device establishes a remote joint shooting connection with at least one second device, a first interface is displayed, where the first interface is a shooting preview interface; Sending a first depth image to a server, where the first depth image is a preview image captured by the at least one depth camera; In response to a second operation of the co-shooting initiator, sending a co-shooting template request to the server, wherein the co-shooting template request is used by the server to generate posture prompt information according to the co-shooting template corresponding to the co-shooting template request, the first depth image, and depth information of a second depth image collected by at least one second device; After receiving the posture prompt information from the server, displaying a third interface, wherein the posture prompt information is used to prompt the photo pose of the initiator or the invitee of the joint photo shoot; Sending a third depth image to the server, where the third depth image is a preview image captured by the at least one depth camera after the photo-taking initiator adjusts the posture according to the posture prompt information; Displaying a fourth interface, wherein the fourth interface includes a pre-captured image, wherein the pre-captured image is obtained by fusing the third depth image and a fourth depth image collected by at least one second device by the server; In response to a first operation of the co-shooting initiator, sending a first captured image to the server, where the first captured image is a depth image captured by the at least one depth camera, and the first captured image includes a background image and a first portrait of the co-shooting initiator; A second interface is displayed, wherein the second interface includes a combined image, wherein the combined image is obtained by the server estimating the posture of the first portrait and the second portrait in the second captured image collected by at least one second device, and determining the target posture and / or target position of the first portrait and the second portrait, and the second captured image also includes a background image; based on the target posture and / or target position of the first portrait and the second portrait, the depth information of the background image, the first portrait and the second portrait is deeply fused, the combined image is obtained by the server identifying the user information of the first portrait and / or the second portrait, and beautifying the first portrait and / or the second portrait according to the identified user information, and / or the combined image is an image obtained by the server performing three-dimensional construction on the first portrait and / or the second portrait, and the background image of the combined image is the background of the first captured image or the background of the second captured image.

2. The method according to claim 1, characterized in that Before responding to the first operation of the co-production initiator, the method further includes: In response to a third operation on the first control, a fourth interface is displayed, wherein the first control is used to switch the background of the combined image, and the background of the preview image displayed in the fourth interface is different from that in the first interface.

3. The method according to claim 1, characterized in that The camera parameters carried in the first captured image and the second captured image are the same, and the camera parameters include at least one of sensitivity, brightness, color, white balance parameter or exposure parameter.

4. The method according to claim 1, characterized in that: The combined image is an image that has been beautified, and the beautification includes at least one of skin smoothing, face slimming, acne removal, eye enlargement, makeup, body shaping, whitening, wrinkle removal, hairdressing, and dark circle removal.

5. A synchronization method, characterized in that: Applied to a server, the method comprises: Receiving a first depth image sent by a first device and a second depth image sent by at least one second device; Generating posture prompt information according to the depth information of the first depth image and the depth information of the second depth image; the generating posture prompt information according to the depth information of the first depth image and the depth information of the second depth image includes: Receiving a synchronization template request sent by the first device or at least one second device; Generate the posture prompt information according to the combined template corresponding to the combined template request, the first depth image, and the depth information of the second depth image; Sending the posture prompt information to the first device or the at least one second device; Receiving a third depth image sent by the first device and a fourth depth image sent by the at least one second device; fusing the third depth image and the fourth depth image to obtain a pre-taken image; Sending the pre-captured image to the first device and the at least one second device; Receiving a first captured image of a first device and a second captured image of at least one second device, wherein the first device and the second device each include at least one depth camera, the first captured image includes a background image and a first portrait of a co-photographing initiator, and the second captured image includes the background image and a second portrait of a co-photographing invitee; Performing posture estimation on the first portrait and the second portrait using a posture estimation algorithm to determine target postures and / or target positions of the first portrait and the second portrait; fusing depth information corresponding to a background image, the first portrait, and the second portrait based on the target postures and / or target positions of the first portrait and the second portrait to obtain a combined image; the background image of the combined image is the background of the first captured image or the background of the second captured image; identifying user information of the first portrait and / or the second portrait in the combined image, performing a beautification operation on the first portrait and / or the second portrait according to the identified user information, and performing three-dimensional construction on the first portrait and / or the second portrait to obtain a processed combined image; The processed combined image is sent to the first device and the at least one second device.

6. The method according to claim 5, characterized in that Before fusing the depth information corresponding to the background image, the first portrait, and the second portrait to obtain a combined image, the method further includes: Receive background switching request; The background image is switched according to the background switching request.

7. The method according to claim 5, characterized in that The camera parameters carried in the first captured image and the second captured image are the same, and the camera parameters include at least one of sensitivity, brightness, color, white balance parameter or exposure parameter.

8. The method according to claim 5, characterized in that Before sending the combined image to the first device and the at least one second device, the method further includes: The combined image is beautified, and the beautification includes at least one of skin smoothing, face slimming, acne removal, eye enlargement, makeup, body shaping, whitening, wrinkle removal, hairdressing, and dark circle removal.

9. A first device, characterized in that: include: one or more processors; Memory; One or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the first device, the first device executes the synchronization method as described in any one of claims 1-4.

10. A server, characterized in that: include: one or more processors; Memory; Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the server, the server executes the co-production method as described in any one of claims 5-8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are read and executed by one or more processors, the co-production method according to any one of claims 1 to 4 is executed, or the co-production method according to any one of claims 5 to 8 is executed.