Synchronous shooting method, first electronic device and second electronic device

Abstract

The embodiment of the present application provides a synchronous shooting method, a first electronic device and a second electronic device. The synchronous shooting method includes: the first electronic device receives a first message from a synchronization server; the first message includes a first planned time point, and information about starting to shoot a video frame at the first planned time point; when the first time offset is greater than a preset threshold, P times of interpolation operations are performed, so that the first electronic device starts shooting a video frame at the first planned time point; the second electronic device receives a first message from a synchronization server; the first message includes a first planned time point, and information about starting to shoot a video frame at the first planned time point; when the second time offset is greater than a preset threshold, Q times of interpolation operations are performed, so that the second electronic device starts shooting a video frame at the first planned time point. The embodiment of the present application can quickly and automatically realize multi-camera synchronous shooting with high synchronization accuracy.

Description

Technical Field

[0001] The present application relates to the field of videography, and in particular to a synchronous shooting method, a first electronic device and a second electronic device. Background Art

[0002] Multi-camera synchronization includes but is not limited to hardware synchronization, content-based synchronization, and timestamp-based synchronization. Hardware synchronization requires providing an external synchronization signal input interface (e.g., Genlock) on the hardware, using which multiple cameras are connected to the trigger source of the same synchronization source, and using the trigger source to simultaneously trigger multiple cameras to start shooting, thereby achieving frame synchronization of multiple cameras. However, this will result in higher hardware costs and a smaller range of selectable cameras, making it difficult to popularize.

[0003] Content-based synchronization requires that multiple cameras first capture overlapping images, and then synchronize the video frames captured by each camera based on the overlapping images and sounds. However, this makes the operation complicated and cannot be processed in real time during video recording.

[0004] Timestamp-based synchronization estimates the time offset between cameras by estimating and calculating the relevant time, thereby achieving synchronization of multiple cameras. The network time protocol (NTP) is generally used. The accuracy of the NTP protocol is greatly affected by the network environment and is generally between 10-500ms. In addition, the NTP protocol does not take into account the different imaging delays of each camera and cannot ensure that the camera captures the image at the same time, which will result in low accuracy of synchronized shooting or even failure to synchronize shooting. Summary of the invention

[0005] In order to solve the above technical problems, the inventors of this application have made improvements based on the time synchronization of NTP after long-term research, and provide a synchronous shooting method, a first electronic device and a second electronic device, which can quickly realize the frame synchronization of multiple cameras, and the synchronization accuracy is high, which is convenient for scenes such as video splicing and 3D reconstruction. In addition, the synchronous shooting method of the embodiment of the present application is not only applicable to TV scenes with multi-screen combinations, but can also be extended to multiple terminal devices with cameras, as long as they can be fixedly connected to the synchronization server for synchronization configuration.

[0006] In a first aspect, an embodiment of the present application provides a synchronous shooting method, which is applied to a display system; the display system includes a first electronic device and a second electronic device; the first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; the first electronic device and the second electronic device communicate with a synchronization server; the synchronization server is used to synchronize the first electronic device and the second electronic device; the first display screen can play a first picture according to first data captured by the first camera; the second display screen can play a second picture according to second data captured by the second camera; the first picture and the second picture can be displayed in combination; the method includes: the first electronic device receives a first message from the synchronization server; the first message includes a first scheduled time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device to shoot at the first planned time point T M ', start shooting a video frame; the first electronic device according to the first planned time point T M ', and the interval time T between any two video frames shot by the first electronic device, obtain the second planned time point T1'; wherein the second planned time point T1' is located at [T1, T1+T]; T1 is the start time point of shooting the first current video frame; the interval time T between any two video frames shot by the first electronic device is equal; the first electronic device obtains the first time offset according to T1 and T1'; after the first time offset is greater than the first preset threshold value and after the shooting of the first current video frame is completed, the first electronic device performs P times of interpolation operation; after performing P times of interpolation operation, the first electronic device sends a second message to the synchronization server; the second message is used to indicate that the first electronic device has completed synchronization with the synchronization server; the second electronic device receives the first message from the synchronization server; the first message includes the first planned time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the second electronic device to shoot at the first planned time point T M ', start shooting a video frame; the second electronic device according to the first planned time point T M', and the interval time T between any two video frames shot by the second electronic device, obtain the third planned time point T1"; wherein, the third planned time point T1" is located at [T0, T0+T]; T0 is the starting time point of shooting the second current video frame; the interval time T between any two video frames shot by the second electronic device is equal; the second electronic device obtains the second time offset according to T0 and T1", after the second time offset is greater than the second preset threshold value and after the shooting of the second current video frame is completed, the second electronic device performs Q frame insertion operations; after performing Q frame insertion operations, the second electronic device sends a third message to the synchronization server; the third message is used to indicate that the second electronic device has completed synchronization with the synchronization server; wherein, after performing P frame insertion operations, the first electronic device starts shooting a video frame at the first planned time point TM'; after performing Q frame insertion operations, the second electronic device starts shooting a video frame at the first planned time point TM'; P, Q are positive integers greater than or equal to 1.

[0007] In a possible implementation, the display system further includes a synchronization server.

[0008] In a second aspect, an embodiment of the present application provides a synchronous shooting method, which is applied to a display system; the display system includes a first electronic device and a second electronic device; the first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; the first electronic device and the second electronic device communicate with each other; the first display screen can play a first picture according to first data captured by the first camera; the second display screen can play a second picture according to second data captured by the second camera; the first picture and the second picture provided by the second electronic device can be displayed in combination; the method includes: the first electronic device receives a first message from the second electronic device; the first message includes a first planned time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device to shoot at the first planned time point T M ', start shooting a video frame; the first electronic device according to the first planned time point T M', and the interval time T between any two video frames shot by the first electronic device, obtain the second planned time point T1'; wherein, the second planned time point T1' is located at [T1, T1+T]; T1 is the starting time point of shooting the first current video frame; the interval time T between any two video frames shot by the first electronic device is equal; the first electronic device obtains the first time offset according to T1 and T1'; after the first time offset is greater than the first preset threshold value and after the shooting of the first current video frame is completed, the first electronic device performs P times of interpolation operation; after performing P times of interpolation operation, the first electronic device sends a second message to the second electronic device; the second message is used to indicate that the first electronic device and the second electronic device have completed synchronization; the second electronic device starts shooting a video frame at the first planned time point TM'; wherein, after performing P times of interpolation operation, the first electronic device starts shooting a video frame at the first planned time point; P is a positive integer greater than or equal to 1.

[0009] In a third aspect, an embodiment of the present application provides a synchronous shooting method, which is applied to a synchronization server; the synchronization server communicates with a first electronic device and a second electronic device; the first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; the first display screen can play a first picture according to the first data captured by the first camera; the second display screen can play a second picture according to the second data captured by the second camera; the first picture and the second picture can be displayed in combination; the synchronization server is used to synchronize the first electronic device and the second electronic device; the method includes: the synchronization server sends a first message to the first electronic device and the second electronic device; the first message includes a first planned time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to indicate that the first electronic device and the second electronic device are at the first planned time point T M ', start shooting a video frame; receive a second message from the first electronic device; the second message is used to indicate that the first electronic device has completed synchronization with the synchronization server; receive a third message from the second electronic device; the third message is used to indicate that the second electronic device has completed synchronization with the synchronization server.

[0010] In a fourth aspect, an embodiment of the present application provides a synchronous shooting method, which is applied to a second electronic device; the second electronic device communicates with the first electronic device; the first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; the first display screen can play a first picture according to the first data shot by the first camera; the second display screen can play a second picture according to the second data shot by the second camera; the first picture and the second picture can be displayed in combination; the method includes: the second electronic device sends a first message to the first electronic device; the first message includes a first planned time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device to shoot at the first planned time point T M ', start shooting a video frame; receive a second message from the first electronic device; the second message is used to indicate that the first electronic device and the second electronic device have completed synchronization; the second electronic device at the first scheduled time point T M 'Start capturing a video frame.

[0011] In a fifth aspect, an embodiment of the present application provides a synchronous shooting method, which is applied to a first electronic device; the first electronic device includes a first camera and a first display screen; the first display screen can play a first picture according to first data captured by the first camera; the first picture and a second picture provided by a second electronic device can be displayed in combination; the second electronic device includes a second camera and a second display screen; the second display screen can play a second picture according to second data captured by the second camera; the first electronic device communicates with a synchronization server; the synchronization server also communicates with the second electronic device, and the synchronization server is used to synchronize the first electronic device and the second electronic device; the method includes: the first electronic device receives a first message from the synchronization server; the first message includes a first scheduled time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device to shoot at the first planned time point T M ', start shooting a video frame; the first electronic device according to the first planned time point T M', and the interval time T between any two video frames shot by the first electronic device, obtain the second planned time point T1'; wherein the second planned time point T1' is located at [T1, T1+T]; T1 is the start time point of shooting the first current video frame; the interval time T between any two video frames shot by the first electronic device is equal; the first electronic device obtains the first time offset according to T1 and T1'; after the first time offset is greater than the first preset threshold value and after the shooting of the first current video frame is completed, the first electronic device performs P times of interpolation operation; after performing P times of interpolation operation, the first electronic device sends a second message to the synchronization server; the second message is used to indicate that the first electronic device has completed synchronization with the synchronization server; wherein, after performing P times of interpolation operation, the first electronic device at the first planned time point T M ', start shooting a video frame; P is a positive integer greater than or equal to 1.

[0012] In the sixth aspect, an embodiment of the present application provides a first electronic device, which includes a first camera, a first display screen, a memory, a processor and a computer program; wherein the computer program is stored in the memory; when the computer program is executed by the processor, the first electronic device implements the synchronous shooting method provided in the first aspect, the second aspect or the fifth aspect.

[0013] In the seventh aspect, an embodiment of the present application provides a second electronic device, which includes a second camera, a second display screen, a memory, a processor and a computer program; wherein the computer program is stored in the memory; when the computer program is executed by the processor, the second electronic device implements the synchronous shooting method provided in the first aspect, the second aspect or the fourth aspect.

[0014] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer program. When the computer program runs on a first electronic device, the first electronic device implements the synchronous shooting method provided in the first aspect, the second aspect, or the fifth aspect.

[0015] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer program. When the computer program runs on a second electronic device, the second electronic device implements the synchronous shooting method provided in the first aspect, the second aspect, or the fourth aspect.

[0016] In the tenth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a first electronic device, the first electronic device executes the synchronous shooting method provided in the first aspect, the second aspect, or the fifth aspect.

[0017] In the eleventh aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a second electronic device, the second electronic device executes the synchronous shooting method provided in the first aspect, the second aspect, or the fourth aspect.

[0018] The solution of the embodiment of the present application is aimed at the time difference of the video frames captured by the cameras of different electronic devices. First, the system time of each electronic device is synchronized with the system time of the synchronization server. Then, each electronic device obtains the planned time point of each video frame respectively. For the camera whose time offset between the actual time point and the planned time point of the video frame exceeds the preset range, the corresponding electronic device will perform at least one frame insertion operation on the camera, such as inserting a new frame of a specified length immediately after shooting a frame, to ensure that the time offset of the subsequent video frames can be within the preset range, so that the time offset of the video frames captured by the cameras of all electronic devices is within the preset range, achieving the purpose of quickly synchronizing the video frames captured by each camera, and the synchronization accuracy is high, which significantly improves the synchronization effect. At the same time, it does not rely on hardware synchronization and shooting content synchronization, has a wider scope of application, does not require manual intervention after starting the synchronization process, is easy to operate, and is easy to popularize. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a scene of a synchronous shooting method provided in an embodiment of the present application;

[0020] Figure 2 This is the principle diagram of NTP time synchronization in client / server mode;

[0021] Figure 3 for Figure 1 A schematic diagram of the software structure of the electronic device in FIG.

[0022] Figure 4 A schematic diagram of a synchronous shooting method provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of a frame insertion operation involved in the synchronous shooting method of an embodiment of the present application;

[0024] Figure 6 A schematic diagram of another frame insertion operation involved in the synchronous shooting method of an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a flow chart of another synchronous shooting method provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of a flow chart of another synchronous shooting method provided in an embodiment of the present application;

[0027] Fig. 9A schematic diagram of the structure of a synchronous shooting device provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of the structure of another synchronous shooting device provided in an embodiment of the present application;

[0029] Fig.11 A schematic diagram of the structure of a synchronization server provided in an embodiment of the present application;

[0030] Fig.12 A schematic diagram of the structure of another synchronization server provided in an embodiment of the present application;

[0031] Fig.13 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in a "or" relationship.

[0033] References to "one embodiment" or "some embodiments" described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "include", "include", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. In addition, in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0034] For example, Figure 1 A schematic diagram of a scene of a synchronous shooting method provided in an embodiment of the present application. Figure 1 As shown, the synchronous shooting system includes a synchronous server and a display system, and the display system includes 4 electronic devices; each electronic device is provided with a camera and a display screen. Figure 1 In the embodiment, the number of electronic devices is 4; however, 4 is only an illustrative example, and any positive integer n greater than 1 can represent the number of electronic devices, which is also within the scope of the present application. The synchronization server can send a synchronous shooting instruction to the 4 electronic devices, so that the 4 cameras shoot synchronously; the 4 display screens are used to combine and display the video frames shot synchronously by the 4 cameras. That is, the 4 electronic devices are combined into a complete device to provide a larger and clearer visual experience. Alternatively, the synchronous shooting system can be an electronic device. Optionally, the synchronous shooting system can also include other devices.

[0035] It should be noted that Figure 1 The synchronous shooting system shown is only a possible system architecture provided by the embodiment of the present application. Based on the system architecture, there may be many possible variations. For example, the synchronization server is integrated with an electronic device.

[0036] The shooting synthesis technology based on n cameras can be used for application scenarios such as identifying the 3D depth of objects and larger-scale image and video data. These application scenarios all rely on the synchronization mechanism between cameras. Therefore, it is necessary to ensure that each camera shoots video frames at the same time. In this way, after obtaining the synchronized frames of N cameras, the synchronized frames are spliced ​​to synthesize a larger range of images or perform 3D effect processing. Multiple spliced ​​synchronized frames are played continuously to generate a video.

[0037] Multi-camera synchronization includes but is not limited to hardware synchronization, content-based synchronization, and timestamp-based synchronization. Among them, hardware synchronization requires providing an external synchronization signal input interface (e.g., Genlock (synchronous lock / synchronous phase lock)) on the hardware, using this interface to connect multiple cameras to the trigger source of the same synchronization source, and using this trigger source to simultaneously trigger multiple cameras to start shooting, etc., thereby achieving frame synchronization of multiple cameras. However, this will result in high hardware costs, and the range of selectable cameras is small, which is not easy to popularize. Content-based synchronization requires first shooting overlapping images with multiple cameras, and then synchronizing the video frames shot by each camera according to the overlapping images and sounds. However, this will lead to complicated operations and cannot be processed in real time during video recording. Timestamp-based synchronization estimates the time offset between cameras by estimating and calculating the relevant time, thereby achieving synchronization of multiple cameras. The network time protocol (NTP) is generally used. The accuracy of the NTP protocol is greatly affected by the network environment, generally between 10-500ms. In addition, the NTP protocol does not take into account the different imaging delays of each camera and cannot ensure that the cameras shoot images at the same time, which will result in low accuracy of synchronized shooting or even the inability to shoot synchronously.

[0038] After long-term research, the inventor of this application has made improvements based on the time synchronization of NTP and provided a synchronous shooting method that can quickly achieve frame synchronization of multiple cameras with high synchronization accuracy, which is convenient for scenes such as video splicing and 3D reconstruction. In addition, the synchronous shooting method of the embodiment of this application is not only applicable to TV scenes with multiple screen combinations, but can also be extended to multiple terminal devices with cameras, as long as they can be fixedly connected to the synchronization server for synchronization configuration.

[0039] The most typical time synchronization method of NTP is the client / server method. For example, Figure 2 The following is a schematic diagram of the NTP time synchronization in client / server mode. Figure 2As shown, the client first sends an NTP packet to the server, and the NTP packet contains the timestamp T1 when the NTP packet leaves the client; when the server receives the NTP packet, it fills in the timestamp T2 when the NTP packet arrives and the timestamp T3 when the NTP packet leaves in sequence, and then immediately returns the NTP packet to the client. When the client receives the NTP packet, it records the timestamp T4 returned by the NTP packet. The client can calculate two key parameters using the above four time parameters: the round-trip time d of the NTP packet and the clock deviation t between the client and the server. The client uses the clock deviation t to adjust the local clock so that its time is consistent with the server time. Among them, timestamps T1 and timestamps T4 are referenced to the client time; timestamps T2 and timestamps T3 are referenced to the server time; d1 is the transmission delay of the NTP packet from the client to the server, and d2 is the transmission delay of the NTP packet from the server to the client; t is the time deviation between the server and the client, and d is the round-trip time of the NTP packet. Given T1, T2, T3, and T4, we want to find t to adjust the client clock:

[0040]

[0041] Assuming d1 = d2, we can solve

[0042]

[0043] It can be seen that t and d are only related to the time offset between T2 and T1 and the time offset between T3 and T4, but not to the time offset between T2 and T3, that is, the final result has nothing to do with the time required for the server to process the request. The client can calculate the time deviation t through T1, T2, T3, and T4. Specifically, it can be repeated K times according to the statistical method, and the minimum result among the K times is selected as the final time deviation t; then, the client adjusts the local clock according to the calculated time deviation t, so that the system time of the client is synchronized with the system time of the server.

[0044] Figure 3 for Figure 1 Schematic diagram of the software structure of the electronic device in the embodiment of the present application, taking the Android system with a layered architecture as an example, the software structure of the electronic device is exemplified. Figure 3 As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the system framework layer, the system library and runtime layer, the hardware abstraction layer (HAL), and the kernel layer.

[0045] The application layer may include a series of application packages. For example, the application package may include programs such as desktop manager, settings, calendar, camera, music, etc. Among them, the desktop manager is used to manage and display the icons of all common applications and their corresponding card information, and it has no icons on the desktop. Common applications such as settings, calendar, camera, music, etc. can be clicked on the desktop icon program, used to provide related services.

[0046] The system framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. Exemplarily, the system framework layer includes, but is not limited to, window managers, activity managers, package managers, resource managers, camera devices, camera managers, and camera services. Among them, the activity manager is used to manage the life cycle of applications and implement the navigation back function of each application. Exemplarily, the activity manager can be responsible for the creation of activity processes and the maintenance of the life cycle of the created activity processes. The window manager is used to manage window programs. It can be understood that the graphical user interface of an application is usually composed of one or more activities, and an activity is composed of one or more views; the window manager can add the view included in the graphical user interface to be displayed to the display screen, or remove the view from the graphical user interface displayed on the display screen. The package manager can manage the data packets corresponding to each application, such as decompressing, verifying, installing, and upgrading each data packet. The resource manager is used to provide access to various non-code resources to the application. The camera device refers to a logical camera device, which is used to provide a class for providing application access to the device. A camera device corresponds to an actual camera device. The camera manager is a system service that can be used to detect and open camera devices and obtain the characteristics of camera devices. The camera service is an independent service process that can be used to access camera devices, receive application requests, process image data, and return results.

[0047] The system library and runtime layer include the system library and Android Runtime. The system library can include multiple functional modules. For example: browser kernel, 3D graphics library (for example: OpenGL ES), font library, etc. The browser kernel is responsible for interpreting the syntax of the web page (such as an application HTML, JavaScript under the standard universal markup language) and rendering (displaying) the web page. The 3D graphics library is used to implement three-dimensional graphics drawing, image rendering, synthesis and layer processing, etc. The font library is used to implement the input of different fonts. The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one part is the function function that the Java language needs to call, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0048] The hardware abstraction layer is an abstract interface for the device kernel driver, which provides an application programming interface to the higher-level Java API framework to access the underlying device. The HAL contains multiple library modules, each of which implements an interface for a specific type of hardware component. For example, the HAL includes camera, display, Bluetooth, and audio.

[0049] The kernel layer is a layer between hardware and software. The kernel layer includes several hardware drivers. Exemplarily, the kernel layer may include a camera driver, a display driver, a Bluetooth driver, and an audio driver. Each driver may collect information collected by the corresponding hardware and report the corresponding monitoring data to the status monitoring service or other functional modules in the system library.

[0050] Continue to refer Figure 3 When the camera application software of the application layer receives the shooting instruction, it will send a capture request to the system framework layer to bind the camera service, and then reach the camera-related interface of the hardware abstraction layer through the system library runtime layer, and then to the camera driver of the kernel layer to drive the camera hardware device to perform the image shooting operation, thereby outputting each frame of data. In addition, the delay from the camera application software receiving the shooting instruction to the camera hardware device executing the image shooting operation is different for different electronic devices. This delay changes dynamically. In order to achieve high-precision synchronous shooting, the shooting time needs to be adjusted according to the shooting delay of different electronic devices.

[0051] For example, Figure 4 A schematic diagram of a synchronous shooting method provided in an embodiment of the present application. The method can be used Figure 1The synchronous shooting system shown. The synchronous shooting system may include a synchronization server and a display system, and the display system may include n electronic devices. Each electronic device has a camera and a display screen, and the display screens of multiple electronic devices are combined to display the video frames captured by the cameras of the multiple electronic devices.

[0052] For example, the display system may include a first electronic device and a second electronic device; the first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; the first electronic device and the second electronic device communicate with a synchronization server; the synchronization server is used to synchronize the first electronic device and the second electronic device; the first display screen can play a first picture based on first data captured by the first camera; the second display screen can play a second picture based on second data captured by the second camera; the first picture and the second picture can be displayed in combination.

[0053] Figure 4 Only one electronic device is shown. Figure 4 As shown, the synchronous shooting method may include the following steps:

[0054] S401, multiple electronic devices synchronize their own system time with the system time of a synchronization server according to the Network Time Protocol NTP.

[0055] Specifically, first, multiple electronic devices are connected to the synchronization server for communication, for example, wirelessly. The synchronization server broadcasts and sends NTP protocol data packets to multiple electronic devices. Multiple electronic devices return NTP synchronization data to the synchronization server respectively to complete system time synchronization. Since multiple electronic devices are synchronized with the synchronization server independently, multiple electronic devices can complete the system time synchronization operation in parallel.

[0056] S402, the synchronization server sends a first message to multiple electronic devices respectively, the first message includes a planned time point T of synchronously shooting a target video frame M ' and a preset range R. Alternatively, the preset range R is pre-set in the electronic device.

[0057] For example, R = 10ms, that is, it is planned to make all cameras M 'Start shooting a video frame within ±10ms. In addition, before the synchronization server sends the first message, the cameras of multiple electronic devices have started preview shooting or formal shooting mode.

[0058] S403, the plurality of electronic devices respectively determine the target video frame at the planned time point T M ' and the interval length T of adjacent video frames to obtain the planned time point for shooting each video frame, wherein the interval length of adjacent video frames is the same.

[0059] Figure 5 FIG. 1 is a schematic diagram of a frame insertion operation involved in the synchronous shooting method of an embodiment of the present application. Figure 5 As shown, the synchronization server sends the planned time point T for synchronously shooting the target video frame. M ', the electronic device obtains T at the first moment Q1 M Then, the electronic device calculates the target video frame according to the planned time point T M ' and the interval time T of the adjacent video frames to obtain the planned time point for shooting each video frame. Specifically, if Tm' is the planned time point for shooting the mth video frame, then Tm'=T M '-NT, N is a positive integer, and N ≥ 1. If Tm is the actual time point of shooting the mth video frame, then Tm ≤ Tm' ≤ Tm+T, that is, the planned time point of the mth video frame is guaranteed to be between the actual time point of the mth video frame and the actual time point of the m+1th video frame. For example, the planned time point of the target video frame T M ' is T4', and the planned time points of each video frame obtained by the electronic device may be T1', T2', and T3', wherein T1'=T4'-3×T, T2'=T4'-2×T, and T3'=T4'-T.

[0060] That is, through S402 and S403, the multiple electronic devices respectively obtain the scheduled time points for synchronously shooting the respective video frames. Alternatively, the scheduled time points for the respective video frames can be directly set in the electronic devices.

[0061] S404, the first electronic device determines whether the time offset of the current video frame captured by the first camera is within a preset range R. The first electronic device is at least one of the plurality of electronic devices, the first camera is a camera of the first electronic device, and the time offset of each video frame is the time difference between the actual time point at which each video frame starts to be captured and the planned time point at which each video frame starts to be captured.

[0062] If the result of the determination is yes, the process proceeds to S405, where the first electronic device generates a second message, where the second message is used to indicate that the synchronization of the first camera is completed. S406, where the first electronic device sends the second message to the synchronization server.

[0063] If the judgment result is no, entering S407, the first electronic device performs at least one frame insertion operation on the first camera according to the time offset of the current video frame, and the frame insertion operation is used to adjust the time offset of the video frame shot by the first camera after the frame insertion operation.

[0064] The first electronic device causes the corresponding camera to perform at least one frame insertion operation according to the time offset of the current video frame. Specifically, the operation can be performed according to the following steps: First, the first electronic device calculates the shooting time t' of the inserted frame according to the time offset Δ of the current video frame. Since the shooting time is too short and the requirement of shooting a video frame cannot be met, the shooting time t' of the inserted frame can be made greater than the shooting time of each video frame. Then, after the shooting time t of the current video frame arrives, the first electronic device inserts the inserted frame according to the shooting time t' of the inserted frame. In addition, the first electronic device can calculate the shooting time t' of the inserted frame according to the time offset of the current video frame in the following two ways, but not limited to:

[0065] Method 1: When the time offset of the current video frame is greater than the shooting time of each video frame, the first electronic device determines the shooting time t' of the inserted frame as the time offset of the current video frame.

[0066] like Figure 5 As shown, when the actual time point T1 of the first video frame is earlier than the planned time point T1' of the first video frame, if the interpolation operation is not performed, the actual time point T2 of the second video frame is still earlier than the planned time point T2' of the second video frame, so the interpolation frame can be inserted after the shooting time t of the first video frame arrives, and the shooting time t' of the interpolation frame can be made the time offset Δ of the first video frame. This is equivalent to extending the shooting time of the first video frame, so that the actual time point T3 of the next video frame (i.e., the third video frame) of the interpolation frame (i.e., the second video frame) can be adjusted to be consistent with the planned time point T2' of the second video frame. In addition, since the shooting time t cannot meet the requirement of shooting a video frame when it is too short, the shooting time t' of the interpolation frame is determined to be Δ only when the time offset Δ is greater than the shooting time t of each video frame. When this method is adopted, the interpolation frame can be regarded as an additional video frame, so that the number of frames in the unit time (such as 1s) including the interpolation operation is more than the number of frames in the unit time without the interpolation operation.

[0067] Method 2: The first electronic device sets a synchronization time point, which is one of the planned time points of each video frame, and the synchronization time point is greater than the planned time point of the current video frame. Then, the first electronic device calculates the first time interval between the actual time point of the current video frame and the synchronization time point. Next, the first electronic device calculates the shooting time of the inserted frame according to the first time interval, the shooting time of the current video frame, and the second time interval. When this method is adopted, the number of frames in the unit time including the frame insertion operation is the same as the number of frames in the unit time not including the frame insertion operation.

[0068] Figure 6 FIG. 2 is a schematic diagram of another frame insertion operation involved in the synchronous shooting method of an embodiment of the present application. Figure 6 As shown, the current video frame is received by TM ', the first electronic device sets the planned time point T3' for shooting the third video frame as the synchronization time point. Next, the first electronic device calculates the first time interval J1 between the actual time point T1 of the first video frame and the synchronization time point T3', J1=2T+Δ, Δ is the time offset of the first video frame. Then, the first electronic device calculates the shooting time t' of the inserted frame according to the first time interval J1, the shooting time t of the first video frame and the second time interval J2, wherein the second time interval J2 is the time interval between the inserted frame and the third video frame, the shooting time of each video frame is t, J2=Tt is set, and the shooting time t' of the inserted frame is t-t-J2=T+Δ.

[0069] S408: The first electronic device determines whether the time offset of the video frame captured after the frame insertion operation is performed on the first camera is within a preset range R.

[0070] If the judgment result is yes, the process returns to S405. If the judgment result is no, the process proceeds to S409, where the first electronic device determines whether the first camera performs a set number of frame insertion operations.

[0071] At S409, if the result of the determination is no, the process proceeds to S410, where the first electronic device causes the first camera to perform the frame insertion operation again, and then the process returns to S408.

[0072] At S409, if the judgment result is yes, then entering S411, the first electronic device generates a feedback message, and the feedback message is used to indicate that the first camera has not completed synchronization.

[0073] S412: The first electronic device sends a feedback message to the synchronization server.

[0074] Next, at S413, there may be but are not limited to the following two situations:

[0075] The first case: S413, the synchronization server determines whether second messages sent by multiple electronic devices are received.

[0076] For example, first, the synchronization server sends a first message to the first electronic device and the second electronic device (S401). The first electronic device has a first camera, and the second electronic device has a second camera. Then, the first electronic device determines that the time offset of the first video frame captured by the first camera is within a preset range R (S404), the first electronic device generates a second message (S405), and sends the second message to the synchronization server (S406); the second electronic device determines that the time offset of the first video frame captured by the second camera is outside the preset range R (S404), the second electronic device performs a frame insertion operation (S407), and then, the second electronic device determines whether the time offset of the video frame captured by the second camera after the frame insertion operation is within the preset range R (S408), wherein:

[0077] If the judgment result of S408 is yes, a third message is generated (S405), and the third message is sent to the synchronization server (S406). The third message is also the second message. In order to distinguish it from the second message of the first electronic device, the third message is used to indicate that the second electronic device has completed synchronization. At this time, the judgment result of S413 is yes;

[0078] If the judgment result of S408 is no, it is determined whether the set number of interpolation operations has been reached (S409). If the set number has not been reached, the interpolation operation is performed again (S410). If the set number has been reached, that is, the judgment result of S409 is yes, a feedback message indicating that the synchronization of the second camera is not completed is generated (S411), and a feedback message is sent to the synchronization server (S412). At this time, the judgment result of S413 is no.

[0079] That is, when the electronic device determines that the time offset Δ of the first video frame is greater than the preset range R, an insertion frame is inserted immediately after the shooting time t of the first video frame is reached, and the shooting time t' of the insertion frame can be calculated based on the time offset Δ of the first video frame in the above two ways. The insertion frame is the second video frame, so that the actual time point of shooting the third video frame can be adjusted so that the time offset Δ of the third video frame is within the preset range R. If the time offset Δ of the third video frame is still outside the preset range R, a new insertion frame is inserted again after the shooting time of the third video frame is reached, and the shooting time of the new insertion frame is determined according to the time offset of the third video frame. If the time offset of the video frame is still outside the preset range R after the set number of insertion operations, it means that the current shooting environment cannot meet the requirements of the preset range R, and the electronic device can send a feedback message to the synchronization server, and the feedback message is used to indicate that the synchronization of the camera corresponding to the electronic device is not completed. Next, the synchronization server can increase the preset range R based on the feedback message, update the increased preset range R in the first message, and then send it to multiple electronic devices respectively, so that the multiple electronic devices can determine whether the time offset of the video frames taken by their respective cameras meets the updated preset range R according to the updated preset range R. If not, interpolation operation can be performed to adjust the time offset of each video frame so that it is within the preset range R.

[0080] The second case: S413, the synchronization server determines whether the synchronization success rate is greater than a set threshold.

[0081] Specifically, the synchronization server calculates the sampling number W based on multiple time offsets obtained in advance. The synchronization server counts the synchronization success rate S after the W electronic devices perform frame insertion operations on their respective corresponding cameras, S=V / W, where V is the number of second messages received by the synchronization server. The synchronization server determines that the synchronization success rate S is less than the set threshold, increases the preset range, updates the increased preset range in the first message, and then sends it to multiple electronic devices respectively. Among them, the synchronization server calculates the sampling number W based on the multiple time offsets obtained in advance according to the following process: First, multiple electronic devices send the time offset of at least one video frame to the synchronization server or at least one electronic device sends the time offset of multiple video frames to the synchronization server to form a sample set. Next, the synchronization server calculates the mean μ and variance σ of the time offset based on the sample set. Then, the synchronization server assumes that the statistical error of the time offset is a Gaussian distribution, and calculates the sampling number W=9σ 2 / μ 2 .

[0082] That is, assuming that the statistical error of the time offset is Gaussian distributed, according to the law of large numbers and the central limit law, 99.7% refers to the confidence rate. That is, when W trials are conducted, the synchronization success rate is 99.7% (i.e., the set threshold), W = 9σ2 / μ 2 , σ is the variance, and μ is the mean. Among them, the mean μ and the variance σ can be calculated based on the sample set of time offset obtained after performing multiple frame insertion operations in advance. And, according to different preset ranges R, different sampling numbers W can be calculated in advance. In this way, when multiple cameras are synchronized for shooting after selecting the preset range R, the W cameras can be synchronized for shooting according to the above-mentioned synchronous shooting method. In theory, the cameras of the W electronic devices should meet the synchronization success rate of 99.7%. If it is not satisfied, it means that the current shooting conditions cannot adapt to the selected preset range R, then the synchronization server can increase the preset range R and carry the increased preset range R in the first message and resend it to multiple electronic devices, so that multiple electronic devices can readjust the shooting time of their respective cameras to achieve synchronous shooting. Among them, the current shooting conditions may include environmental factors that affect the synchronization accuracy, such as the network and the performance parameters of the camera equipment itself.

[0083] For example, assuming that the threshold is set to 99%, and the synchronous shooting system includes two electronic devices. If both electronic devices send the second message indicating that the camera synchronization is completed, the synchronization success rate is 100%, which is greater than the set threshold. At this time, the judgment result at S413 is yes; if one of the two electronic devices sends the second message and the other sends a feedback message, the synchronization success rate is 50%, which is less than the set threshold. At this time, the judgment result at S413 is no.

[0084] At S413 , if the judgment result is no, then the process proceeds to S414 , where the synchronization server updates the preset range in the first message, and then returns to step 402 .

[0085] At S413, if the judgment result is yes, then enter S415, and the synchronization server sends synchronization shooting instructions to multiple electronic devices respectively.

[0086] S416, the plurality of electronic devices respectively cause their respective cameras to perform synchronous shooting according to the synchronous shooting instruction.

[0087] It should be noted that when the cameras of multiple electronic devices are in the formal shooting mode, S415 and S416 may not be performed. At this time, the first message sent at S402 can be considered as a synchronous shooting instruction, that is, multiple electronic devices always shoot according to the planned time points of each video frame, and the planned time points of each video frame can be obtained by the planned time points of the synchronous shooting target video frames at S402 and the interval duration of adjacent video frames. When the time offset of the video frames shot by the cameras of one or some electronic devices is outside the preset range, the corresponding electronic device will promptly perform frame insertion operations on its camera to adjust the time offset of the subsequently shot video frames, thereby ensuring that the time offsets of the video frames shot by the cameras of multiple electronic devices are all within the preset range, so that the synchronization accuracy is higher.

[0088] Figure 7 A flowchart of another synchronous shooting method provided in an embodiment of the present application. The synchronous shooting method is performed by an electronic device. The electronic device has a camera and a display screen, and the display screens of multiple electronic devices can be combined to display video frames captured by the cameras of the multiple electronic devices, such as Figure 7 As shown, the method comprises the following steps:

[0089] S701, synchronizing the system time with the system time of the synchronization server according to the Network Time Protocol NTP.

[0090] S702: When the camera is in a preview shooting or formal shooting mode, a first message sent by a synchronization server is obtained, where the first message includes a planned time point T for synchronously shooting a target video frame. M '. The first message sent by the synchronization server to the electronic device may also include a preset range R. Alternatively, the preset range R is pre-set in the electronic device.

[0091] S703, obtaining the planned time point for shooting each video frame according to the planned time point of the target video frame and the interval time T between adjacent video frames. Specifically, if Tm' is the planned time point for shooting the mth video frame, then Tm'=T M '-NT, N is a positive integer, and N ≥ 1, if Tm is the actual time point when the camera shoots the mth video frame, then Tm ≤ Tm' ≤ Tm + T. That is, it is ensured that the planned time point of the mth video frame is between the actual time point of the mth video frame and the actual time point of the m+1th video frame.

[0092] That is, through S702 and S703, the electronic device can obtain the scheduled time point for synchronously shooting each video frame. Alternatively, the scheduled time point for each video frame can be directly set in the electronic device.

[0093] S704, determining whether the time offset of the current video frame captured by the camera is outside a preset range R, wherein the time offset of each video frame is the time difference between the actual time point at which each video frame starts to be captured and the planned time point at which each video frame starts to be captured.

[0094] If the judgment result of S704 is no, proceed to S705, and send a second message to the synchronization server, the second message is used to indicate that the camera has completed synchronization. S706, receive the synchronization shooting instruction sent by the synchronization server and make the camera perform synchronization shooting according to the synchronization shooting instruction. The synchronization shooting instruction is generated by the synchronization server after receiving the messages indicating the completion of synchronization sent by multiple electronic devices.

[0095] When the camera of the electronic device is in the formal shooting mode, S706 may not be performed. At this time, the first message sent by the synchronization server at S702 can be considered as a synchronous shooting instruction. That is, the electronic device always shoots according to the planned time point of each video frame. The planned time point of each video frame can be obtained by the planned time point of the synchronous shooting target video frame at S702 and the interval length of adjacent video frames. When the time offset of the video frame shot by the camera of the electronic device is outside the preset range, the electronic device will timely perform frame insertion operation on the camera to adjust the time offset of the subsequently shot video frame to ensure that the time offset of the video frame shot by the camera of the electronic device is within the preset range, so that the synchronization accuracy is high.

[0096] If the judgment result of S704 is yes, enter S707, and perform a frame insertion operation after the shooting time of the current video frame arrives according to the time offset of the current video frame. The frame insertion operation is used to adjust the time offset of the video frame shot by the camera after the frame insertion operation.

[0097] The specific process of the electronic device performing the frame insertion operation on the camera may refer to the related description of S407 above.

[0098] S708, determining whether the time offset of the video frame captured by the camera after the frame insertion operation is within a preset range.

[0099] If the judgment result is yes, return to S705.

[0100] If the determination result is no, the process proceeds to S709 to determine whether the set number of frame insertion operations has been reached.

[0101] If the judgment result of S709 is no, return to S707.

[0102] If the judgment result of S709 is yes, then enter S710 and send a feedback message to the synchronization server, the feedback message indicating that the camera synchronization is not completed.

[0103] S711, receiving the updated first message sent by the synchronization server, the updated first message includes the preset range increased by the synchronization server based on the feedback message. In addition, the planned time point of the target video frame in the updated first message can also be reset. Then, return to step S703.

[0104] Figure 8 A flowchart of another synchronous shooting method provided in an embodiment of the present application is shown. The steps involved in the synchronous shooting method are executed by a synchronization server. Figure 8 As shown, the following steps may be included:

[0105] S801, synchronizing system time of multiple electronic devices according to the Network Time Protocol NTP, wherein each electronic device includes a camera and a display screen, and the display screens of the multiple electronic devices are combined to display video frames captured by the cameras of the multiple electronic devices.

[0106] S802, sending a first message to multiple electronic devices in a preview shooting or formal shooting mode, the first message including a planned time point T for synchronously shooting a target video frame M ' and a preset range R. The preset range R may also be pre-set in the electronic device.

[0107] S803, receiving second messages sent by multiple electronic devices respectively, the second messages are used to indicate that the time offset Δ of the video frames captured by the camera of the electronic device is within a preset range R, the time offset of each video frame is the time difference between the actual time point of starting to capture each video frame and the planned time point of starting to capture each video frame, and the planned time point of each video frame is determined by the electronic device according to the planned time point of the target video frame and the interval duration of adjacent video frames; wherein, when the time offset of the current video frame captured by the first camera is outside the preset range, the first electronic device performs at least one interpolation operation on the first camera, the interpolation operation is used to adjust the time offset of the video frame captured by the first camera, the first electronic device is at least one of the multiple electronic devices, and the first camera is the camera of the first electronic device.

[0108] S804: Determine whether all electronic devices have returned the second message or whether the synchronization success rate S is greater than a set threshold. The synchronization success rate S is calculated in the same manner as described in S413 above.

[0109] If the result of the determination is no, then the process proceeds to S805, where the preset range R is increased and the increased preset range is updated in the first message, and at this time, the planned time point for synchronously shooting the target video frame can also be updated when necessary. Then, the process returns to S802.

[0110] If the judgment result is yes, then enter S806 to send a synchronous shooting instruction to multiple electronic devices, where the synchronous shooting instruction is used to enable the cameras of the multiple electronic devices to perform synchronous shooting.

[0111] It should be noted that when the cameras of multiple electronic devices are in the formal shooting mode, S806 may not be performed. In this case, the first message sent by the synchronization server at S802 can be considered as a synchronous shooting instruction. In other words, multiple electronic devices always shoot according to the planned time points of each video frame. The planned time points of each video frame can be obtained by the planned time points of the synchronous shooting target video frames at S802 and the interval length of adjacent video frames. When the time offset of the video frame shot by the camera of one or some electronic devices is outside the preset range, the corresponding electronic device will timely perform frame insertion operations on its camera to adjust the time offset of the video frame, ensuring that the time offsets of the video frames shot by the cameras of multiple electronic devices are all within the preset range, so that the synchronization accuracy is high.

[0112] In addition, the synchronous shooting method may further include the following steps: the synchronization server may further receive a feedback message sent by the first electronic device, the feedback message being used to indicate that the first camera has not completed synchronization. Then, the synchronization server may increase the preset range based on the feedback message and update the increased preset range in the first message, and then return to S802.

[0113] To summarize, the synchronous shooting method of the embodiment of the present application first synchronizes the system time of each electronic device / camera with the system time of the synchronization server, and then, in view of the differences in camera shooting delays of different electronic devices, the synchronization server sends the planned time point for synchronously shooting the target video frame. Each electronic device can obtain the planned time point for shooting each video frame according to the planned time point of the target video frame and the interval duration of adjacent video frames. For cameras whose time offset between the actual time point and the planned time point of the video frame exceeds a preset range, at least one frame insertion operation is performed, such as inserting a new frame of a specified duration immediately after shooting a frame, to ensure that the time offset of subsequently shot video frames can be within the preset range, so that the time offset of video frames shot by all cameras is within the preset range, and the video frames shot by each camera can be quickly synchronized with high synchronization accuracy, which significantly improves the synchronization effect.

[0114] Furthermore, the automated synchronous shooting method of the embodiment of the present application does not rely on hardware synchronization or shooting content synchronization, and has a wider scope of application. Compared with the technology based on timestamp synchronization that only performs system time synchronization based on NTP, the synchronous shooting method of the embodiment of the present application has higher video synchronization accuracy. In addition, since the planned time point and preset range of synchronous shooting can be specified, the larger the preset range, the shorter the time it takes for multiple cameras to achieve synchronous shooting. On the contrary, the smaller the preset range, the longer the time it takes for multiple cameras to achieve synchronous shooting. In this way, different preset ranges can be selected to adjust the speed of multi-camera synchronization, so that the synchronization speed and synchronization accuracy (i.e., preset range) can be adjusted according to actual needs when shooting video, so as to better meet user needs. At the same time, the synchronous shooting method of the embodiment of the present application does not require manual intervention after starting the synchronization process, and is simple to operate and easy to popularize.

[0115] Based on the method in the above embodiment, the present application also provides a synchronous shooting device, see Fig. 9 , Fig. 9 The present invention provides a schematic diagram of a synchronous shooting device according to an embodiment of the present invention. The synchronous shooting device is deployed in an electronic device including a camera and a display screen. The display screens of multiple electronic devices can be combined to display the video frames captured by the cameras of the multiple electronic devices. Fig. 9 As shown, the synchronous shooting device 900 includes an acquisition module 901, an interpolation module 902 and a sending module 903. The acquisition module 901 is used to acquire the planned time point of synchronous shooting of each video frame when the camera is in a preview shooting or formal shooting mode. The interpolation module 902 is used to perform at least one interpolation operation after the shooting time of the current video frame arrives according to the time offset of the current video frame when it is determined that the time offset of the current video frame shot by the camera is outside the preset range. The interpolation operation is used to adjust the time offset of the video frame shot by the camera after the interpolation operation, wherein the time offset of each video frame is the time difference between the actual time point of shooting each video frame and the planned time point of each video frame. The sending module 903 is used to send a second message to the synchronization server when it is determined that the time offset of the video frame shot before or after the interpolation operation is performed on the camera is within the preset range. The second message is used to indicate that the camera has completed synchronization.

[0116] In one example, the acquisition module 901 is further used to receive a synchronous shooting instruction sent by a synchronous server; the synchronous shooting device 900 also includes a shooting module 904, and the shooting module 904 is used to enable the camera to perform synchronous shooting according to the synchronous shooting instruction.

[0117] In one example, the insertion frame module 902 is specifically used to calculate the shooting time of the insertion frame according to the time offset of the current video frame and insert the insertion frame according to the shooting time of the insertion frame after the shooting time of the current video frame arrives, wherein the shooting time of the insertion frame is greater than the shooting time of each video frame.

[0118] In one example, when the interpolation module 902 calculates the shooting time of the inserted frame according to the time offset of the current video frame, it is specifically used to: first, set a synchronization time point, where the synchronization time point is one of the planned time points of each video frame, and the synchronization time point is greater than the planned time point of the current video frame; then, calculate the first time interval between the actual time point of the current video frame and the synchronization time point; then, calculate the shooting time of the inserted frame according to the first time interval, the shooting time of the current video frame and the second time interval, wherein the second time interval is the time interval between the inserted frame and the synchronization time point.

[0119] In one example, when the frame insertion module 902 calculates the shooting time of the inserted frame according to the time offset of the current video frame, it is specifically used to determine the shooting time of the inserted frame as the time offset of the current video frame when the time offset of the current video frame is greater than the shooting time of each video frame.

[0120] In one example, the acquisition module 901 is specifically used to: first, obtain a first message sent by a synchronization server, the first message including a planned time point for synchronously shooting a target video frame; then, obtain a planned time point for shooting each video frame according to the planned time point of the target video frame and the interval duration between adjacent video frames, wherein the interval duration between adjacent video frames is the same.

[0121] In one example, when the first message includes a preset range, the sending module 903 is also used to determine that the time offset of the video frame captured by the camera after performing a set number of interpolation operations is outside the preset range, and send a feedback message to the synchronization server, and the feedback message is used to indicate that the camera has not completed synchronization; the acquisition module 901 is also used to receive an updated first message sent by the synchronization server, and the updated first message includes the preset range increased by the synchronization server based on the feedback message.

[0122] In an example, the synchronous shooting device 900 further includes a time synchronization module 905, which is used to synchronize the system time with the system time of the synchronization server according to the network time protocol NTP before acquiring the first message sent by the synchronization server.

[0123] It should be understood that the synchronous shooting device is used to execute the method in the above embodiments, such as Figure 7The described method and the corresponding program modules in the device, their implementation principles and technical effects are similar to those described in the above method. The working process of the device can refer to the corresponding process in the above method, and will not be repeated here.

[0124] Based on the method in the above embodiment, the present application embodiment also provides another synchronous shooting device. Fig.10 , Fig.10 Schematic diagram of another synchronous shooting device provided in an embodiment of the present application. The synchronous shooting device can be used to implement the method described in the above method embodiment, such as Figure 7 The method described.

[0125] like Fig.10 As shown, the synchronous shooting device 1000 includes at least one processor 1001, and the at least one processor 1001 can support the synchronous shooting device 1000 to implement the method provided in the embodiment of the present application. The processor 1001 can be a general-purpose processor or a dedicated processor. For example, the processor 1001 may include a central processing unit (CPU) and / or a baseband processor. Among them, the baseband processor can be used to process communication data (for example, determine the target screen terminal), and the CPU can be used to implement corresponding control and processing functions, execute software programs, and process data of software programs.

[0126] Further, the synchronous shooting device 1000 may also include a transceiver 1003 to implement signal input (reception) and output (transmission). The transceiver 1003 may include a communication interface. Optionally, the synchronous shooting device may also include an antenna, which may be used to support the transceiver 1003 to implement the transceiver function of the synchronous shooting device. Optionally, the synchronous shooting device may include one or more memories 1002, on which a program (or instruction or code) is stored, and the program can be run by the processor 1001, so that the processor 1001 executes the method described in the above method embodiment. Optionally, data may also be stored in the memory 1002. Optionally, the processor 1001 may also read the data stored in the memory 1002, and the data may be stored at the same storage address as the program, or the data may be stored at a different storage address from the program.

[0127] The processor 1001 and the memory 1002 may be provided separately or integrated together, for example, integrated on a single board or a system on chip (SOC).

[0128] For a detailed description of the operations performed by the synchronous shooting device in the above-mentioned various possible designs, reference can be made to the description in the embodiment of the synchronous shooting method provided in the embodiment of the present application, and will not be repeated here one by one.

[0129] Based on the synchronous shooting device in the above embodiments, an embodiment of the present application further provides an electronic device, which includes a camera, a display screen and any synchronous shooting device provided in the above embodiments.

[0130] Based on the method in the above embodiment, the embodiment of the present application also provides a synchronization server. Fig.11 , Fig.11 The schematic diagram of the structure of a synchronization server provided in an embodiment of the present application is as follows: Specifically, the synchronization server 1100 may include a sending module 1101 and a receiving module 1102 .

[0131] The sending module 1101 is used to send a first message to multiple electronic devices in a preview shooting or formal shooting mode, the first message includes a planned time point for synchronously shooting a target video frame, wherein each electronic device includes a camera and a display screen, and the display screens of multiple electronic devices are combined to display the video frames shot by the cameras of multiple electronic devices. The receiving module 1102 is used to receive a second message sent by multiple electronic devices respectively, the second message is used to indicate that the time offset of the video frame shot by the camera of the electronic device is within a preset range, the time offset of each video frame is the time difference between the actual time point of shooting each video frame and the planned time point of each video frame, and the planned time point of each video frame is determined by the electronic device according to the planned time point of the target video frame and the interval length of adjacent video frames, wherein the first electronic device performs at least one interpolation operation on the first camera when the time offset of the current video frame shot by the first camera is outside the preset range, and the interpolation operation is used to adjust the time offset of the video frame shot by the first camera, the first electronic device is at least one of the multiple electronic devices, and the first camera is the camera of the first electronic device.

[0132] In one example, the sending module 1101 is further used to send a synchronous shooting instruction to multiple electronic devices, and the synchronous shooting instruction is used to instruct the cameras of multiple electronic devices to perform synchronous shooting. In one example, the first message includes a preset range, and the receiving module 1102 is further used to receive a feedback message sent by the first electronic device, and the feedback message is used to indicate that the first camera has not completed synchronization. The sending module 1102 is also used to increase the preset range based on the feedback message and update the increased preset range in the first message and then send it to multiple electronic devices respectively.

[0133] In one example, the first message includes a preset range, and the synchronization server also includes a processing module 1103, which is used to: first, calculate the sampling number W based on multiple time offsets obtained in advance; then, count the synchronization success rate S after W electronic devices perform frame insertion operations on their respective cameras, S=V / W, where V is the number of second messages received; then, determine that the synchronization success rate S is less than a set threshold, increase the preset range, update the increased preset range in the first message, and then send it to multiple electronic devices respectively.

[0134] In one example, when the processing module 1103 calculates the sampling number W according to the pre-acquired multiple time offsets, it is specifically used to: first receive the time offset of at least one video frame sent by multiple electronic devices or receive the time offset of multiple video frames sent by at least one electronic device to form a sample set; then calculate the mean μ and variance σ of the time offset according to the sample set; then, assuming that the statistical error of the time offset is a Gaussian distribution, calculate the sampling number W=9σ 2 / μ 2 .

[0135] It should be understood that the synchronization server is used to execute the method in the above embodiments, such as Figure 8 The described method and the corresponding program module in the synchronization server, their implementation principles and technical effects are similar to those described in the above method. The working process of the synchronization server can refer to the corresponding process in the above method, and will not be repeated here.

[0136] Based on the method in the above embodiment, the present application embodiment also provides another synchronization server. Fig.12 , Fig.12 This is a schematic diagram of the structure of another synchronization server provided in an embodiment of the present application. The synchronization server can be used to implement the method described in the above method embodiment.

[0137] like Fig.12 As shown, the synchronization server 1200 includes at least one processor 1201, and the at least one processor 1201 can support the synchronization server 1200 to implement the method provided in the embodiment of the present application. The processor 1201 can be a general-purpose processor or a special-purpose processor. For example, the processor 1201 may include a central processing unit (CPU) and / or a baseband processor. Among them, the baseband processor can be used to process communication data (for example, determine the target screen terminal), and the CPU can be used to implement corresponding control and processing functions, execute software programs, and process data of software programs.

[0138] Further, the synchronization server 1200 may also include a transceiver 1203 to implement signal input (reception) and output (transmission). The transceiver 1203 may include a communication interface. Optionally, the synchronization server 1200 may also include an antenna, which can be used to support the transceiver 1203 to implement the transceiver function of the synchronization server 1200. Optionally, the synchronization server 1200 may also include one or more memories 1202, on which a program (or instruction or code) is stored, and the program can be run by the processor 1201, so that the processor 1201 executes the method described in the above method embodiment. Optionally, data may also be stored in the memory 1202. Optionally, the processor 1201 may also read the data stored in the memory 1202, and the data may be stored at the same storage address as the program, or the data may be stored at a different storage address from the program.

[0139] The processor 1201 and the memory 1202 may be provided separately or integrated together, for example, integrated on a single board or a system on chip (SOC).

[0140] For a detailed description of the operations performed by the synchronization server in the above-mentioned various possible designs, please refer to the description in the embodiment of the synchronous shooting method provided in the embodiment of the present application, and will not be repeated here.

[0141] Based on the method in the above embodiment, the present application embodiment also provides a chip. Fig.13 , Fig.13 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig.13 As shown, the chip 1300 includes one or more processors 1301 and an interface circuit 1302. Optionally, the chip 1300 may also include a bus 1303. Among them:

[0142] Processor 1301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in processor 1301 or an instruction in software form. The above-mentioned processor 1301 can be a general-purpose processor, a digital communicator (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods and steps in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The interface circuit 1302 can be used for sending or receiving data, instructions or information, and the processor 1301 can use the data, instructions or other information received by the interface circuit 1302 for processing, and the processing completion information can be sent out through the interface circuit 1302.

[0143] Optionally, the chip also includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM). Optionally, the memory stores executable software modules or data structures, and the processor may perform corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system).

[0144] Optionally, the interface circuit 1302 may be used to output the execution result of the processor 1301 .

[0145] It should be noted that the corresponding functions of the processor 1301 and the interface circuit 1302 can be implemented through hardware design, software design, or a combination of hardware and software, and there is no limitation here.

[0146] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or a software-based instruction in a processor. The chip can be applied to the above electronic device to implement the method provided in the embodiment of the present application.

[0147] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0148] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0149] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0150] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. A synchronous shooting method, applied to a display system; the display system comprises a first electronic device and a second electronic device; characterized in that: The first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; the first electronic device and the second electronic device communicate with a synchronization server; The synchronization server is used to realize synchronization between the first electronic device and the second electronic device; The first display screen can play a first picture according to the first data captured by the first camera; The second display screen can play a second picture according to the second data captured by the second camera; The first picture and the second picture can be displayed in combination; the method comprises: The first electronic device receives a first message from the synchronization server; the first message includes a first scheduled time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device to shoot at the first planned time point T M ', start shooting a video frame; The first electronic device M ', and the interval time T between any two video frames shot by the first electronic device, to obtain the second planned time point T1'; wherein the second planned time point T1' is located at [T1, T1+T]; T1 is the start time point of shooting the first current video frame; the interval time T between any two video frames shot by the first electronic device is equal; The first electronic device obtains a first time offset according to T1 and T1'; After the first time offset is greater than a first preset threshold and after the first current video frame is shot, the first electronic device performs P frame insertion operations; After performing the frame insertion operation P times, the first electronic device sends a second message to the synchronization server; the second message is used to indicate that the first electronic device has completed synchronization with the synchronization server; The second electronic device receives a first message from the synchronization server; the first message includes a first scheduled time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the second electronic device to shoot at the first planned time point T M ', start shooting a video frame; The second electronic device M ', and the interval time T between any two video frames shot by the second electronic device, to obtain a third planned time point T1"; wherein the third planned time point T1" is located at [T0, T0+T]; T0 is the start time point of shooting the second current video frame; the interval time T between any two video frames shot by the second electronic device is equal; The second electronic device obtains a second time offset according to T0 and T1'; After the second time offset is greater than a second preset threshold and after the shooting of the second current video frame is completed, the second electronic device performs Q frame insertion operations; After performing Q frame insertion operations, the second electronic device sends a third message to the synchronization server; the third message is used to indicate that the second electronic device has completed synchronization with the synchronization server; After executing the frame insertion operation P times, the first electronic device performs the frame insertion operation at the first planned time point T M ', start shooting a video frame; after performing Q frame insertion operations, the second electronic device at the first planned time point T M ', start shooting a video frame; P, Q are positive integers greater than or equal to 1.

2. The synchronous shooting method according to claim 1, characterized in that: The display system also includes the synchronization server.

3. A synchronous shooting method, applied to a display system; the display system comprises a first electronic device and a second electronic device; characterized in that: The first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; the first electronic device communicates with the second electronic device; the first display screen can play a first picture according to the first data captured by the first camera; The second display screen can play a second picture according to the second data captured by the second camera; The first picture and the second picture provided by the second electronic device can be displayed in combination; the method includes: The first electronic device receives a first message from the second electronic device; the first message includes a first scheduled time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device to shoot at the first planned time point T M ', start shooting a video frame; The first electronic device M ', and the interval time T between any two video frames shot by the first electronic device, to obtain the second planned time point T1'; wherein the second planned time point T1' is located at [T1, T1+T]; T1 is the start time point of shooting the first current video frame; the interval time T between any two video frames shot by the first electronic device is equal; The first electronic device obtains a first time offset according to T1 and T1'; After the first time offset is greater than a first preset threshold and after the first current video frame is shot, the first electronic device performs P frame insertion operations; After performing the frame insertion operation P times, the first electronic device sends a second message to the second electronic device; the second message is used to indicate that the first electronic device and the second electronic device have completed synchronization; The second electronic device is at the first planned time point T M 'Start capturing a video frame; After executing P frame insertion operations, the first electronic device starts shooting a video frame at the first planned time point; P is a positive integer greater than or equal to 1.

4. A synchronous shooting method, applied to a synchronization server; the synchronization server communicates with a first electronic device and a second electronic device; the first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; characterized in that: The first display screen can play a first picture according to the first data captured by the first camera; the second display screen can play a second picture according to the second data captured by the second camera; The first picture and the second picture can be displayed in combination; The synchronization server is used to synchronize the first electronic device with the second electronic device; the method includes: The synchronization server sends a first message to the first electronic device and the second electronic device; the first message includes a first scheduled time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device and the second electronic device to start shooting a video frame at the first planned time point T M ', start shooting a video frame; wherein the first electronic device according to the first planned time point T M ', and the interval time T between any two video frames shot by the first electronic device, obtain the second planned time point T1'; wherein, the second planned time point T1' is located at [T1, T1+T]; T1 is the start time point of shooting the first current video frame; the interval time T between any two video frames shot by the first electronic device is equal; wherein, the first electronic device obtains a first time offset according to T1 and T1'; wherein, after the first time offset is greater than a first preset threshold value and after the shooting of the first current video frame is completed, the first electronic device performs P times of interpolation operation; wherein, the second electronic device performs P times of interpolation operation according to the first planned time point T M ', and the interval time T between any two video frames shot by the second electronic device, to obtain a third planned time point T1"; wherein the third planned time point T1" is located at [T0, T0+T]; T0 is the start time point of shooting the second current video frame; the interval time T between any two video frames shot by the second electronic device is equal; wherein the second electronic device obtains a second time offset according to T0 and T1", wherein after the second time offset is greater than a second preset threshold value and after the shooting of the second current video frame is completed, the second electronic device performs Q frame insertion operations; receiving a second message from the first electronic device; the second message is used to indicate that the first electronic device has completed synchronization with the synchronization server; wherein, after performing P times of interpolation operations, the first electronic device sends the second message to the synchronization server, and the first electronic device completes synchronization at the first scheduled time point T M ', start shooting a video frame; receiving a third message from the second electronic device; the third message is used to indicate that the second electronic device has completed synchronization with the synchronization server; wherein, after performing Q times of interpolation operations, the second electronic device sends the third message to the synchronization server, and the second electronic device completes synchronization at the first scheduled time point T M ', start shooting a video frame; P, Q are positive integers greater than or equal to 1.

5. A synchronous shooting method, applied to a second electronic device; the second electronic device communicates with a first electronic device; the first electronic device includes a first camera and a first display screen; the second electronic device includes a second camera and a second display screen; characterized in that: The first display screen can play a first picture according to the first data captured by the first camera; The second display screen can play a second picture according to the second data captured by the second camera; The first picture and the second picture can be displayed in combination; the method comprises: The second electronic device sends a first message to the first electronic device; the first message includes a first planned time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device to shoot at the first planned time point T M ', start shooting a video frame; wherein the first electronic device according to the first planned time point T M ', and the interval time T between any two video frames shot by the first electronic device, to obtain the second planned time point T1'; wherein the second planned time point T1' is located at [T1, T1+T]; T1 is the start time point of shooting the first current video frame; the interval time T between any two video frames shot by the first electronic device is equal; wherein the first electronic device obtains the first time offset according to T1 and T1'; wherein after the first time offset is greater than the first preset threshold value and after the shooting of the first current video frame is completed, the first electronic device performs P times of interpolation operation; receiving a second message from the first electronic device; the second message is used to indicate that the first electronic device and the second electronic device have completed synchronization; wherein after performing P times of frame insertion operations, the first electronic device sends the second message to the second electronic device; The second electronic device is at the first planned time point T M 'Start capturing a video frame.

6. A synchronous shooting method, applied to a first electronic device; the first electronic device comprises a first camera and a first display screen; characterized in that: The first display screen can play a first picture according to the first data captured by the first camera; the first picture and a second picture provided by a second electronic device can be displayed in combination; the second electronic device includes a second camera and a second display screen; the second display screen can play the second picture according to the second data captured by the second camera; the first electronic device communicates with a synchronization server; The synchronization server also communicates with the second electronic device, and the synchronization server is used to synchronize the first electronic device with the second electronic device; the method includes: The first electronic device receives a first message from the synchronization server; the first message includes a first scheduled time point T M ', and at the first planned time point T M 'Start shooting a video frame; the first message is used to instruct the first electronic device to shoot at the first planned time point T M ', start shooting a video frame; The first electronic device M ', and the interval time T between any two video frames shot by the first electronic device, to obtain the second planned time point T1'; wherein the second planned time point T1' is located at [T1, T1+T]; T1 is the start time point of shooting the first current video frame; the interval time T between any two video frames shot by the first electronic device is equal; The first electronic device obtains a first time offset according to T1 and T1'; After the first time offset is greater than a first preset threshold and after the first current video frame is shot, the first electronic device performs P frame insertion operations; After performing the frame insertion operation P times, the first electronic device sends a second message to the synchronization server; the second message is used to indicate that the first electronic device has completed synchronization with the synchronization server; After executing the frame insertion operation P times, the first electronic device performs the frame insertion operation at the first planned time point T M ', start shooting a video frame; P is a positive integer greater than or equal to 1.

7. A first electronic device, characterized in that: The first electronic device includes a first camera, a first display screen, a memory, a processor and a computer program; wherein the computer program is stored in the memory; when the computer program is executed by the processor, the first electronic device implements the synchronous shooting method as described in claim 6.

8. A second electronic device, characterized in that: The second electronic device includes a second camera, a second display screen, a memory, a processor and a computer program; wherein the computer program is stored in the memory; when the computer program is executed by the processor, the second electronic device implements the synchronous shooting method as described in claim 5.

9. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program runs on the first electronic device, the first electronic device implements the synchronous shooting method as claimed in claim 6.

10. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program runs on the second electronic device, the second electronic device implements the synchronous shooting method as claimed in claim 5.

Images