Video fusion method, device, terminal equipment and storage medium

Abstract

The embodiment of the present application provides a video fusion method, apparatus, terminal device and storage medium, the video fusion method comprising: obtaining recording capability information of N terminal devices interconnected with a first terminal device, determining recording parameters of each terminal device in the N terminal devices and recording parameters of the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device; sending the recording parameters of each terminal device to each terminal device; when the first terminal device and the N terminal devices receive a joint recording instruction, performing video recording based on the recording parameters of the first terminal device to obtain a first video frame set; obtaining N video frame sets obtained by performing video recording based on the recording parameters of each terminal device by the N terminal devices; fusing the first video frame set with the N video frame sets to obtain a fused video. The embodiment of the present application can improve the recording effect of the fused video.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video fusion method, apparatus, terminal equipment and storage medium. Background Art

[0002] At present, some terminal devices (such as mobile phones) can support slow-motion shooting functions, such as supporting video recording at 120 or 240 frames per second (FPS), which is equivalent to capturing 5 to 10 times slower slow-motion compared to normal 24FPS video recording. However, for some scenes, the ability of terminal devices to capture slow-motion wonderful moments is still very weak. For example, the moment when a high-speed steel ball penetrates a Sprite can or a beer bottle may only take 0.05 seconds. If the terminal device only slows down the recording by 5 to 10 times, the wonderful moment may only be captured in 2 to 3 frames, and the video recording effect is poor. Summary of the invention

[0003] The embodiments of the present application provide a video fusion method, apparatus, terminal device and storage medium, which can improve the video recording effect.

[0004] A first aspect of an embodiment of the present application provides a video fusion method, including:

[0005] The first terminal device obtains recording capability information of N terminal devices interconnected with the first terminal device, and determines recording parameters of each terminal device in the N terminal devices and recording parameters of the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, where N is a positive integer;

[0006] The first terminal device sends recording parameters of each terminal device to each terminal device;

[0007] When the first terminal device and the N terminal devices receive the joint recording instruction, the first terminal device performs video recording based on the recording parameters of the first terminal device to obtain a first video frame set;

[0008] The first terminal device obtains N video frame sets obtained by the N terminal devices performing video recording based on the recording parameters of the respective terminal devices; wherein the N video frame sets correspond to the N terminal devices one by one, the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, and the acquisition time point of at least one video frame in the N video frame sets is within the acquisition time range of the first video frame set;

[0009] The first terminal device fuses the first video frame set and the N video frame sets to obtain a fused video.

[0010] A second aspect of an embodiment of the present application provides a video fusion device, including:

[0011] An obtaining unit, used to obtain recording capability information of N terminal devices interconnected with the first terminal device;

[0012] a determining unit, configured to determine recording parameters of each of the N terminal devices and the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, where N is a positive integer;

[0013] A sending unit, used for sending the recording parameters of each terminal device to each terminal device;

[0014] a recording unit, configured to perform video recording based on a recording parameter of the first terminal device to obtain a first video frame set when the first terminal device and the N terminal devices receive a joint recording instruction;

[0015] The obtaining unit is further used to obtain N video frame sets obtained by the N terminal devices performing video recording based on the recording parameters of the respective terminal devices; wherein the N video frame sets correspond to the N terminal devices one by one, the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, and the acquisition time point of at least one video frame in the N video frame sets is within the acquisition time range of the first video frame set;

[0016] The fusion unit is used to fuse the first video frame set and the N video frame sets to obtain a fused video.

[0017] A third aspect of an embodiment of the present application provides a terminal device, including a processor and a memory, the memory being used to store a computer program, the computer program including program instructions, and the processor being configured to call the program instructions to execute the step instructions in the first aspect of the embodiment of the present application.

[0018] The fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the above-mentioned computer-readable storage medium stores a computer program for electronic data exchange, and the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the step instructions in the first aspect of the embodiment of the present application.

[0019] The fifth aspect of the embodiments of the present application provides a computer program product, wherein the above-mentioned computer program product includes a computer program, and the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the step instructions in the first aspect of the embodiments of the present application.

[0020] In an embodiment of the present application, a first terminal device obtains recording capability information of N terminal devices interconnected with the first terminal device, and determines recording parameters of each terminal device in the N terminal devices and recording parameters of the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, where N is a positive integer; the first terminal device sends the recording parameters of each terminal device to each terminal device; when the first terminal device and the N terminal devices receive a joint recording instruction, the first terminal device performs video recording based on the recording parameters of the first terminal device to obtain a first video frame set; the first terminal device obtains N video frame sets obtained by the N terminal devices performing video recording based on the recording parameters of the respective terminal devices; wherein the N video frame sets correspond to the N terminal devices one-to-one, and the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, and there is at least one video frame in the N video frame sets whose acquisition time point is within the acquisition time range of the first video frame set; the first terminal device fuses the first video frame set and the N video frame sets to obtain a fused video. The video fusion method of the embodiment of the present application fuses a first video frame set recorded by a first terminal device and N video frame sets recorded by N terminal devices to obtain a fused video. The video frame sets recorded by at least two terminal devices can be fused. Since the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, there is at least one video frame in the N video frame sets whose acquisition time point is within the acquisition time range of the first video frame set, so that more video frames within the acquisition time range of the first video frame set participate in the fusion process, thereby improving the frame rate of the fused video, and further improving the recording effect of the fused video. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1It is a flowchart of a video fusion method provided in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of another video fusion method provided in an embodiment of the present application;

[0024] Figure 3 It is a comparative schematic diagram of fusion using cutting and splicing and fusion not using cutting and splicing provided in an embodiment of the present application;

[0025] Figure 4 This is a schematic diagram of a process for performing slow-motion video fusion using two mobile phones as an example provided in an embodiment of the present application;

[0026] Figure 5 It is a structural schematic diagram of a video fusion device provided in an embodiment of the present application;

[0027] Figure 6 It is a structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0030] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0031] The terminal device involved in the embodiments of the present application is a terminal with at least one camera that can realize video recording. It can be a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an on-board unit (OBU), a wearable device (for example, a watch, a bracelet, a smart helmet, etc.), a smart home device (rice cooker, audio, home butler device, etc.), an augmented reality (AR) / virtual reality (VR) device, etc.

[0032] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a video fusion method provided in an embodiment of the present application. Figure 1 As shown, the method may include the following steps.

[0033] 101. A first terminal device obtains recording capability information of N terminal devices interconnected with the first terminal device, and determines recording parameters of each of the N terminal devices and the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, where N is a positive integer.

[0034] In an embodiment of the present application, before executing step 101, the first terminal device and N terminal devices can be interconnected. That is, the first terminal device and the N terminal devices are connected to each other (wired connection or wireless connection). Wired connection may include universal serial bus (USB) connection, OTG connection, etc. Wireless connection may include: wifi connection, Bluetooth connection, etc. After the first terminal device is interconnected with the N terminal devices, the first terminal device and the N terminal devices can exchange data. For example, the first terminal device can scan and interconnect with the N terminal devices to discover the recording capability information of the other party.

[0035] The recording capability information of the terminal device refers to the capability information of video recording supported by the terminal device. The recording capability information may include camera capability information, and the camera capability information may include the maximum resolution and maximum frame rate of the recorded video. For example, the maximum resolution supported by the first terminal device for recording is 1080P, and the maximum frame rate is 120 frames per second (FPS). The maximum resolution supported by the second terminal device (one of the N terminal devices) for recording is 2K, and the maximum frame rate is 120FPS. The recording capability information may also include the supported white balance capability, fill light capability, and focus capability, etc.

[0036] In one embodiment, the first terminal device may send a recording capability information reporting request to N terminal devices, and receive the recording capability information reported by the N terminal devices. The recording capability information of the N terminal devices may include the recording capability information of each of the N terminal devices, and each of the N terminal devices may report its own recording capability information to the first terminal device.

[0037] In another embodiment, after the first terminal device and the N terminal devices can be interconnected, the N terminal devices can actively report their own recording capability information to the first terminal device.

[0038] The recording parameters may include a unified resolution, a configured frame rate, and a start recording time point. The start recording time points in the recording parameters of each of the N terminal devices are different from each other, and the start recording time points in the recording parameters of each of the N terminal devices and the start recording time points in the recording parameters of the first terminal device are different from each other. The first terminal device may determine the recording parameters of each of the N terminal devices and the recording parameters of the first terminal device based on the recording capability information of the N terminal devices and the recording capability information of the first terminal device. Among them, the unified resolution may select the minimum value of the maximum resolution supported by the first terminal device and the N terminal devices. The configuration frame rate and the start recording time point may follow the following principle: the acquisition time points of each video frame set recorded by the first terminal device and the N video frame set recorded by the N terminal devices may be as evenly distributed as possible without reducing the maximum frame rate of each terminal device as much as possible. For example, taking N=1 as an example, the N terminal devices include the second terminal device. The maximum resolution supported by the first terminal device for recording is 1080P, and the maximum frame rate is 120FPS. The maximum resolution supported by the second terminal device for recording is 1080P, and the maximum frame rate is 120FPS. The recording parameters of the first terminal device can be set to include: the uniform resolution of the first terminal device is 1080P, the configured frame rate of the first terminal device is 120FPS, and the starting recording time point of the first terminal device is 0 milliseconds (ms); the recording parameters of the second terminal device can be set to include: the uniform resolution of the second terminal device is 1080P, the configured frame rate of the second terminal device is 120FPS, and the starting recording time point of the second terminal device is 4.16ms.

[0039] Optionally, the recording parameters also include white balance parameters, fill light parameters and focus parameters. In order to ensure that the effects of the video frames recorded by the first terminal device and each of the N terminal devices are as consistent as possible, each terminal device can be set to use the same white balance parameters, the same fill light parameters and the same focus parameters when recording the video.

[0040] Optionally, the recording capability information includes a maximum resolution and a maximum frame rate, and the recording parameters include a uniform resolution, a configured frame rate, and a start recording time point; in step 101, the first terminal device determines the recording parameters of each of the N terminal devices and the recording parameters of the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, which may include the following steps:

[0041] (11) The first terminal device selects the lowest resolution from the maximum resolution supported by the N terminal devices and the maximum resolution supported by the first terminal device as the unified resolution of the N terminal devices and the first terminal device;

[0042] (12) The first terminal device determines the configured frame rate and the starting recording time point of the first terminal device and the configured frame rate and the starting recording time point of each terminal device among the N terminal devices based on the maximum frame rate supported by the N terminal devices and the maximum frame rate supported by the first terminal device; the configured frame rate of the first terminal device is less than or equal to the maximum frame rate supported by the first terminal device, and the configured frame rate of each terminal device is less than or equal to the maximum frame rate of each terminal device.

[0043] In an embodiment of the present application, in order to ensure the effect of the first terminal device and N terminal devices jointly recording a video, it is necessary to unify the resolutions of the first terminal device and the N terminal devices during video recording. Generally speaking, a terminal device that supports high resolution can be compatible with low resolution, and a terminal device that supports low resolution cannot be compatible with high resolution. For example, a terminal device that supports 1080P can be compatible with a resolution less than 1080P (for example, 720P). The lowest resolution can be selected from the maximum resolution supported by the N terminal devices and the maximum resolution supported by the first terminal device as the unified resolution of the N terminal devices and the first terminal device, ensuring that the first terminal device and the N terminal devices can both record videos at the same unified resolution.

[0044] Terminal devices that support high frame rates can be compatible with low frame rates. For example, terminal devices that support 120FPS can be compatible with frame rates less than 120FPS (for example, 60FPS).

[0045] In the embodiment of the present application, if the first terminal device and the N terminal devices use the maximum frame rate they support, by reasonably configuring the start recording time point of each terminal device, the first video frame set recorded by the first terminal device and the N video frame sets recorded by the N terminal devices can be evenly distributed according to the acquisition time point, then the configured frame rate of the first terminal device can be determined to be the maximum frame rate it supports, the configured frame rate of each of the N terminal devices can be determined to be the maximum frame rate it supports, and the start recording time point of the first terminal device and the start recording time point of each of the N terminal devices can be determined. For example, taking N=2 as an example, the N terminal devices include the second terminal device and the third terminal device. The maximum resolution supported by the first terminal device for recording is 1080P, and the maximum frame rate is 120FPS. The maximum resolution supported by the second terminal device and the third terminal device for recording is 1080P, and the maximum frame rate is 60FPS. Then, the recording parameters of the first terminal device can be determined to include: the unified resolution of the first terminal device is 1080P, the configured frame rate of the first terminal device is 120FPS, and the starting recording time point of the first terminal device is 0 milliseconds (ms); the recording parameters of the second terminal device can be determined to include: the unified resolution of the second terminal device is 1080P, the configured frame rate of the second terminal device is 60FPS, and the starting recording time point of the second terminal device is 4.16ms; the recording parameters of the third terminal device can be determined to include: the unified resolution of the third terminal device is 1080P, the configured frame rate of the third terminal device is 60FPS, and the starting recording time point of the third terminal device is 12.5ms. The first terminal device performs video recording based on the recording parameters of the first terminal device, and obtains the acquisition time points of each video frame in the first video frame set as: 0ms, 8.33ms, 16.66ms…. The second terminal device performs video recording based on the recording parameters of the second terminal device, and obtains that the acquisition time points of each video frame in the second video frame set are: 4.16ms, 20.83ms, 37.5ms... The third terminal device performs video recording based on the recording parameters of the third terminal device, and obtains that the acquisition time points of each video frame in the third video frame set are: 12.5ms, 29.16ms, 45.83ms...

[0046] It should be noted that, when the maximum frame rate supported by the first terminal device is the same as the maximum frame rate supported by each of the N terminal devices, the first video frame set recorded by the first terminal device and the N video frame sets recorded by the N terminal devices can be evenly distributed according to the acquisition time point by reasonably configuring the start recording time point of each terminal device. For example, if the maximum frame rate supported by the first terminal device and the N terminal devices is f1, then the start recording time points of the N+1 terminal devices (the first terminal device and the N terminal devices) are: 0, 1 / [f1*(N+1)], 2 / [f1*(N+1)], ..., N / [f1*(N+1)]. 0, 1 / [f1*(N+1)], 2 / [f1*(N+1)], ..., N / [f1*(N+1)] can be assigned to the N+1 terminal devices, and there is no limitation on which terminal device uses which start recording time point, and it can be randomly assigned or assigned according to a rule.

[0047] When the maximum frame rate supported by the first terminal device is the same as the maximum frame rate supported by each of the N terminal devices, the frame rate of the fused video can reach (N+1) times the above maximum frame rate, which can maximize the frame rate of the fused video and thereby improve the recording effect of the fused video.

[0048] If the first terminal device and the N terminal devices use their respective maximum supported frame rates, no matter how the start recording time points of the various terminal devices are configured, the first video frame set recorded by the first terminal device and the N video frame sets recorded by the N terminal devices cannot be evenly distributed according to the acquisition time points. In this case, a frame rate priority strategy or a frame rate uniform strategy (frame interval uniform strategy) can be adopted.

[0049] The frame rate priority strategy is that the first terminal device and the N terminal devices use the maximum frame rate supported by each of them, and by configuring the start recording time point of each terminal device, the first video frame set recorded by the first terminal device and the N video frame sets recorded by the N terminal devices can be distributed as close to the uniform distribution as possible according to the acquisition time point. For example, taking N=1 as an example, the N terminal devices include the second terminal device. Among them, the maximum resolution supported by the first terminal device for recording is 1080P, the maximum frame rate is 120FPS, and the maximum resolution supported by the second terminal device for recording is 1080P, and the maximum frame rate is 60FPS. Then, the recording parameters of the first terminal device can be determined to include: the unified resolution of the first terminal device is 1080P, the configured frame rate of the first terminal device is 120FPS, and the start recording time point of the first terminal device is 0 milliseconds (ms); the recording parameters of the second terminal device can be determined to include: the unified resolution of the second terminal device is 1080P, the configured frame rate of the second terminal device is 60FPS, and the start recording time point of the second terminal device is 4.16ms. By adopting a frame rate priority strategy, the frame rate of each terminal device can be fully utilized, so that the first video frame set recorded by the first terminal device and the N video frame sets recorded by N terminal devices can contain more video frames, thereby improving the video fusion effect.

[0050] The frame rate unification strategy (uniform frame interval strategy) is that the first terminal device and N terminal devices adopt a unified frame rate (select the lowest frame rate from the maximum frame rate supported by the N terminal devices and the maximum frame rate supported by the first terminal device as the unified frame rate of the N terminal devices and the first terminal device), and configure the start recording time point of each terminal device so that the first video frame set recorded by the first terminal device and the N video frame sets recorded by the N terminal devices can be evenly distributed according to the acquisition time point. For example, taking N=3 as an example, the N terminal devices include the second terminal device, the third terminal device and the fourth terminal device. The maximum resolution supported by the first terminal device, the second terminal device, and the third terminal device for recording are all 1080P, and the maximum frame rate is 120FPS. The maximum resolution supported by the fourth terminal device for recording is 1080P, and the maximum frame rate is 150FPS. Then, the recording parameters of the first terminal device can be determined to include: the unified resolution of the first terminal device is 1080P, the configured frame rate of the first terminal device is 120FPS, and the starting recording time point of the first terminal device is 0 milliseconds (ms); the recording parameters of the second terminal device can be determined to include: the unified resolution of the second terminal device is 1080P, the configured frame rate of the second terminal device is 120FPS, and the starting recording time point of the second terminal device is 2.08ms; the recording parameters of the third terminal device can be determined to include: the unified resolution of the third terminal device is 1080P, the configured frame rate of the third terminal device is 120FPS, and the starting recording time point of the third terminal device is 4.16ms. The recording parameters of the fourth terminal device can be determined to include: the unified resolution of the fourth terminal device is 1080P, the configured frame rate of the fourth terminal device is 120FPS, and the starting recording time point of the fourth terminal device is 6.24ms. The first terminal device performs video recording based on the recording parameters of the first terminal device, and obtains the acquisition time points of each video frame in the first video frame set as: 0ms, 8.33ms, 16.66ms…. The second terminal device performs video recording based on the recording parameters of the second terminal device, and obtains the acquisition time points of each video frame in the second video frame set as: 2.08ms, 10.41ms, 18.74ms…. The third terminal device performs video recording based on the recording parameters of the third terminal device, and obtains the acquisition time points of each video frame in the third video frame set as: 4.16ms, 12.5ms, 20.82ms…. The fourth terminal device performs video recording based on the recording parameters of the fourth terminal device, and obtains the acquisition time points of each video frame in the fourth video frame set as: 6.24ms, 14.57ms, 22.9ms…. By adopting a unified frame rate strategy, the acquisition time points of each video frame in the first video frame set recorded by the first terminal device and the N video frame sets recorded by N terminal devices can be evenly distributed, the frame intervals are uniform, and the fused video will not have any inharmonious pictures, thereby improving the video fusion effect.

[0051] 102. The first terminal device sends recording parameters of each terminal device to each terminal device.

[0052] In an embodiment of the present application, the first terminal device may send the recording parameters of each terminal device in the N terminal devices to the corresponding terminal device. For example, taking N=3 as an example, the N terminal devices include a second terminal device, a third terminal device, and a fourth terminal device. The first terminal device may send the recording parameters of the second terminal device to the second terminal device, send the recording parameters of the third terminal device to the third terminal device, and send the recording parameters of the fourth terminal device to the fourth terminal device.

[0053] The recording parameters may include at least the start recording time point. When the recording capability information of the first terminal device and each of the N terminal devices is the same, other recording parameters (such as resolution, frame rate, etc.) are the same, and the first terminal device only needs to send the recording parameters including the respective start recording time points to the N terminal devices, without sending the recording parameters such as resolution and frame rate to the N terminal devices.

[0054] 103. When the first terminal device and N terminal devices receive a joint recording instruction, the first terminal device performs video recording based on the recording parameters of the first terminal device to obtain a first video frame set.

[0055] In the embodiment of the present application, the joint recording instruction can be initiated by the user on the first terminal device, or can be initiated by the user on any terminal device among the N terminal devices. When the user initiates the joint recording instruction on the first terminal device, the first terminal device can send the joint recording instruction to the N terminal devices at the same time. At this time, the first terminal device can perform video recording according to the recording parameters of the first terminal device determined in step 101 to obtain a first video frame set. The N terminal devices can perform video recording according to the recording parameters of each of the N terminal devices determined in step 101 to obtain N video frame sets. For example, taking N=3 as an example, the N terminal devices include a second terminal device, a third terminal device, and a fourth terminal device. The maximum resolution supported by the first terminal device, the second terminal device, the third terminal device and the fourth terminal device for recording are all 1080P, and the maximum frame rate is 120FPS. Then, the recording parameters of the first terminal device can be determined to include: the uniform resolution of the first terminal device is 1080P, the configured frame rate of the first terminal device is 120FPS, and the starting recording time point of the first terminal device is 0 milliseconds (ms); the recording parameters of the second terminal device can be determined to include: the uniform resolution of the second terminal device is 1080P, the configured frame rate of the second terminal device is 120FPS, and the starting recording time point of the second terminal device is 2.08ms; the recording parameters of the third terminal device can be determined to include: the uniform resolution of the third terminal device is 1080P, the configured frame rate of the third terminal device is 120FPS, and the starting recording time point of the third terminal device is 4.16ms. The recording parameters of the fourth terminal device can be determined to include: the uniform resolution of the fourth terminal device is 1080P, the configured frame rate of the fourth terminal device is 120FPS, and the starting recording time point of the fourth terminal device is 6.24ms. The first terminal device performs video recording based on the recording parameters of the first terminal device, and obtains the acquisition time points of each video frame in the first video frame set as: 0ms, 8.33ms, 16.66ms…. The second terminal device performs video recording based on the recording parameters of the second terminal device, and obtains the acquisition time points of each video frame in the second video frame set as: 2.08ms, 10.41ms, 18.74ms…. The third terminal device performs video recording based on the recording parameters of the third terminal device, and obtains the acquisition time points of each video frame in the third video frame set as: 4.16ms, 12.5ms, 20.82ms…. The fourth terminal device performs video recording based on the recording parameters of the fourth terminal device, and obtains the acquisition time points of each video frame in the fourth video frame set as: 6.24ms, 14.57ms, 22.9ms….

[0056] It should be noted that before the first terminal device performs video recording based on the recording parameters of the first terminal device and obtains the first video frame set, the system time of the first terminal device and the system time of the N terminal devices have been synchronized.

[0057] The first terminal device and the N terminal devices may start video recording immediately upon receiving the joint recording instruction, or may start video recording at the time point indicated by the joint recording instruction after receiving the joint recording instruction. For example, the joint recording instruction may indicate that video recording should start 3 seconds after the current time. Then the first terminal device may start video recording immediately after 3s, the second terminal device may start video recording after 3s+2.08ms, the third terminal device may start video recording after 3s+4.16ms, and the fourth terminal device may start video recording after 3s+6.24ms.

[0058] 104. The first terminal device obtains N video frame sets obtained by N terminal devices performing video recording based on the recording parameters of each terminal device; wherein the N video frame sets correspond to the N terminal devices one-to-one, acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, and there is at least one video frame in the N video frame sets whose acquisition time point is within the acquisition time range of the first video frame set.

[0059] In an embodiment of the present application, after N terminal devices perform video recording based on their respective recording parameters to obtain a set of video frames, they can send their respective recorded video frame sets to the first terminal device. When the video frames are fused, only one video frame is required to participate in the fusion at the same acquisition time point. When the first terminal device and the N terminal devices perform joint recording, if multiple video frames are acquired at the same time point, it is necessary to make a choice, resulting in wasted video frames. The acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other. The first video frame set and the N video frame sets acquired by the first terminal device and the N terminal devices can participate in the fusion process, ensuring that as many video frames as possible can participate in the fusion process per unit time, and the frame rate of the fused video can be maximized.

[0060] The first video frame set may include at least two video frames, and each of the N video frame sets may include at least one video frame.

[0061] The acquisition time range of the first video frame set includes: the acquisition time range consisting of the earliest and latest acquisition time points of at least two video frames included in the first video frame set. For example, the acquisition time points of the video frames included in the first video frame set are: 0ms, 8.33ms, 16.66ms, ... 1000ms, respectively, then the acquisition time range of the first video frame set is 0 to 1000ms.

[0062] 105. The first terminal device fuses the first video frame set and N video frame sets to obtain a fused video.

[0063] In an embodiment of the present application, the first video frame set and the N video frame sets may be subjected to video fusion processing in the order of acquisition time points to obtain a fused video. The fusion processing may include video frame preprocessing and video frame fusion processing. Before the N video frame sets participate in the video frame fusion processing, the video frame preprocessing may be performed, so that the image quality of the video frames in the N video frame sets after preprocessing is as consistent as possible with that of the video frames in the first video frame set, thereby improving the effect of the fused video. The fusion processing may be performed using an artificial intelligence algorithm, which may be implemented specifically using the artificial intelligence and neural network of a chip.

[0064] In an embodiment of the present application, a first video frame set recorded by a first terminal device and N video frame sets recorded by N terminal devices are fused to obtain a fused video. Video frame sets recorded by at least two terminal devices can be fused. Since the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, there is at least one video frame in the N video frame sets whose acquisition time point is within the acquisition time range of the first video frame set, so that more video frames within the acquisition time range of the first video frame set participate in the fusion process, thereby improving the frame rate of the fused video, and further improving the recording effect of the fused video.

[0065] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of another video fusion method provided in an embodiment of the present application. Figure 2 As shown, the method may include the following steps.

[0066] 201. A first terminal device sends a joint shooting request to N terminal devices.

[0067] In an embodiment of the present application, before the first terminal device sends a joint shooting request to N terminal devices, it can establish a communication connection with the N terminal devices. The first terminal device can discover the N terminal devices by means of interconnection scanning and establish a communication connection with the N terminal devices. The interconnection scanning method is a method of realizing interconnection, and it is also a method of discovering capabilities. For example, the interconnection scanning method is a method of scanning a QR code, or it can be a method of Bluetooth scanning or WiFi scanning. The other party is discovered by Bluetooth and WiFi. If the other party is found to have the ability to support joint recording, the first terminal device will initiate a request, and after the other party agrees, the interconnection can be realized. Interconnection is equivalent to forming a network. After the first terminal device is interconnected with the N terminal devices, the first terminal device can interact with the N terminal devices through UI to determine which terminal device is placed in which position. For example, a user has three mobile phones. When networking, you can specify which mobile phone is placed on the left, the middle, and the right. After interconnection and networking, joint recording can be achieved. After interconnection and networking, the three mobile phones can communicate with each other about the camera capabilities and their positions (which phone is on the left, which is in the middle and right, which is on top and which is on the bottom). The capability difference of each phone's camera can be obtained to better allocate the frame rate and number of frames for each phone, and better choose a resolution, thereby achieving better joint recording.

[0068] The first terminal device of the embodiment of the present application is the main shooting device, and the N terminal devices are auxiliary shooting devices. Taking a mobile phone as an example, the terminal device can open the recording software of mobile phone 1 (the first terminal device) to select the joint slow-motion mode, and set mobile phone 1 as the main shooting device and the other mobile phones as auxiliary shooting devices.

[0069] The embodiment of the present application can scan other mobile phones through high-speed interconnection such as high-speed wifi or USB. The auxiliary shooting mobile phone sends a joint shooting request (for example, a slow-motion continuous shooting request) to other mobile phones, and can only execute subsequent steps 202 to 208 after the other mobile phone agrees. If the first terminal device and the N terminal devices are devices under the same user account, the N terminal devices can directly return consent after receiving the joint shooting request. For example, the first terminal device can be a mobile phone under the same user account, and the N terminal devices can be smart home cameras under the same user account.

[0070] The embodiments of the present application can utilize the interconnection capabilities of mobile phones for scanning. For example, an OPPO mobile phone can detect screen projection with video connection and can detect car keys. Supporting joint slow-motion shooting is also a capability of the mobile phone. Mobile phones with this capability support being discovered by nearby networks and establishing connections to open up collaborative data channels.

[0071] In one possible implementation, the first terminal device and the N terminal devices may be multiple mobile phones of the same model. Since the camera capabilities and effects of multiple mobile phones of the same model are basically the same, joint shooting through multiple mobile phones of the same model facilitates subsequent fusion processing (no consistency processing or a small amount of consistency processing), improves the speed of fusion processing, and can also improve the effect of fused video.

[0072] Optionally, during the video recording process, the relative positions of the first terminal device and any two of the N terminal devices remain unchanged.

[0073] For example, a mobile terminal can fix multiple mobile phones through a mobile phone holder. The mobile phone holder can be a mobile phone video holder. Multiple mobile phones can be fixed on a mobile phone video holder. The mobile phone holder can fix 2 or more mobile phones side by side or vertically. The relative position remains unchanged during video recording, so that the viewing angle difference between the first terminal device and the N terminal devices does not change during recording, thereby improving the effect of video fusion.

[0074] 202. When the N terminal devices agree to the joint shooting request, the first terminal device sends a calibration instruction to the N terminal devices.

[0075] 203 , the first terminal device performs image calibration according to the calibration image frame taken by the first terminal device and the calibration image frames taken by the N terminal devices to obtain calibration parameters of the N terminal devices.

[0076] In an embodiment of the present application, after N terminal devices agree to the joint shooting request, the first terminal device can send a calibration instruction to the N terminal devices to perform camera pre-shooting calibration, and obtain calibration parameters of the N terminal devices by comparison. The calibration parameters may include translation parameters, and the translation parameters may include perspective difference parameters. After the first terminal device and the N terminal devices are fixed, the relative positions of the first terminal device and any two of the N terminal devices remain unchanged, that is, the relative positions between the main shooting device and the auxiliary shooting device are fixed. Pre-adjustment can be performed first to obtain images of each terminal device and perform video synthesis. Because the cameras of multiple closely spaced terminal devices shoot at fixed positions and there are image differences (for example, perspective differences), but most of the content in the image is basically the same, the image fusion technology of 360 panoramic scanning can be used to fuse and splice images from different angles. While synchronizing the focus and white balance parameters of the first terminal device and the N terminal devices, the image difference can be compared to perform image recognition and determine the image difference DELTA (X, Y) between the images taken by the cameras of the N terminal devices and the image taken by the camera of the first terminal device. The image difference refers to the amount of translation of the images taken by the cameras of the N terminal devices relative to the image taken by the camera of the first terminal device on the x-axis and y-axis, and can also be called a translation parameter. The translation parameter can prepare for the subsequent video frame fusion processing.

[0077] 204. The first terminal device obtains recording capability information of N terminal devices interconnected with the first terminal device, and determines recording parameters of each of the N terminal devices and the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, where N is a positive integer.

[0078] 205. The first terminal device sends recording parameters of each terminal device to each terminal device.

[0079] 206. When the first terminal device and the N terminal devices receive the joint recording instruction, the first terminal device performs video recording based on the recording parameters of the first terminal device to obtain a first video frame set.

[0080] 207, the first terminal device obtains N video frame sets obtained by N terminal devices performing video recording based on the recording parameters of each terminal device; wherein the N video frame sets correspond to the N terminal devices one by one, the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, and there is at least one video frame in the N video frame sets whose acquisition time point is within the acquisition time range of the first video frame set.

[0081] The specific implementation of steps 204 to 207 can be found in Figure 1The description of steps 101 to 104 shown will not be repeated here.

[0082] 208. The first terminal device fuses the first video frame set with the N video frame sets based on the calibration parameters of the N terminal devices to obtain a fused video.

[0083] In an embodiment of the present application, when the calibration parameters include translation parameters, N video frame sets can be translated according to the translation parameters of the corresponding terminal device to obtain N video frame sets after translation, and the first video frame set and the N video frame sets after translation can be fused in the order of acquisition time points to obtain a fused video.

[0084] Optionally, the calibration parameters include translation parameters, and step 208 may include the following steps:

[0085] (21) the first terminal device performs a translation process on a second video frame in the set of video frames recorded by the second terminal device based on a translation parameter of the second terminal device to obtain a second translation-processed video frame, where the second video frame is any video frame in the set of video frames recorded by the second terminal device, and the second terminal device is any one of the N terminal devices;

[0086] (22) The first terminal device performs cropping, splicing and consistency processing on the second translation-processed video frame and the third video frame to obtain a second fused video frame, where the third video frame is a video frame in the first video frame set whose acquisition time point is closest to the acquisition time point of the second video frame;

[0087] (23) The first terminal device fuses the first video frame set and the N video frame sets that have been subjected to translation processing, cropping and splicing processing, and consistency processing in the order of acquisition time points to obtain a fused video.

[0088] In the embodiment of the present application, for any video frame in the set of video frames recorded by each of the N terminal devices, the translation processing, cropping and splicing processing and consistency processing can be performed in the manner of step (21) and step (22), thereby improving the fusion effect of the fused video.

[0089] Although the second translation-processed video frame can eliminate the perspective difference with the video frames in the first video frame set, the size of the second translation-processed video frame will become smaller, the content area will become smaller, and blank areas will appear. In order to improve the fusion processing effect, it is necessary to fill the blank part after the translation processing. The second translation-processed video frame and the third video frame can be cropped and spliced ​​so that the video frames used for fusion processing are all video frames of the same size. Specifically, the cropping and splicing processing may include the following steps: cropping the content area of ​​the second translation-processed video frame, and splicing the cropped content area with the splicing area, and the splicing area is the area in the third video frame corresponding to the blank area of ​​the second translation-processed video frame.

[0090] See also Figure 3 , Figure 3 is a comparative schematic diagram of a fusion using cutting and splicing and a fusion not using cutting and splicing provided in an embodiment of the present application. Figure 3 It can be clearly seen that the effect of the fusion image that is fused by cropping and splicing is obviously better than the effect of the fusion image that is not fused by cropping and splicing.

[0091] Consistency processing can be consistency in brightness, exposure, sensitivity, white balance, and color tone. Different terminal devices, if they are different models or even if they are the same model but belong to different batches, may have different effects when photographed by their cameras. There will be certain differences in brightness, exposure, sensitivity, white balance, and color tone. Even if the recording parameters such as white balance parameters, fill light parameters, and focus parameters are unified, the video frames shot by different terminal devices will still have certain differences. Therefore, consistency processing can be performed to make the video frames shot by each terminal device as consistent as possible in brightness, exposure, sensitivity, white balance and color tone, thereby improving the effect of the fused video obtained by fusion processing.

[0092] Optionally, in step (22), the first terminal device performs cropping, splicing and consistency processing on the second translation-processed video frame and the third video frame to obtain a second fused video frame, which may specifically include the following steps:

[0093] (221) The first terminal device performs cropping and splicing processing on the second translation-processed video frame and the third video frame to obtain a second cropped and spliced ​​video frame;

[0094] (222) The first terminal device determines an image consistency difference parameter based on the third video frame and the second cropped and spliced ​​video frame, and performs consistency processing on the second cropped and spliced ​​video frame according to the image consistency difference parameter to obtain a second fused video frame.

[0095] In an embodiment of the present application, image consistency difference parameters may include parameters such as brightness difference, exposure difference, sensitivity difference, white balance difference, and hue difference. By comparing the third video frame with the second cropped and spliced ​​video frame, the brightness difference, exposure difference, sensitivity difference, white balance difference, and hue difference between the two video frames are determined. According to the brightness difference, exposure difference, sensitivity difference, white balance difference, and hue difference between the two video frames, the second cropped and spliced ​​video frame is processed for consistency to obtain a second fused video frame, so that the image consistency difference between the second fused video frame and the third video frame is as small as possible, thereby improving the subsequent fusion processing effect.

[0096] Optionally, the calibration parameters may also include white balance calibration parameters and color tone calibration parameters.

[0097] The purpose of calibration is to make the camera effects of each terminal device as similar as possible. Take mobile phones as an example. Different models of mobile phones, such as A series mobile phones and B series mobile phones, the B series has been optimized based on the A series. Although the camera of the A series mobile phone is worse than that of the B series, when the two mobile phones are fused, the white balance parameters of the A series mobile phone can be selected as the white balance calibration parameters of the video frames shot by the B series mobile phone according to the optimized parts of the B series relative to the A series, and the color tone parameters of the A series mobile phone can be selected as the color tone calibration parameters of the video frames shot by the B series mobile phone. In terms of white balance and color tone, the effects of the video frames shot by the two mobile phones are kept as consistent as possible, so that the fusion effect of the two is better. According to the differences between the two series of mobile phones, an optimal calibration parameter can be found to improve the fusion effect.

[0098] In the embodiment of the present application, the first video frame set and the N video frame sets are fused based on the calibration parameters of the N terminal devices to obtain a fused video. The video frame sets recorded by at least two terminal devices can be fused. Since the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different, there is at least one video frame in the N video frame sets whose acquisition time point is within the acquisition time range of the first video frame set, so that more video frames within the acquisition time range of the first video frame set participate in the fusion process, thereby improving the frame rate of the fused video and further improving the recording effect of the fused video. Since the N video frame sets are calibrated with the calibration parameters, the viewing angle difference between the video frames in the N video frame sets after the calibration process and the video frames in the first video frame set can be eliminated, thereby improving the recording effect of the fused video.

[0099] The video fusion method of the embodiment of the present application can be applied to slow-motion scenes. Figure 4 , Figure 4FIG. 1 is a flow chart of a method for performing slow-motion video fusion using two mobile phones as an example provided in an embodiment of the present application. Figure 4 As shown, mobile phone 1 is the main shooting device, and mobile phone 2 is the auxiliary shooting device. The camera module of mobile phone 1 may include a video capture submodule, a slow shot setting submodule, a video calibration submodule, a slow shot interconnection submodule, a slow shot recording submodule, and a slow shot fusion submodule. The camera module of mobile phone 2 may include a video capture submodule, a slow shot setting submodule, a video calibration submodule, and a slow shot interconnection submodule. Figure 4 The method may include the following steps:

[0100] 1. The user uses a bracket to combine multiple mobile phones for slow shooting. Open the main mobile phone video software and select joint slow shooting. Set mobile phone 1 as the main shooter and other mobile phones as auxiliary shooters. The bracket can fix 2 or more mobile phones side by side or vertically. Multiple mobile phones can be installed through the video stand.

[0101] 2. Scan other mobile phones through high-speed WIFI / USB and other high-speed interconnection. The auxiliary shooting mobile phone sends a slow-motion joint shooting request to other mobile phones, and the connection can be achieved after the other mobile phone agrees.

[0102] If it is a mobile phone or smart home device under the same user account, you can directly return to agree.

[0103] The embodiment of the present application can use the mobile phone interconnection capability to scan, for example, OPPO mobile phones can find screen projection with video connection, can find car keys, and support joint slow shooting is also a capability of mobile phones. Mobile phones with this capability support being discovered by the near network and establish a connection to open up a collaborative data channel. The embodiment of the present application can be multiple mobile phones of the same model for joint shooting, because the camera capabilities and effects of multiple mobile phones of the same model are basically the same, which is convenient for the subsequent slow motion video fusion.

[0104] 3. After the main and auxiliary machines are connected, the main shooting mobile phone setting module calls the calibration module to perform camera pre-shooting calibration, and obtains the viewing angle difference parameters of the side-by-side shooting mobile phones through comparison. After the first step, the relative positions of the main and auxiliary machines are fixed, so you can pre-adjust to obtain the images of each mobile phone and perform video synthesis. Because the cameras of multiple closely spaced mobile phones shoot in a fixed position, there will be image differences (for example, viewing angle differences), but most of the content in the image is basically the same. The image fusion technology of 360 panoramic scanning can be used to fuse and splice images from different angles. The image difference can prepare for the subsequent fusion of video frames from multiple mobile phones.

[0105] 4. The auxiliary shooting mobile phone is unified with accurate time difference and resolution (recording parameters such as white balance parameters can also be unified), and the frame rate difference time point of each mobile phone shooting is calculated and allocated according to the camera capability of each mobile phone. The frame rate difference time point is the starting recording time point.

[0106] For example, if Mobile Phone 1 and Mobile Phone 2 are of the same model and both support 1080P 120FPS frame rate, then you only need to synchronize the time of both devices.

[0107] 5. The user issues a slow-motion command, and all connected mobile phones start slow-motion shooting, and sample at the frame rate difference time points calculated and allocated by mobile phone 1 (the main shooter). Mobile phone 1 samples at 0 milliseconds, 8.333 milliseconds, 16.666 milliseconds, etc., and mobile phone 2 samples at 4.16 milliseconds, 12.5 milliseconds, 20.832.5 milliseconds, etc.

[0108] That is to say, the sampling time points of each mobile phone are staggered in the timestamp to achieve the effect of increasing the frame rate. In theory, the more mobile phones there are, the stronger the slow-motion capability will be.

[0109] 6. Combine the high frame rate slow motion videos of each mobile phone. The angles of different mobile phones will be slightly different. The main shooting mobile phone can obtain the video frames shot by each mobile phone through the near network high speed and combine them to generate a slow motion video.

[0110] In the embodiment of the present application, multiple mobile phones can be used to jointly shoot high-speed photos of a certain location when the content of the photos is similar, and the interconnection technology can be used to coordinate, recognize video frames, and perform fusion processing to generate wonderful ultra-high frame slow-motion videos. It can support the joint slow-motion effect of multiple mobile phones, and use multiple mobile phones to shoot at the same time to enhance the slow-motion capability of a single mobile phone, thereby improving the ability to capture faster and exciting moment videos (for example, the moment when a high-speed steel ball penetrates a Sprite can or a beer bottle).

[0111] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that in order to realize the above functions, the terminal device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment provided in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0112] The embodiment of the present application can divide the terminal device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0113] See also Figure 5 , Figure 5 is a structural schematic diagram of a video fusion device provided in an embodiment of the present application. The video fusion device 500 is applied to a terminal device. The video fusion device 500 may include an obtaining unit 501, a determining unit 502, a sending unit 503, a recording unit 504 and a fusion unit 505, wherein:

[0114] The obtaining unit 501 is used to obtain recording capability information of N terminal devices interconnected with the first terminal device;

[0115] The determining unit 502 is configured to determine the recording parameters of each of the N terminal devices and the recording parameters of the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, where N is a positive integer;

[0116] The sending unit 503 is used to send the recording parameters of each terminal device to each terminal device;

[0117] The recording unit 504 is configured to perform video recording based on the recording parameters of the first terminal device to obtain a first video frame set when the first terminal device and the N terminal devices receive a joint recording instruction;

[0118] The obtaining unit 501 is further configured to obtain N video frame sets obtained by the N terminal devices performing video recording based on the recording parameters of the respective terminal devices; wherein the N video frame sets correspond to the N terminal devices one by one, the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, and the acquisition time point of at least one video frame in the N video frame sets is within the acquisition time range of the first video frame set;

[0119] The fusion unit 505 is used to fuse the first video frame set and the N video frame sets to obtain a fused video.

[0120] Optionally, the recording capability information includes a maximum resolution and a maximum frame rate, and the recording parameters include a unified resolution, a configured frame rate and a starting recording time point; the determination unit 502 determines the recording parameters of each of the N terminal devices and the recording parameters of the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, including: selecting the lowest resolution from the maximum resolution supported by the N terminal devices and the maximum resolution supported by the first terminal device as the unified resolution for the N terminal devices and the first terminal device; determining the configured frame rate and the starting recording time point of the first terminal device and the configured frame rate and the starting recording time point of each of the N terminal devices according to the maximum frame rate supported by the N terminal devices and the maximum frame rate supported by the first terminal device; the configured frame rate of the first terminal device is less than or equal to the maximum frame rate supported by the first terminal device, and the configured frame rate of each of the terminal devices is less than or equal to the maximum frame rate of each of the terminal devices.

[0121] Optionally, the video fusion device 500 may further include a calibration unit 506;

[0122] The sending unit 503 is further configured to send a joint shooting request to the N terminal devices;

[0123] The sending unit 503 is further configured to send a calibration instruction to the N terminal devices if the N terminal devices agree to the joint shooting request;

[0124] The calibration unit 506 is configured to perform image calibration according to the calibration image frame taken by the first terminal device and the calibration image frames taken by the N terminal devices, so as to obtain calibration parameters of the N terminal devices;

[0125] The fusion unit 505 fuses the first video frame set and the N video frame sets to obtain a fused video, including: fusing the first video frame set and the N video frame sets based on calibration parameters of the N terminal devices to obtain a fused video.

[0126] Optionally, the video fusion device 500 may further include an interconnect scanning unit;

[0127] The interconnection scanning unit is used to discover the N terminal devices through an interconnection scanning method;

[0128] The sending unit 503 is further configured to send a joint shooting request to the N terminal devices after the interconnection scanning unit discovers the N terminal devices through the interconnection scanning method;

[0129] Optionally, the calibration parameters include translation parameters, and the fusion unit 505 fuses the first video frame set and the N video frame sets based on the calibration parameters of the N terminal devices to obtain a fused video, including: performing translation processing on the second video frame in the video frame set recorded by the second terminal device based on the translation parameter of the second terminal device to obtain a second translation-processed video frame, the second video frame is any video frame in the video frame set recorded by the second terminal device, and the second terminal device is any one of the N terminal devices; cropping, splicing and consistency processing on the second translation-processed video frame and the third video frame to obtain a second fused video frame, the third video frame is a video frame in the first video frame set whose acquisition time point is closest to the acquisition time point of the second video frame; fusing the first video frame set and the N video frame sets that have been subjected to translation processing, cropping, splicing and consistency processing in the fused video frames in the order of acquisition time points to obtain a fused video.

[0130] Optionally, the fusion unit 505 performs cropping, splicing and consistency processing on the second translation-processed video frame and the third video frame to obtain a second fused video frame, including: cropping and splicing the second translation-processed video frame and the third video frame to obtain a second cropped and spliced ​​video frame; determining an image consistency difference parameter based on the third video frame and the second cropped and spliced ​​video frame, and performing consistency processing on the second cropped and spliced ​​video frame according to the image consistency difference parameter to obtain a second fused video frame.

[0131] Optionally, the relative positions of the first terminal device and any two of the N terminal devices remain unchanged.

[0132] Optionally, the recording parameters also include white balance parameters, fill light parameters and focus parameters.

[0133] The determining unit 502, the fusion unit 505, and the calibration unit 506 in the embodiment of the present application may be a processor in the terminal device. The obtaining unit 501, the sending unit 503, and the interconnected scanning unit in the embodiment of the present application may be a communication module in the terminal device. The recording unit 504 may be a camera module in the terminal device.

[0134] Figure 5 The specific implementation of the video fusion device 500 shown can be found in Figure 1 to Figure 2 The method embodiment shown will not be described in detail here.

[0135] In an embodiment of the present application, a first video frame set recorded by a first terminal device and N video frame sets recorded by N terminal devices are fused to obtain a fused video. Video frame sets recorded by at least two terminal devices can be fused. Since the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, there is at least one video frame in the N video frame sets whose acquisition time point is within the acquisition time range of the first video frame set, so that more video frames within the acquisition time range of the first video frame set participate in the fusion process, thereby improving the frame rate of the fused video, and further improving the recording effect of the fused video.

[0136] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application, such as Figure 6 As shown, the terminal device 600 includes a processor 601 and a memory 602, and the processor 601 and the memory 602 can be connected to each other via a communication bus 603. The communication bus 603 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 603 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The memory 602 is used to store computer programs. The computer programs include program instructions. The processor 601 is configured to call the program instructions. The program includes instructions for executing Figure 1-2 Some or all of the steps in the method shown.

[0137] The memory 602 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0138] In addition, the terminal device 600 may also include a communication module 604 and a camera module 605. The communication module 604 may include common components such as a communication interface, and the camera module 605 may include at least one camera, which will not be described in detail here.

[0139] In an embodiment of the present application, a first video frame set recorded by a first terminal device and N video frame sets recorded by N terminal devices are fused to obtain a fused video. Video frame sets recorded by at least two terminal devices can be fused. Since the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, there is at least one video frame in the N video frame sets whose acquisition time point is within the acquisition time range of the first video frame set, so that more video frames within the acquisition time range of the first video frame set participate in the fusion process, thereby improving the frame rate of the fused video, and further improving the recording effect of the fused video.

[0140] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the video fusion methods recorded in the above method embodiments.

[0141] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0142] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, the functional units in the various embodiments of the application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0146] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0147] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0148] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video fusion method, characterized in that: include: The first terminal device obtains recording capability information of N terminal devices interconnected with the first terminal device, and determines recording parameters of each terminal device in the N terminal devices and recording parameters of the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, where N is a positive integer; The recording capability information includes a maximum resolution and a maximum frame rate, and the recording parameters include a uniform resolution, a configured frame rate, and a start recording time point; the configured frame rate of the first terminal device is less than or equal to the maximum frame rate supported by the first terminal device, and the configured frame rate of each terminal device is less than or equal to the maximum frame rate of each terminal device; The first terminal device sends recording parameters of each terminal device to each terminal device; When the first terminal device and the N terminal devices receive the joint recording instruction, the first terminal device performs video recording based on the recording parameters of the first terminal device to obtain a first video frame set; The first terminal device obtains N video frame sets obtained by the N terminal devices performing video recording based on the recording parameters of the respective terminal devices; wherein the N video frame sets correspond to the N terminal devices one by one, the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, and the acquisition time point of at least one video frame in the N video frame sets is within the acquisition time range of the first video frame set; The first terminal device sends a joint shooting request to the N terminal devices; When the N terminal devices agree to the joint shooting request, the first terminal device sends a calibration instruction to the N terminal devices; The first terminal device performs image calibration according to the calibration image frame taken by the first terminal device and the calibration image frames taken by the N terminal devices to obtain calibration parameters of the N terminal devices; The first terminal device fuses the first video frame set and the N video frame sets based on the calibration parameters of the N terminal devices to obtain a fused video.

2. The method according to claim 1, characterized in that The determining, according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, the recording parameters of each terminal device in the N terminal devices and the recording parameter of the first terminal device includes: Selecting the lowest resolution from the maximum resolutions supported by the N terminal devices and the maximum resolution supported by the first terminal device as the unified resolution of the N terminal devices and the first terminal device; The configured frame rate and the starting recording time point of the first terminal device and the configured frame rate and the starting recording time point of each terminal device in the N terminal devices are determined according to the maximum frame rate supported by the N terminal devices and the maximum frame rate supported by the first terminal device.

3. The method according to claim 1, characterized in that The calibration parameters include translation parameters, and the first terminal device fuses the first video frame set with the N video frame sets based on the calibration parameters of the N terminal devices to obtain a fused video, including: The first terminal device performs translation processing on a second video frame in the set of video frames recorded by the second terminal device based on the translation parameter of the second terminal device to obtain a second translation-processed video frame, wherein the second video frame is any video frame in the set of video frames recorded by the second terminal device, and the second terminal device is any one of the N terminal devices; The second translation processed video frame and the third video frame are subjected to cropping, splicing and consistency processing to obtain a second fused video frame, wherein the third video frame is a video frame in the first video frame set whose acquisition time point is closest to the acquisition time point of the second video frame; The fused video frames in the first video frame set and the N video frame sets that have been subjected to translation processing, cropping and splicing processing, and consistency processing are fused in the order of acquisition time points to obtain a fused video.

4. The method according to claim 3, characterized in that The step of performing cropping, splicing and consistency processing on the second translation processed video frame and the third video frame to obtain a second fused video frame includes: The second translation processed video frame and the third video frame are cropped and spliced ​​to obtain a second cropped and spliced ​​video frame; An image consistency difference parameter is determined based on the third video frame and the second cropped and spliced ​​video frame, and consistency processing is performed on the second cropped and spliced ​​video frame according to the image consistency difference parameter to obtain a second fused video frame.

5. The method according to any one of claims 1 to 4, characterized in that: During the video recording process, the relative positions of the first terminal device and any two of the N terminal devices remain unchanged.

6. The method according to any one of claims 1 to 4, characterized in that: The recording parameters also include white balance parameters, fill light parameters and focus parameters.

7. A video fusion device, characterized in that: include: An obtaining unit, used to obtain recording capability information of N terminal devices interconnected with the first terminal device; a determining unit, configured to determine recording parameters of each of the N terminal devices and the first terminal device according to the recording capability information of the N terminal devices and the recording capability information of the first terminal device, where N is a positive integer; The recording capability information includes a maximum resolution and a maximum frame rate, and the recording parameters include a uniform resolution, a configured frame rate, and a start recording time point; the configured frame rate of the first terminal device is less than or equal to the maximum frame rate supported by the first terminal device, and the configured frame rate of each terminal device is less than or equal to the maximum frame rate of each terminal device; A sending unit, used for sending the recording parameters of each terminal device to each terminal device; a recording unit, configured to perform video recording based on a recording parameter of the first terminal device to obtain a first video frame set when the first terminal device and the N terminal devices receive a joint recording instruction; The obtaining unit is further used to obtain N video frame sets obtained by the N terminal devices performing video recording based on the recording parameters of the respective terminal devices; wherein the N video frame sets correspond to the N terminal devices one by one, the acquisition time points of any two video frames in the first video frame set and the N video frame sets are different from each other, and the acquisition time point of at least one video frame in the N video frame sets is within the acquisition time range of the first video frame set; The sending unit is further used to send a joint shooting request to the N terminal devices; The sending unit is further configured to send a calibration instruction to the N terminal devices if the N terminal devices agree to the joint shooting request; a calibration unit, configured to perform image calibration according to the calibration image frame taken by the first terminal device and the calibration image frames taken by the N terminal devices, and obtain calibration parameters of the N terminal devices; A fusion unit is used to fuse the first video frame set and the N video frame sets based on the calibration parameters of the N terminal devices to obtain a fused video.

8. A terminal device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 6.

Images