Method, device and system for replaying hundred million-level pixel fusion video
By determining the time reference for the recording files of each unit's recording period in the million-pixel computing imaging system and performing frame group overlap recording, the problem of frame drop and continuity caused by the out-of-synchronization of the frame number in the fused video is solved, and the problem of poor frame loss and continuity is realized The synchronization accuracy and continuity of videos are improved.
Patent Information
- Application Number
- CN202311653804.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-05
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-12-05
AI Technical Summary
In the existing 100 million pixel computing imaging system, independent exposure and encoding of local cameras leads to the same video frame number of video frames when viewing the fused video, resulting in the problems of frame dropping and poor continuity of fused videos.
By obtaining the target video file in the recording period of each unit recorded in the stored video file, determining the splicing start and end time reference, ensuring that the video frame timestamps of each group of video files are synchronized, and using the timestamp difference value to write to the video file to achieve frames Combining overlap recording and splicing fusion ensures the continuity of video viewing.
The picture synchronization accuracy and viewing continuity of the fused video are improved, and the problem of frame drop and poor continuity of the fused video is solved.
Smart Images

Figure CN119094681A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video processing technology, and in particular to a method and device for reviewing billion-pixel fusion videos, a billion-pixel computational imaging system, a computer device, a storage medium, and a computer program product. Background Art
[0002] Video surveillance has been widely used in the security field due to its advantages of high reliability, strong timeliness, and easy viewing. When the monitoring range is large, in order to achieve ultra-high-definition video monitoring of large scenes, a billion-pixel computational imaging system based on array cameras has emerged. The array camera contains multiple local cameras with long focal lengths and narrow fields of view to shoot multiple local high-definition videos of the monitoring scene, and then the local videos are spliced and fused to obtain ultra-high-definition fused videos with large or wide fields of view up to billion pixels. It is suitable for video shooting and security monitoring of large scenes such as airports, highways, parks, sports stadiums, borders, and sea surfaces.
[0003] In related technologies, the billion-pixel computational imaging system can also provide a fused video playback function. In order to achieve fused video playback, each local video is usually recorded in segments to obtain multiple groups of video files. When the fused video needs to be played back, the corresponding frames in each group of video files are spliced and fused.
[0004] However, since each local camera performs exposure and encoding independently, it is difficult to synchronize the shooting time of each local video. In the video files of each local camera recorded in the same period (a group of video files), the video frames with the same frame numbers (such as the first frame, the second frame, etc.) are not shot synchronously. Therefore, the start time and end time of each local video in a group of video files are inconsistent, and there is a certain time difference. This time difference will result in some video frames in each group of video files that cannot be used for synchronous stitching (stitching synchronously shot videos), which will lead to frame loss in the fused video and poor continuity in video playback. Summary of the invention
[0005] Based on this, it is necessary to provide a method, device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product for reviewing billion-pixel fused videos that can improve the continuity of fused video review in response to the above-mentioned technical problems.
[0006] In a first aspect, the present application provides a method for reviewing a billion-pixel fusion video. The method comprises:
[0007] In response to a review instruction for the fused video, determine a video review time, and obtain a plurality of groups of target video files corresponding to the video review time from each group of video files corresponding to each unit recording time period stored;
[0008] Among them, a group of video files corresponding to a unit recording period is obtained by recording, for each local video stream corresponding to the fused video, each video frame after the first key frame after the start time of the unit recording period and before the last reference frame of the video frame group corresponding to the end time of the unit recording period; in two adjacent unit recording periods, the start time of the next unit recording period is earlier than the end time of the previous unit recording period, and the time difference between the start time of the next unit recording period and the end time of the previous unit recording period is not less than the frame group length of the local video stream;
[0009] Splicing and fusion are performed based on each group of the target video files to obtain the fused video for playback.
[0010] In one embodiment, the time difference is n times the frame group length of the local video stream, where n is a positive integer not less than 1; the step of performing splicing and fusion based on each group of the target video files to obtain the fused video for playback includes:
[0011] For a first group of target video files in each group of the target video files, determine a first timestamp of a first video frame and a second timestamp of a last video frame of each of the target video files in the group, use the largest first timestamp among the first timestamps as a start time reference for splicing the group of target video files, and use the smallest second timestamp among the second timestamps as an end time reference for splicing the group of target video files;
[0012] For the target video files of other groups except the first group of target video files, determine the third timestamp of the first video frame and the fourth timestamp of the last video frame of each target video file, use the timestamp of the n+1th key frame in the target video file with the largest third timestamp in the group as the start time reference for splicing the target video files of the group, and use the smallest fourth timestamp among the fourth timestamps as the end time reference for splicing the target video files of the group;
[0013] Based on the splicing start time reference and the splicing end time reference of each group of the target video files, each group of the target video files is spliced and fused to obtain the fused video for playback.
[0014] In one embodiment, the time difference is the frame group duration of the local video stream; before responding to the review instruction for the fused video, the method further includes:
[0015] In response to a recording instruction of the fused video, the start time and the end time of each unit recording period are determined according to the recording start time, the unit recording duration and the frame group duration of the local video stream, wherein, in two adjacent unit recording periods, the start time of the next unit recording period is earlier than the end time of the previous unit recording period, and the time difference between the start time of the next unit recording period and the end time of the previous unit recording period is the frame group duration;
[0016] When the current time reaches the start time of the current unit recording period, for each of the local video streams, the video frames after the first key frame after the start time of the current unit recording period and before the last reference frame of the video frame group corresponding to the end time of the current unit recording period are recorded as a recording file corresponding to the current unit recording period, and the recording files of each local video stream in the same unit recording period are stored as a group of recording files.
[0017] In one embodiment, the method further comprises:
[0018] The camera timestamp of the first key frame obtained after the start time of the video recording is used as the reference timestamp;
[0019] The difference between the camera timestamp of each video frame in each video file and the reference timestamp is used as the video timestamp of each video frame, and the video timestamp is written into the video file.
[0020] In one embodiment, the method is applied to a server, the server includes a collaboration module and two recording modules, the time difference is the frame group duration of the local video stream; before responding to the review instruction for the fused video, the method further includes:
[0021] The collaborative module responds to the recording instruction of the fused video by sending a first start recording instruction to the first recording module. The first recording module responds to the first start recording instruction by acquiring each channel of local video streams corresponding to the fused video, and for each channel of the local video stream, starting from the first key frame of the local video stream acquired after the current moment, records the first key frame and subsequent video frames as a recording file of the current unit recording period.
[0022] When the current time reaches the start time of the next unit recording period, the collaboration module sends a second start recording instruction to the second recording module. The second recording module responds to the second start recording instruction to obtain each channel of local video streams corresponding to the fused video, and for each channel of the local video stream, starting from the first key frame of the local video stream obtained after the current time, the first key frame and subsequent video frames are recorded as the video file of the next unit recording period.
[0023] When the current time reaches the end time of the current unit recording period, the collaboration module sends a first stop recording instruction to the first recording module, and the first recording module responds to the first stop recording instruction, writes all video frames of the video frame group where the video frame acquired at the current moment is located into the video file of the current unit recording period, and then stops recording;
[0024] When the current time reaches the end time of the next unit recording period, the collaboration module sends a second stop recording instruction to the second recording module, and the second recording module responds to the second stop recording instruction, writes all video frames of the video frame group where the video frame acquired at the current moment is located into the video file of the next unit recording period, and then stops recording;
[0025] The start time of the next unit recording period is earlier than the end time of the current unit recording period, and the time difference between the start time of the next unit recording period and the end time of the current unit recording period is the frame group duration of the local video stream.
[0026] In a second aspect, the present application also provides a device for reviewing billion-pixel fusion videos. The device includes:
[0027] An acquisition module is used to respond to a review instruction for a fused video, determine a video review time, and obtain several groups of target video files corresponding to the video review time from each group of video files corresponding to each stored unit recording period; wherein a group of video files corresponding to a unit recording period is obtained by recording, for each local video stream corresponding to the fused video, each video frame after the first key frame after the start time of the unit recording period and before the last reference frame of the video frame group corresponding to the end time of the unit recording period; of two adjacent unit recording periods, the start time of the next unit recording period is earlier than the end time of the previous unit recording period, and the time difference between the start time of the next unit recording period and the end time of the previous unit recording period is not less than the frame group length of the local video stream;
[0028] The fusion module is used to perform splicing and fusion based on each group of the target video files to obtain the fused video for playback.
[0029] In a third aspect, the present application also provides a billion-pixel computational imaging system. The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes multiple local cameras, wherein:
[0030] The array camera is used to shoot local videos through each of the local cameras and send each local video stream to the server;
[0031] The server is used to respond to the recording instruction of the fused video and store the acquired local video streams as multiple groups of recording files of multiple unit recording time periods; wherein a group of recording files corresponding to a unit recording time period is obtained by recording, for each local video stream corresponding to the fused video, each video frame after the first key frame after the start time of the unit recording time period and before the last reference frame of the video frame group corresponding to the end time of the unit recording time period; in two adjacent unit recording time periods, the start time of the next unit recording time period is earlier than the end time of the previous unit recording time period, and the time difference between the start time of the next unit recording time period and the end time of the previous unit recording time period is not less than the frame group length of the local video stream;
[0032] The server is also used to respond to a review instruction for the fused video, determine the video review time, and obtain several groups of target video files corresponding to the video review time from each group of video files corresponding to each unit recording time period stored; and perform splicing and fusion based on each group of the target video files to obtain the fused video for review.
[0033] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.
[0034] In a fifth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0035] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0036] The above-mentioned review method, device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product of billion-pixel fusion video start the video recording of the next recording period before the end time of the current recording period and at a time interval of at least one frame group length, so that there is at least one overlapping video frame group data in the video files of adjacent recording periods. Since the shooting time of each local camera is not absolutely synchronized, when reviewing, the fusion video obtained based on the video file of the previous period will end with the last video frame with the smallest timestamp, and the last video frame used for splicing in other video files in the group may be an intermediate reference frame (not necessarily the last video frame) in the last video frame group in the video file. Since there is at least one overlapping video frame group data in the next period and the previous period, a video frame continuous with the end time of the previous period can be found in the video file of the next period, and the video file also includes the key frame of the video frame group where the video frame is located, so the continuous video frames can be decoded to obtain a complete image, so that a fusion video with accurate picture synchronization and time continuity can be spliced. Therefore, based on this solution, the fused video playback can ensure the accuracy of the fused video picture synchronization while improving the continuity of the fused video playback, and can solve the problems of frame loss and poor continuity of the fused video in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1a A schematic diagram of a billion-pixel computational imaging system in an example;
[0038] Figure 1b to Figure 1c The following is a schematic diagram of recording a local video stream in an example; Figure 1b The middle part shows the situation where all local videos are shot synchronously. Figure 1c The situation where each local video is shot asynchronously;
[0039] Figure 2 A schematic diagram of a flow chart of a method for reviewing a billion-pixel fusion video in an embodiment;
[0040] Figure 3 This is a schematic diagram of recording a local video stream based on this solution in an example;
[0041] Figure 4 It is a structural block diagram of a device for reviewing billion-pixel fusion video in an embodiment;
[0042] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] First, before specifically introducing the technical solution of the embodiment of the present application, the technical background or technical evolution context based on the embodiment of the present application is introduced. In order to realize ultra-high-definition video surveillance of large scenes, a billion-pixel computing imaging system has emerged, such as Figure 1a In the illustrated billion-pixel computational imaging system, the array camera 102 and the display terminal 106 communicate with the server 104 through the network. The array camera 102 includes multiple local cameras with different shooting angles, which can respectively shoot local high-definition videos of parts of the monitoring scene, and then splice and fuse the local videos through the server to obtain a billion-pixel fused video with a large or wide field of view and up to the billion-pixel level, which is used for display terminal playback or display, and is suitable for video shooting and security monitoring of large scenes such as airports, highways, parks, sports stadiums, borders, and sea surfaces.
[0045] In order to meet the needs of users to playback and view the fused video of historical time periods, the billion-pixel computing imaging system can also provide a fused video playback function, which involves the problem of recording and storing ultra-high-definition fused videos. In practical applications, in order to save computing resources, the local videos shot by each local camera of the array camera are usually recorded separately to obtain multiple video files. When the user needs to review the fused video of a certain period of time, the local video data of the period can be spliced and fused, thereby avoiding the splicing and fusion of local videos that do not need to be reviewed, saving computing resources. In addition, if a long local video stream (high-definition video, large data volume) is recorded as a file, the video file will be large, which is not conducive to storage and use. Therefore, in order to improve the convenience and flexibility of storage and use, each local video is usually recorded in segments to obtain multiple groups of time-continuous video files. For example, the duration of each video file or the duration of each segment can be set to L (such as 120,000 milliseconds, i.e. 2 minutes).
[0046] The current encoding standards used for local video streams (such as H264, etc.) are usually encoded using a video frame group (GOP, Group of picture). The first frame of each video frame group is a key frame (I frame) containing complete image information, and other frames after the key frame are reference frames (P frames or B frames) containing partial image information (usually difference information from the video frame before the frame). The reference frame must be based on the data of other video frames after the key frame of the group (including the key frame) and before the frame (possibly including the subsequent reference frame) in order to be decoded to obtain a complete image.
[0047] like Figure 1b As shown, this example uses the local video recording of three local cameras (A11, A12, and A13) as an example for explanation. The continuous line segment in the middle represents the local video stream shot by the local camera. Each small line segment represents a video frame group (including the front end point but not the back end point). The front end point of the line segment is the key frame (first frame) of the video frame group, and the back end point is the key frame of the next video frame group. The time to start recording is S0. When the server receives the recording instruction, it starts to pull the local video streams of each local camera, where I 1_A11 The first key frame of the local camera A11 obtained after the video starts (since after S0, I 1_A11 Previously, it was only a reference frame of the previous video frame group and could not be fully decoded, so recording started from the key frame). 1_A11 Start recording and generate video file F 1_A11 When the recording time reaches the set L, the current complete video frame group is written into the video file F 1_A11 (This file contains keyframes I 1_A11 , corresponding to the solid dots in the figure, excluding keyframe I m1_A11 , corresponding to the hollow dots in the figure), the next video frame group (I m1_A11 The frame group) is recorded as a new video file F 2_A11 (Including keyframe I m1_A11 ), so as to achieve segmented recording. Similarly, the video files of other local cameras can be obtained (F 1_A12 、F 1_A13 The video files of each local camera recorded in the same segment are taken as a group. When the fused video needs to be reviewed, the video files of each group are obtained for splicing and fusion.
[0048] If the exposure time of each local camera is exactly the same, then each frame with the same sequence number (such as the first frame I) in a group of video files in the same segment will be recorded by each local camera. 1_A11 ,I 1_A12 ,I 1_A13 ) are shot synchronously (with the same shooting time), so starting from the first frame, each frame is stitched and fused in sequence (each first frame is stitched, each second frame is stitched, etc.), and an accurate fused video can be obtained. However, each local camera of the current array camera is usually exposed and encoded independently. In fact, the shooting time of each local video cannot be absolutely consistent, and there is a certain time difference. Figure 1cAs shown in the figure, after the server receives the recording instruction, the shooting time of the first key frame of each local video stream pulled is different. Therefore, each frame with the same sequence number in each video file is not shot synchronously. If the frames are spliced and fused sequentially from the first frame, a fused video image with severe picture distortion and poor imaging quality will be obtained. In the related art, the timestamp reference of each frame in each video file can be unified, and the splicing and fusion of synchronously shot video images can be realized based on the timestamp after the reference is unified. For example, the first key frame I shot by the local camera A11 after starting the recording is 1_A11 The time is T0, and the first key frame (I 1_A12 and I 1_A13 ) are all earlier than T0, and the video frame shot at T0 is a reference frame in the video frame group where the first key frame is located. For the video file group F shot during this period (S0~S0+L), 1_A11 、F 1_A12 、F 1_A13 , you can use the time after T0 (including T0) and T 12 Before time (excluding T 12 ) to obtain a fused video with synchronized images and accurate splicing. 2_A11 、F 2_A12 、F 2_A13 , you can use T 12 After time (including T 12 ) are spliced and fused accordingly.
[0049] The above method can achieve synchronization and accuracy of the playback screen of the fused video. However, this method has the problem of frame loss in the fused video, and the continuity of the video playback is poor. Figure 1c In the example, based on a set of video files (F 1_A11 、F 1_A12 、F 1_A13 ) and a set of video files for the next period (F 2_A11 、F 2_A12 、F 2_A13 ), all cannot be spliced and fused to get time T 12 To T 11 The fused video between the two parts is the frame with the earliest shooting time and the smallest timestamp among the last frames of each local video stream in the previous period (I m1_A12 The frame with the latest shooting time and the largest timestamp (the frame before the last one in this example) m1_A11 If there is a time difference between the previous frame and the previous frame, the fused video frame within the time difference will be lost. If something important happens during this time gap, it will affect the review and evidence of the video.
[0050] Based on this background, the applicant, through long-term research and development and experimental verification, proposed the review method of the billion-pixel fusion video of this application, which can improve the continuity of the fusion video review while ensuring the accuracy of the fusion video picture synchronization, and solve the problems of frame loss and poor continuity of the fusion video in the related technology. In addition, it should be noted that the applicant has made a lot of creative efforts in the discovery of the technical problem of this application and the technical solutions introduced in the following embodiments.
[0051] In one embodiment, Figure 2 As shown in the figure, a method for replaying billion-pixel fusion video is provided, which can be applied to Figure 1a The server in the billion-pixel computational imaging system shown in FIG. The server may be implemented as an independent server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:
[0052] Step 201, in response to a playback instruction for a fused video, determine a video playback time, and obtain a plurality of groups of target video files corresponding to the video playback time from each group of video files corresponding to each unit recording time period stored.
[0053] In implementation, the user can select a video review time and trigger a review instruction for the fused video at that time, so that the server can respond to the review instruction and obtain the video review time. The video review time can be a time period, such as selecting to review the video of a certain day, or reviewing the video of a certain period of time in a certain day, or it can be a video review start time. It can be understood that if the video review time is a time period, the server can obtain each group of target video files corresponding to the time period for splicing and fusion to obtain a fused video; if the video review time is the start time of the review, the server can determine the unit recording time period where the start time is located, and obtain several groups of target video files in and after the unit recording time period, so as to splice and fuse to obtain a continuous fused video until the user triggers a stop review instruction, or the recorded video ends.
[0054] The video files are pre-recorded and stored. The array camera can send the captured local video streams to the server, so that the server can obtain the local video streams. If the user currently needs to view the real-time fused video, the server can splice and fuse the local video streams in real time, and send the fused video to the display terminal for playback. The server can also record the acquired local video streams in segments based on the user's instructions. The server can store the video files recorded by each local video stream in the same unit time period (such as a duration of L) as a group of video files, such as storing them in the same folder, and each unit recording time period corresponds to a folder. The start time or end time of the recording time period can be used as the name of the folder, which is convenient for indexing the video files of each unit recording time period that is continuous in time based on the folder name.
[0055] Among them, a group of video files corresponding to a recording period is obtained by recording each video frame after the first key frame after the start time of the recording period (including the key frame) and before the last reference frame of the video frame group corresponding to the end time of the recording period (including the last reference frame) for each local video stream corresponding to the fused video. In order to ensure that each video frame in the video file can be decoded to obtain a complete image, each video file is usually recorded from the key frame and ends with the last frame of the key frame group. In two adjacent unit recording periods, the start time of the next unit recording period is earlier than the end time of the previous unit recording period, and the time difference between the start time of the next unit recording period and the end time of the previous unit recording period is not less than the frame group length of the local video stream (which can be recorded as G). The frame group length G is the length of a video frame group from the first frame (key frame) to the end of the last frame (the length of a GOP group), which can be preset or calculated by the server based on the timestamps of two adjacent key frames in the local video stream (timestamp difference). Usually, the frame group length of the local video stream of each local camera is the same.
[0056] For example, Figure 3 As shown in the figure, S0 is the start time of recording, which is also the start time of the first unit recording period. The end time of the first unit recording period is S0+L. The start time of the second unit recording period is S0+LG and the end time is S0+2L. A set of recording files (F 1_A11 、F 1_A12 、F 1_A13 ), the first video frame is I 1_A11 ,I 1_A12 ,I 1_A13 , the last video frame is I m1_A11 ,I m1_A12 ,I m1_A13The previous frame of the second unit recording period (the hollow circle indicates that the endpoint is not included). 2_A11 、F 2_A12 、F 2_A13 ), the first video frame is I m1-1_A11 ,I m1-1_A12 ,I m1-1_A13 In this example, the start time of the second unit recording period differs from the end time of the first unit recording period by a frame group length G. Therefore, the video file of the second unit recording period and the video file of the first unit recording period have an overlapping video frame group. In this example, the overlapping frame group is key frame I. m1-1 The video frame group.
[0057] In other examples, the time difference between the start time of the next unit recording period and the end time of the previous unit recording period can be greater than the length of a frame group, such as n times the length of the frame group (n is a positive integer not less than 1). As a result, the video files of two adjacent unit recording periods will have n overlapping video frame groups.
[0058] Step 202: perform splicing and fusion based on each group of target video files to obtain a fused video for playback.
[0059] In practice, after the server obtains each group of target video files, it can stitch and merge each group of video files in sequence according to the time sequence. Since the local videos are not shot synchronously, the frames with the same frame number cannot be stitched in sequence. Instead, it is necessary to stitch and merge the frames with the same or similar timestamps according to the timestamps of each video frame (the timestamps after the reference is unified). Figure 3 As shown in the figure, for a set of video files (F 1_A11 、F 1_A12 、F 1_A13 ), starting from the video frame shot at time T0 (including the video frame at that time), until T 12 The video frame ends at time (excluding T 12 For a set of video files (F 2_A11 、F 2_A12 、F 2_A13 ), from T 12 The video frames shot at the moment start to be spliced. In this way, it can be ensured that a complete image can be obtained based on the decoding of the group of video files at each splicing moment, and each spliced image is shot synchronously, so the spliced fused video screen is synchronized accurately. In addition, the last frame of the fused video spliced from the previous group of video files is continuous in time with the first frame of the fused video spliced from the next group of video files, so the problem of frame loss in the fused video can be avoided.
[0060] In one embodiment, the time difference between the start time of the next unit recording period and the end time of the previous unit recording period may be n times the frame group length of the local video stream, where n is a positive integer not less than 1. The process of splicing and fusing based on each group of target video files in step 202 specifically includes the following steps:
[0061] Step 2021, for the first group of target video files in each group of target video files, determine the first timestamp of the first video frame and the second timestamp of the last video frame in each target video file, use the largest first timestamp among the first timestamps as the start time reference for splicing the group of target video files, and use the smallest second timestamp among the second timestamps as the end time reference for splicing the group of target video files.
[0062] The timestamp of each video frame is a timestamp after a unified reference, which can reflect the shooting time of each video frame. For example, the camera time of each local camera can be synchronized, and then the timestamp of each video frame is determined based on the camera timestamp.
[0063] In one implementation, based on existing video coding standards (such as H264, etc.), the timestamp of each video frame in a video file is usually in the form of time difference. Generally, the camera timestamp of the first frame in the video file is used as a reference, and the timestamp of each video frame is subtracted from the reference to obtain the video timestamp of each video frame, such as 0 for the first frame and 40 for the second frame. This timestamp can only reflect the sequence of each video frame in this video file, and cannot reflect the absolute time sequence of each video frame in different video files. For two video files shot at asynchronous times, two video frames with the same video timestamp are not shot synchronously. If splicing and fusion are performed based on this timestamp, it is impossible to obtain a fused video with accurate picture synchronization. Therefore, it is necessary to unify the benchmark of the timestamps of each video frame in the video file.
[0064] Specifically, the server can use the camera timestamp of the first key frame obtained after the recording start time S0 as the reference timestamp, and then use the difference between the camera timestamp of each video frame in each recording file and the reference timestamp as the video timestamp of each video frame, and write the video timestamp into the recording file. Figure 3 As shown, the first key frame obtained after the video start time S0 is I taken by the local camera A12. 1_A12 , so the local camera A12 can be used to shoot I 1_A12The camera timestamp at the time of recording is used as the reference timestamp. Then the difference between the camera timestamp of each video frame and the reference timestamp is written into the video file as the video timestamp. In this way, the timestamp reference of each video frame can be unified, and the video frames with the same or similar timestamps in each video file are guaranteed to be synchronously shot. Based on the video timestamps, splicing and fusion can be performed to obtain a fused video with accurate picture synchronization.
[0065] It should be noted that the first group of target video files refers to the group of target video files with the earliest recording time among the several groups of target video files corresponding to the acquired video playback time, and is not the same concept as the first group of video files after the recording start time. For example, if the first group of target video files is a group of video files recorded during the second unit recording period (F 2_A11 、F 2_A12 、F 2_A13 ), the first video frame of each target video file in the group is I m1-1_A11 ,I m1-1_A12 ,I m1-1_A13 , the last video frame is I m2_A11 ,I m2_A12 ,I m2_A13 The previous frame, or I m2-1_A11 ,I m2-1_A12 ,I m2-1_A13 The last video frame of the video frame group. m1-1_A11 ,I m1-1_A12 ,I m1-1_A13 middle, I m1-1_A11 The timestamp of I is the largest (the shooting time is the latest), m2_A11 ,I m2_A12 ,I m2_A13 middle, I m2_A12 The timestamp of the local camera A11 is the smallest (the shooting time is the earliest), so the video file F corresponding to the local camera A11 is 2_A11 The timestamp of the first video frame in the video file is used as the starting time reference for the splicing of the target video files. The video file F corresponding to the local camera A12 is 2_A12 The last video frame in m2_A11 The timestamp of the previous frame) is used as the splicing end time reference of the group of target video files.
[0066] Step 2022, for the target video files in other groups except the first group of target video files, determine the third timestamp of the first video frame and the fourth timestamp of the last video frame in each target video file, and use the timestamp of the n+1th key frame in the target video file with the largest third timestamp in the group as the start time reference for splicing the target video files in the group, and use the smallest fourth timestamp among the fourth timestamps as the end time reference for splicing the target video files in the group.
[0067] For example, for a set of target video files F 3_A11 、F 3_A12 、F 3_A13 , target video file F 3_A11 The first video frame I m2-1_A11 The timestamp of the target video file is the largest. 3_A12 The timestamp of the last video frame is the smallest, so the target video file F 3_A11 The n+1th key frame, in this example n=1, i.e. the second key frame I m2_A11 The timestamp of is used as the starting time reference for the splicing of the target video files (the next frame of the ending time reference for the splicing of the previous target video files). The target video file F 3_A12 The timestamp of the last video frame is used as the splicing end time reference of the group of target video files.
[0068] Step 2023, based on the splicing start time reference and the splicing end time reference of each group of target video files, each group of target video files is spliced and merged to obtain a merged video for playback.
[0069] During implementation, after the server determines the splicing start time reference and the splicing end time reference of each group of target video files, the video frames after the splicing start time reference (including this reference) in each group of video files can be sequentially spliced and merged until the video frame corresponding to the splicing end time reference ends (including this reference). The timestamps of the sequentially spliced video frames are the same or similar, which is synchronous video splicing. In adjacent recording time periods, the time of the last frame of the fused video in the previous recording period is continuous with the time of the first frame of the fused video in the next recording period, avoiding the frame loss problem.
[0070] In one embodiment, the server can control the two video recording modules to record video files through the collaboration module. Specifically, the user can trigger a video recording instruction (such as setting the server to automatically record when it is turned on), and the collaboration module can respond to the video recording instruction of the fused video and send a first start video recording instruction to the first video recording module, the instruction including the video recording start time S0. The first video recording module can respond to the first start video recording instruction to obtain each local video stream corresponding to the fused video, and for each local video stream, starting from the first key frame of the local video stream obtained after the current moment, record the first key frame and subsequent video frames as a video file of the current recording period.
[0071] After the collaboration module sends the first start recording instruction, the timer can be started, and when the current time reaches the start time of the next recording period, the collaboration module can send a second start recording instruction to the second recording module. The start time of the next recording period is earlier than the end time of the current recording period, and the time difference between the start time of the next recording period and the end time of the current recording period is the frame group length of the local video stream.
[0072] The second recording module can respond to the second start recording instruction to obtain the local video streams corresponding to the fused video. For each local video stream, starting from the first key frame of the local video stream obtained after the current moment, the first key frame and subsequent video frames are recorded as the recording file of the next recording period.
[0073] When the current time reaches the end time of the current recording period, the collaborative module sends a first stop recording instruction to the first recording module, and when the current time reaches the end time of the next recording period, the collaborative module sends a second stop recording instruction to the second recording module. In response to the first stop recording instruction, the first recording module writes all video frames of the video frame group where the video frame acquired at the current moment is located to the recording file of the current recording period, and then stops recording. In response to the second stop recording instruction, the second recording module writes all video frames of the video frame group where the video frame acquired at the current moment is located to the recording file of the next recording period, and then stops recording.
[0074] When the current time reaches the start time of the next recording period of the above-mentioned next recording period (which can be used as the new current recording period), a first start recording instruction can be sent to the first recording module. In this way, the recording files are recorded in a loop by the two recording modules, so that an overlapping video frame group can exist in the recording files of two adjacent recording periods to ensure the continuity of the fused video.
[0075] The above-mentioned review method of billion-pixel fusion video starts the video recording of the next recording period by starting the video recording of the next recording period before the end time of the current recording period and at a time interval of at least one frame group length, so that there is at least one overlapping video frame group data in the video files of adjacent recording periods. Since the shooting time of each local camera is not absolutely synchronized, when reviewing, the fusion video obtained based on the video file of the previous period will end with the last video frame with the smallest timestamp, and the last video frame used for splicing in other video files in the group will be an intermediate reference frame (not the last video frame) in the last video frame group in the video file. Since there is at least one overlapping video frame group data between the next period and the previous period, a video frame that is continuous with the end time of the previous period can be found in the video file of the next period, and the video file also includes the key frame of the video frame group where the video frame is located, so the continuous video frames can be decoded to obtain a complete image, so that the fusion video with accurate picture synchronization and time continuity can be spliced. Therefore, based on this scheme, the fusion video review can improve the continuity of the fusion video review while ensuring the accurate picture synchronization of the fusion video, and can solve the problem of frame loss and poor continuity of the fusion video in the related technology.
[0076] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0077] Based on the same inventive concept, the embodiment of the present application also provides a device for replaying billion-pixel fusion videos for implementing the above-mentioned method for replaying billion-pixel fusion videos. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of the device for replaying one or more billion-pixel fusion videos provided below can be referred to the limitations of the method for replaying billion-pixel fusion videos above, and will not be repeated here.
[0078] In one embodiment, Figure 4 As shown, a playback device 400 for billion-pixel fusion video is provided, including: an acquisition module 401 and a fusion module 402, wherein:
[0079] The acquisition module 401 is used to respond to the playback instruction for the fused video, determine the video playback time, and obtain several groups of target video files corresponding to the video playback time in each group of video files corresponding to each stored unit recording time period; wherein, a group of video files corresponding to a unit recording time period is obtained by recording, for each local video stream corresponding to the fused video, each video frame after the first key frame after the start time of the unit recording time period and before the last reference frame of the video frame group corresponding to the end time of the unit recording time period; of two adjacent unit recording time periods, the start time of the next unit recording time period is earlier than the end time of the previous unit recording time period, and the time difference between the start time of the next unit recording time period and the end time of the previous unit recording time period is not less than the frame group length of the local video stream.
[0080] The fusion module 402 is used to perform splicing and fusion based on each group of the target video files to obtain the fused video for playback.
[0081] In one of the embodiments, the time difference is n times the frame group length of the local video stream, where n is a positive integer not less than 1. The fusion module 402 is specifically used to: for the first group of target video files in each group of the target video files, determine the first timestamp of the first video frame and the second timestamp of the last video frame of each of the target video files in the group, use the largest first timestamp among the first timestamps as the splicing start time reference of the group of target video files, and use the smallest second timestamp among the second timestamps as the splicing end time reference of the group of target video files; for the other groups of target video files except the first group of target video files, determine the third timestamp of the first video frame and the fourth timestamp of the last video frame of each of the target video files, use the timestamp of the n+1th key frame in the target video file with the largest third timestamp in the group as the splicing start time reference of the group of target video files, and use the smallest fourth timestamp among the fourth timestamps as the splicing end time reference of the group of target video files; based on the splicing start time reference and the splicing end time reference of each group of the target video files, splice and fuse the target video files in each group to obtain the fused video for playback.
[0082] In one embodiment, the time difference is the frame group duration of the local video stream. The device also includes a recording module, which is used to respond to the recording instruction of the fused video, and determine the start time and end time of each unit recording period according to the recording start time, the unit recording duration and the frame group duration of the local video stream, wherein, in two adjacent unit recording periods, the start time of the next unit recording period is earlier than the end time of the previous unit recording period, and the time difference between the start time of the next unit recording period and the end time of the previous unit recording period is the frame group duration; when the current time reaches the start time of the current unit recording period, for each local video stream, the video frames after the first key frame after the start time of the current unit recording period and before the last reference frame of the video frame group corresponding to the end time of the current unit recording period in the local video stream are recorded as the recording file corresponding to the current unit recording period, and the recording files of each local video stream in the same unit recording period are stored as a group of recording files.
[0083] In one of the embodiments, the device also includes a writing module, which is used to use the camera timestamp of the first key frame obtained after the start time of the recording as a reference timestamp; use the difference between the camera timestamp of each video frame in each of the recording files and the reference timestamp as the video timestamp of each of the video frames, and write the video timestamp into the recording file.
[0084] In one embodiment, the device includes a collaboration module and two video recording modules, wherein:
[0085] The cooperation module is used for sending a first start recording instruction to the first recording module in response to the recording instruction of the fused video.
[0086] The first recording module is used to respond to the first start recording instruction, obtain the local video streams corresponding to the fused video, and for each of the local video streams, starting from the first key frame of the local video stream obtained after the current moment, record the first key frame and subsequent video frames as a recording file of the current unit recording period.
[0087] The collaboration module is further configured to send a second start recording instruction to the second recording module when the current time reaches the start time of the next unit recording period, wherein the start time of the next unit recording period is earlier than the end time of the current unit recording period, and the time difference between the start time of the next unit recording period and the end time of the current unit recording period is the frame group duration of the local video stream.
[0088] The second recording module is used to respond to the second start recording instruction, obtain the local video streams corresponding to the fused video, and for each of the local video streams, starting from the first key frame of the local video stream obtained after the current moment, record the first key frame and subsequent video frames as the recording file of the next unit recording period.
[0089] The cooperation module is further configured to send a first stop recording instruction to the first recording module when the current time reaches the end time of the current unit recording period.
[0090] The first recording module is further configured to, in response to the first recording stop instruction, stop recording after writing all video frames of the video frame group in which the video frame acquired at the current moment belongs to the video file of the current unit recording period.
[0091] The cooperation module is further configured to send a second stop recording instruction to the second recording module when the current time reaches the end time of the next unit recording period.
[0092] The second recording module is further configured to respond to the second recording stop instruction, write all video frames of the video frame group in which the video frame acquired at the current moment belongs to the video file of the next unit recording period, and then stop recording.
[0093] Each module in the above-mentioned device for reviewing billion-pixel fusion video can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0094] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required or generated for executing the above-mentioned method for reviewing billion-level pixel fusion videos. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for reviewing billion-level pixel fusion videos is implemented.
[0095] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0096] In one embodiment, a billion-pixel computational imaging system is provided. The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes multiple local cameras, wherein:
[0097] The array camera is used to shoot local videos through each local camera and send each local video stream to the server;
[0098] The server is used to respond to the recording instruction of the fused video and store the acquired local video streams as multiple groups of recording files of multiple unit recording time periods; wherein a group of recording files corresponding to a unit recording time period is obtained by recording, for each local video stream corresponding to the fused video, the video frames after the first key frame after the start time of the unit recording time period and before the last reference frame of the video frame group corresponding to the end time of the unit recording time period; in two adjacent unit recording time periods, the start time of the next unit recording time period is earlier than the end time of the previous unit recording time period, and the time difference between the start time of the next unit recording time period and the end time of the previous unit recording time period is not less than the frame group length of the local video stream;
[0099] The server is also used to respond to the playback instruction for the fused video, determine the video playback time, and obtain several groups of target video files corresponding to the video playback time from each group of video files corresponding to each unit recording time period stored; and perform splicing and fusion based on each group of target video files to obtain a fused video for playback.
[0100] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.
[0101] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0102] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0103] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0105] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0106] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for reviewing billion-pixel fusion videos. It is characterized in that The method comprises: In response to a review instruction for the fused video, determine a video review time, and obtain a plurality of groups of target video files corresponding to the video review time from each group of video files corresponding to each unit recording time period stored; Among them, a group of video files corresponding to a unit recording period is obtained by recording, for each local video stream corresponding to the fused video, each video frame after the first key frame after the start time of the unit recording period and before the last reference frame of the video frame group corresponding to the end time of the unit recording period; in two adjacent unit recording periods, the start time of the next unit recording period is earlier than the end time of the previous unit recording period, and the time difference between the start time of the next unit recording period and the end time of the previous unit recording period is not less than the frame group length of the local video stream; Splicing and fusion are performed based on each group of the target video files to obtain the fused video for playback.
2. The method according to claim 1, It is characterized in that The time difference is n times the frame group duration of the local video stream, where n is a positive integer not less than 1; the splicing and fusion based on each group of the target video files to obtain the fused video for playback includes: For a first group of target video files in each group of the target video files, determine a first timestamp of a first video frame and a second timestamp of a last video frame of each of the target video files in the group, use the largest first timestamp among the first timestamps as a start time reference for splicing the group of target video files, and use the smallest second timestamp among the second timestamps as an end time reference for splicing the group of target video files; For the target video files of other groups except the first group of target video files, determine the third timestamp of the first video frame and the fourth timestamp of the last video frame of each target video file, use the timestamp of the n+1th key frame in the target video file with the largest third timestamp in the group as the start time reference for splicing the target video files of the group, and use the smallest fourth timestamp among the fourth timestamps as the end time reference for splicing the target video files of the group; Based on the splicing start time reference and the splicing end time reference of each group of the target video files, each group of the target video files is spliced and fused to obtain the fused video for playback.
3. The method according to claim 1, It is characterized in that The time difference is the frame group duration of the local video stream; Before responding to the review instruction for the fused video, the method further includes: In response to a recording instruction of the fused video, the start time and the end time of each unit recording period are determined according to the recording start time, the unit recording duration and the frame group duration of the local video stream, wherein, in two adjacent unit recording periods, the start time of the next unit recording period is earlier than the end time of the previous unit recording period, and the time difference between the start time of the next unit recording period and the end time of the previous unit recording period is the frame group duration; When the current time reaches the start time of the current unit recording period, for each of the local video streams, the video frames after the first key frame after the start time of the current unit recording period and before the last reference frame of the video frame group corresponding to the end time of the current unit recording period are recorded as a recording file corresponding to the current unit recording period, and the recording files of each local video stream in the same unit recording period are stored as a group of recording files.
4. The method according to claim 3, It is characterized in that The method further comprises: The camera timestamp of the first key frame obtained after the start time of the video recording is used as the reference timestamp; The difference between the camera timestamp of each video frame in each video file and the reference timestamp is used as the video timestamp of each video frame, and the video timestamp is written into the video file.
5. The method according to claim 1, It is characterized in that The method is applied to a server, the server includes a collaboration module and two video recording modules, and the time difference is the frame group duration of the local video stream; Before responding to the review instruction for the fused video, the method further includes: The collaborative module responds to the recording instruction of the fused video by sending a first start recording instruction to the first recording module. The first recording module responds to the first start recording instruction by acquiring each channel of local video streams corresponding to the fused video, and for each channel of the local video stream, starting from the first key frame of the local video stream acquired after the current moment, records the first key frame and subsequent video frames as a recording file of the current unit recording period. When the current time reaches the start time of the next unit recording period, the collaboration module sends a second start recording instruction to the second recording module. The second recording module responds to the second start recording instruction to obtain each channel of local video streams corresponding to the fused video, and for each channel of the local video stream, starting from the first key frame of the local video stream obtained after the current time, the first key frame and subsequent video frames are recorded as the video file of the next unit recording period. When the current time reaches the end time of the current unit recording period, the collaboration module sends a first stop recording instruction to the first recording module, and the first recording module responds to the first stop recording instruction, writes all video frames of the video frame group where the video frame acquired at the current moment is located into the video file of the current unit recording period, and then stops recording; When the current time reaches the end time of the next unit recording period, the collaboration module sends a second stop recording instruction to the second recording module, and the second recording module responds to the second stop recording instruction, writes all video frames of the video frame group where the video frame acquired at the current moment is located into the video file of the next unit recording period, and then stops recording; The start time of the next unit recording period is earlier than the end time of the current unit recording period, and the time difference between the start time of the next unit recording period and the end time of the current unit recording period is the frame group duration of the local video stream.
6. A device for reviewing billion-pixel fusion video. It is characterized in that The device comprises: An acquisition module is used to respond to a review instruction for a fused video, determine a video review time, and obtain several groups of target video files corresponding to the video review time from each group of video files corresponding to each stored unit recording period; wherein a group of video files corresponding to a unit recording period is obtained by recording, for each local video stream corresponding to the fused video, each video frame after the first key frame after the start time of the unit recording period and before the last reference frame of the video frame group corresponding to the end time of the unit recording period; of two adjacent unit recording periods, the start time of the next unit recording period is earlier than the end time of the previous unit recording period, and the time difference between the start time of the next unit recording period and the end time of the previous unit recording period is not less than the frame group length of the local video stream; The fusion module is used to perform splicing and fusion based on each group of the target video files to obtain the fused video for playback.
7. A billion-pixel computational imaging system, It is characterized in that The billion-pixel computational imaging system includes an array camera and a server, wherein the array camera includes a plurality of local cameras, wherein: The array camera is used to shoot local videos through each of the local cameras and send each local video stream to the server; The server is used to respond to the recording instruction of the fused video and store the acquired local video streams as multiple groups of recording files of multiple unit recording time periods; wherein a group of recording files corresponding to a unit recording time period is obtained by recording, for each local video stream corresponding to the fused video, each video frame after the first key frame after the start time of the unit recording time period and before the last reference frame of the video frame group corresponding to the end time of the unit recording time period; in two adjacent unit recording time periods, the start time of the next unit recording time period is earlier than the end time of the previous unit recording time period, and the time difference between the start time of the next unit recording time period and the end time of the previous unit recording time period is not less than the frame group length of the local video stream; The server is also used to respond to a review instruction for the fused video, determine the video review time, and obtain several groups of target video files corresponding to the video review time from each group of video files corresponding to each unit recording time period stored; and perform splicing and fusion based on each group of the target video files to obtain the fused video for review.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Hundred million-level pixel video playback method and device, medium and equipment
CN111601151A
Multi-video automatic splicing method and system for track and field project, and medium
CN114401378A
Audio processing method and electronic equipment
CN116052701A
Using worker nodes in a distributed video encoding system
US20170078376A1
