High-efficiency real-time conference video synthesis system
Through an efficient real-time conference video synthesis system, the audio and video streams of multiple users are obtained and processed in real time, and the recording and synthesis scripts are generated, which solves the problem of insufficient recording integration and automatic recording functions in the existing technology, and achieves high-quality, real-time and flexible conferences Video management and storage.
Patent Information
- Application Number
- CN202411324415.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-31
AI Technical Summary
The existing video conferencing system cannot effectively integrate multi-user video streams, resulting in limited real-time and completeness of conference recordings. The automatic recording function cannot meet the user's flexible needs, increasing the storage burden and difficulty of searching.
An efficient real-time conference recording synthesis system was designed, including a conference web client, conference synthesis microservice module and video stream forwarding server. By obtaining and processing audio and video streams in real time, a video synthesis script is generated, and video editing multimedia framework is used to record. Synthesis and persistence save, support user control of recording start and stop, update recording styles in real time, and convert them into standardized MP4 format through FFmpeg.
Real-time and synchronization of video recording, simplify user operations, improve the quality and storage efficiency of video recording, ensure safe storage and fast acquisition of video recording files, adapt to multi-user scenarios, provide high-quality video output, and reduce format Incompatibility issues and storage pressure.
Smart Images
Figure CN119232866A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video conferencing and multimedia data processing, and in particular to an efficient real-time conference video synthesis system. Background Art
[0002] With the rapid development of information technology, remote collaboration and online meetings have become increasingly important. They are key tools for business communication, education and social interaction. Video conferencing systems enable face-to-face communication across geographical boundaries by transmitting audio and video streams in real time. With the growing demand for remote collaboration and online meetings, the demand for high-quality recording and post-processing of meeting content is also increasing.
[0003] In a multi-user conference environment, existing conference systems have some obvious limitations. Specifically, the system can only generate independent source video files for each participant that are processed by the video stream forwarding server. This decentralized video file management method does not provide a mechanism to integrate these fragments into a coherent conference recording, which affects the integrity of the conference record and the viewing experience. In addition, due to the lack of the ability to integrate video streams in real time, the real-time nature of the conference recording is also affected, which is particularly inconvenient when the conference content needs to be reviewed immediately. On the other hand, although the existing conference system has the function of automatic recording and can automatically save all source video files, this automatic recording method does not take into account the actual needs of users. In many cases, users may only need to record the key parts of the meeting instead of the entire meeting process. This indiscriminate recording not only increases the storage burden, but also makes it more difficult and inefficient for users to find and use specific meeting content later. Summary of the invention
[0004] The present invention provides an efficient real-time conference video synthesis system, which can solve the above problems.
[0005] In order to solve the above problems, the technical solution adopted by the present invention is as follows:
[0006] The present invention provides an efficient real-time conference video synthesis system, comprising:
[0007] The conference web client is used to control the start and stop of recording, and to update the recording style in real time according to the increase and decrease of users and the adjustment of the video window, and obtain the audio and video stream of each user;
[0008] The conference synthesis microservice module is used to process the video style, process the independent video files, generate the video synthesis script according to the processed independent video files and video styles, call the video editing multimedia framework to perform video synthesis, and save the synthesized video files persistently to the video stream forwarding server;
[0009] Video stream forwarding server, used to convert each user's audio and video stream into an independent video file;
[0010] Video editing multimedia framework, used to execute video synthesis scripts and complete video synthesis.
[0011] As a further description of the above technical solution, the conference synthesis microservice module includes a recording style processing module for processing recording styles, an original recording processing module for processing independent recording files, a synthesis script generation module for generating a recording synthesis script based on the processed independent recording files and recording styles, and a fusion recording persistence module for calling the video editing multimedia framework for recording synthesis and persisting the synthesized recording files to the video stream forwarding server.
[0012] As a further description of the above technical solution, the conference web client and the conference synthesis microservice module, and the conference synthesis microservice module and the video stream forwarding server communicate via the HTTP protocol; the conference web client communicates with the video stream forwarding server via the WebRTC protocol.
[0013] As a further description of the above technical solution, the video stream forwarding server converts the user's audio and video stream into an independent MP4 format video file through FFmpeg software, and stores it in the video stream forwarding server.
[0014] As a further description of the above technical solution,
[0015] When the user clicks to start recording in the conference web client, the conference web client immediately sends an http request to the conference synthesis microservice module, which records the start time of this recording and activates the recording style processing module; the conference web client captures the coordinates and size of each video window, and saves the coordinates and size of each video window to the recording style processing module in real time through an http request;
[0016] When the user clicks to end recording in the conference web client, the conference web client sends an http request to the conference synthesis microservice module again. The conference synthesis microservice module records the end time of this recording and stops capturing the coordinates and size of the video window. Subsequently, the conference synthesis microservice module calls the original recording processing module, obtains each independent recording file from the video stream forwarding server through an http request, and processes them.
[0017] As a further description of the above technical solution, the process of the original video processing module processing independent video files includes: filtering and deleting invalid independent video files, invalid independent video files refer to files whose end time is earlier than the start time of the video recording and whose start time is later than the end time of the video recording, and all retained valid files are persisted and saved to the video stream forwarding server.
[0018] As a further description of the above technical solution, the video synthesis script has a video file segmentation mechanism, that is, for a certain video file, if its duration exceeds one hour, it will be automatically segmented into multiple video files.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] 1) The system can obtain the audio and video streams of each user in real time, and dynamically update the recording style according to the increase or decrease of users and the video window adjustment action. This real-time adjustment mechanism avoids the problem of inconsistent recording content caused by recording delay or asynchrony in traditional systems, thereby improving the real-time and synchronization of recording; Through the conference web client, users can easily control the start and stop of recording and can directly affect the final recording style. This simplified operation interface greatly improves the user-friendliness of the system, allowing users to complete conference recording without complicated operation steps, greatly improving the user satisfaction and efficiency; The conference synthesis microservice module can automatically generate a recording synthesis script and call the video editing multimedia framework for recording synthesis. The synthesized recording file can be efficiently and automatically saved to the video stream forwarding server, ensuring the safe storage and fast acquisition of the recording file; It can adapt to multi-user conference scenarios, and can realize flexible recording synthesis by processing independent recording files and generating synthesis scripts according to different recording styles; Whether it is a single-person presentation, a multi-person discussion, or a complex multi-window scenario, the system can provide high-quality recording output.
[0021] 2) Use FFmpeg software to convert the user's audio and video streams into independent MP4 format video files and store them in the video stream forwarding server, ensuring the standardization and high compatibility of the video files. In this way, the system can achieve efficient and stable audio and video recording, reduce playback or editing problems caused by format incompatibility, and also lay a solid foundation for subsequent video synthesis and distribution, improving the overall performance of the system and user experience.
[0022] 3) Through the interaction between the conference web client and the conference synthesis microservice module, precise control and dynamic adjustment of conference recording are achieved; when the user starts recording, the system can record the start time in real time and activate the recording style processing module, capture and save the coordinates and size of each video window, ensuring the accuracy and flexibility of the layout during the recording process. When the recording ends, the system records the end time and stops capturing video window information, and then obtains and processes each independent recording file through the original recording processing module. This design not only ensures the integrity and high-quality synthesis of the recording, but also improves the accuracy of recording control and the degree of automation of the system, greatly simplifying user operations.
[0023] 4) Filter and delete invalid independent video files through the original video processing module to ensure that the final saved files are valid meeting content. In this way, the system can reduce the storage space occupied by invalid data, improve storage efficiency, and ensure the accuracy and completeness of the final generated video files, thereby improving the overall recording quality and user experience.
[0024] 5) Through the file segmentation mechanism in the video synthesis script, video files longer than one hour are automatically segmented into multiple parts. This design effectively avoids the storage and transmission pressure caused by oversized files, improves the flexibility and convenience of file management, and also ensures the smoothness of post-processing and playback, which helps to optimize the user experience.
[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the embodiments of the present invention are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0027] Figure 1 is a schematic block diagram of the structure of the efficient real-time conference video synthesis system described in the embodiment;
[0028] Figure 2 This is the first interface image after the conference web client starts recording the conference in the embodiment;
[0029] Figure 3 This is a second interface diagram after the conference web client starts recording the conference in the embodiment;
[0030] Figure 4 It is an execution flow chart of the video synthesis script described in the embodiment. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.
[0032] Please refer to Figure 1 This embodiment provides an efficient real-time conference video synthesis system, including a conference web client 1, a conference synthesis microservice module 2, a video stream forwarding server 3 and a video editing multimedia framework 4.
[0033] The conference web client 1 is used to control the start and stop of the video recording, and to update the video recording style in real time according to the increase and decrease of users and the video window adjustment action, and to obtain the audio and video stream of each user.
[0034] The conference synthesis microservice module 2 includes a video style processing module 21, an original video processing module 22, a synthesis script generation module 23 and a fusion video persistence module 24. The video style processing module 21 is used to process the video style. The original video processing module 22 is used to process independent video files. The synthesis script generation module 23 is used to generate a video synthesis script based on the processed independent video files and video styles. The fusion video persistence module 24 is used to call the video editing multimedia framework 4; the video editing multimedia framework 4 is used to execute the video synthesis script to complete the video synthesis; the fusion video persistence module 24 persists the synthesized video file to the video stream forwarding server 3.
[0035] The audio and video stream interaction between the conference web client 1 and the video stream forwarding server 3 is realized through the webrtc technology. The video stream forwarding server 3 is responsible for receiving the audio and video streams of each user transmitted by webrtc, and converting them into independent video files in MP4 format through FFmpeg software. These independent video files are then stored in the file system of the video stream forwarding server 3, and the specific path is as follows:
[0036] / transserver / www / record / 2024-07-31 / live / 12bf605f-22be-4c98-aa85-e6ceffd14cbc-234844 / 05-26-13.mp4.
[0037] The naming of each video file contains a timestamp to ensure the uniqueness and order of the file. The metadata of the video file contains the start timestamp, duration, conference ID, and user ID. The start timestamp is the starting time point of recording the video stream; the duration is the duration of the video stream in seconds; the conference ID is an identifier that uniquely identifies the relevant conference; and the user ID is a unique identifier for the user participating in the conference.
[0038] The conference web client 1 and the conference synthesis microservice module 2, and the conference synthesis microservice module 2 and the video stream forwarding server 3 communicate via the http protocol.
[0039] When the user clicks to start recording in the conference web client 1, the conference web client 1 immediately sends an http request carrying the conference ID to the conference synthesis microservice module 2. The conference synthesis microservice module 2 generates a record of the current recording including the start time of the recording and the conference ID, and the recording style processing module 21 is activated. Figure 2 , Figure 3 As shown, after starting recording, the conference web client 1 captures the coordinates and size of each video window, and saves the coordinates and size of each video window to the recording style processing module 21 in real time through an http request, and the recording style processing module 21 processes the recording style. In the processing of the recording style, it is necessary to establish an association relationship for the video source in the layout. By comparing the start time of the layout with the start time of the video, and comparing the end time of the layout with the end time of the video, the layout with the smallest end time difference is selected to establish an association with the video source. The start and end times of these associations are marked in the original video source list to facilitate subsequent video capture operations.
[0040] When the user clicks to end the recording on the conference web client 1, the conference web client 1 sends an http request to the conference synthesis microservice module 2 again. The conference synthesis microservice module 2 records the end time of this recording and stops capturing the coordinates and size of the video window. Subsequently, the conference synthesis microservice module 2 calls the original recording processing module 22, which obtains the independent recording files of each user from the video stream forwarding server 3 through an http request and processes them.
[0041] The process of the original video processing module 22 processing the independent video file includes:
[0042] Filter and delete invalid independent video files, which refer to files whose end time is earlier than the start time of recording and whose start time is later than the end time of recording, and permanently save all the valid files to the video stream forwarding server 3.
[0043] Next, the synthesis script generation module 23 generates a video synthesis script according to the processed independent video file and the video style, as follows:
[0044] In order to manage large video files, the present invention designs a video file segmentation mechanism, that is, for a certain video file, if its recording time exceeds one hour, it will be automatically segmented into multiple video files, and the synthetic script generation module 23 will establish the association relationship between these segmented files. If the end time of a video file with the same user ID is greater than or equal to the start time of the subsequent video file, then the two video files are considered to be associated. This segmentation strategy not only facilitates video retrieval and access, but also helps to improve storage efficiency and flexibility of post-processing.
[0045] In this embodiment, the video synthesis script is a Melt rule script for video synthesis. Figure 4 As shown, the generation process of the video synthesis script involves traversing and combining the processed independent video files and video style files. The video synthesis script is generated based on the following steps:
[0046] 1) Define video synthesis parameters: Use the -profile parameter to specify the resolution and frame rate of the synthesized video. For example, hdv_720_25p represents 720p resolution and 25fps frame rate.
[0047] 2) Set the video duration: Use the colour:black out=length parameter to define the total duration of the video after synthesis. The duration is calculated based on the start time of the conference recording and the actual recording duration.
[0048] 3) Add video source: Add video source through the -track parameter. Each video source includes:
[0049] -blank: Specifies the time difference between the video source and the start of the composite video.
[0050] -attach: Specifies the text attached to the video source, usually used to display the user name.
[0051] 4) Specify the video source file: Use the . / 1.mp4 parameter to specify the video source file to be used. These files are generated by the original video processing module 22.
[0052] 5) Define video capture parameters: Use the length=, in=, and out= parameters to define the capture duration, start point, and end point of the video source.
[0053] 6) Apply a clip: Use the -transition composite parameter to describe the clip of the video source, including size, position, start and end time.
[0054] 7) Audio processing: Use the -transition mix parameter to define the start and end times of the audio clip.
[0055] 8) Composite video output: Use the -consumer avformat parameter to specify the output format and codec of the composite video.
[0056] In this embodiment, the fusion video persistence module 24 plays a key role. It is responsible for calling the video editing multimedia framework 4 to execute the video synthesis script and complete the video synthesis task.
[0057] During the synthesis process, the video editing multimedia framework 4 follows the detailed instructions of the video synthesis script to accurately merge the independent video files of different users, ensuring that the final generated conference video is not only coherent and smooth, but also maintains high quality standards. After the synthesis task is completed, the fusion video persistence module 24 further ensures the long-term value and practicality of the video, and persists the synthesized video file to the video stream forwarding server 3. This step not only ensures the security of the video data, but also enhances the accessibility of the video, allowing users to trace back and access these important materials at any time in the future.
[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An efficient real-time conference video synthesis system, characterized in that: include: The conference web client is used to control the start and stop of recording, and to update the recording style in real time according to the increase and decrease of users and the adjustment of the video window, and obtain the audio and video stream of each user; The conference synthesis microservice module is used to process the video style, process the independent video files, generate the video synthesis script according to the processed independent video files and video styles, call the video editing multimedia framework to perform video synthesis, and save the synthesized video files persistently to the video stream forwarding server; Video stream forwarding server, used to convert each user's audio and video stream into an independent video file; Video editing multimedia framework, used to execute video synthesis scripts and complete video synthesis.
2. The efficient real-time conference video synthesis system according to claim 1 is characterized in that: The conference synthesis microservice module includes a recording style processing module for processing recording styles, an original recording processing module for processing independent recording files, a synthesis script generation module for generating a recording synthesis script based on the processed independent recording files and recording styles, and a fusion recording persistence module for calling the video editing multimedia framework for recording synthesis and persisting the synthesized recording files to the video stream forwarding server.
3. The efficient real-time conference video synthesis system according to claim 1 is characterized in that: The conference web client and the conference synthesis microservice module, and the conference synthesis microservice module and the video stream forwarding server communicate through the HTTP protocol; the conference web client communicates with the video stream forwarding server through the WebRTC protocol.
4. The efficient real-time conference video synthesis system according to claim 3 is characterized in that: The video stream forwarding server converts the user's audio and video streams into independent MP4 format video files through FFmpeg software and stores them in the video stream forwarding server.
5. The efficient real-time conference video synthesis system according to claim 3 is characterized in that: When the user clicks to start recording in the conference web client, the conference web client immediately sends an HTTP request to the conference synthesis microservice module. The conference synthesis microservice module records the start time of this recording and activates the recording style processing module. The conference web client captures the coordinates and size of each video window, and saves the coordinates and size of each video window to the video style processing module in real time through HTTP request.
6. The efficient real-time conference video synthesis system according to claim 5 is characterized in that: When the user clicks to end recording in the conference web client, the conference web client sends an http request to the conference synthesis microservice module again. The conference synthesis microservice module records the end time of this recording and stops capturing the coordinates and size of the video window. Subsequently, the conference synthesis microservice module calls the original recording processing module, obtains each independent recording file from the video stream forwarding server through an http request, and processes them.
7. The efficient real-time conference video synthesis system according to claim 6 is characterized in that: The process of processing independent video files by the original video processing module includes: filtering and deleting invalid independent video files, which refer to files whose end time is earlier than the start time of recording and whose start time is later than the end time of recording, and persisting all retained valid files to the video stream forwarding server.
8. The efficient real-time conference video synthesis system according to claim 5 is characterized in that: The video synthesis script has a video file segmentation mechanism, that is, for a video file, if its duration exceeds one hour, it will be automatically segmented into multiple video files.