Method and system for recording double-stream video conference, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a method and a system for recording a double-stream video conference, which are applied to a first server, and the method comprises the following steps: obtaining audio and video stream data to be recorded; analyzing first video stream data and second video stream data from the audio and video stream data; the first video stream data and the second video stream data are recorded respectively to obtain a first video file and a second video file, respective processing identification information is added, and the processing identification information is used for identifying and distinguishing different video files; and storing the first video file, the second video file and the processing identification information corresponding to the first video file and the second video file in a database. The embodiment of the invention supports effective file management and retrieval functions. The problems that multiple paths of video streams cannot be recorded at the same time and main and auxiliary streams are asynchronous in the prior art are solved, and the recording efficiency and the information integrity are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video conferencing, and in particular to a dual-stream video conference recording method, a dual-stream video conference recording system, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the continuous development of modern communication technology and video conferencing systems, remote conferencing has become an indispensable part of enterprises, education and various organizations. Through video conferencing, participants can achieve remote real-time communication, reduce geographical restrictions and improve work efficiency. However, existing video conferencing systems still face many technical limitations when recording multiple video streams (such as the speaker's screen and the screen of shared content), and cannot meet the growing user needs.

[0003] In actual video conferencing, there are usually two main types of video streams: mainstream and auxiliary. The mainstream is usually used to transmit the speaker's screen, while the auxiliary is used to transmit screen sharing content, presentation documents or other related display information. These two types of video streams each have important functions in transmitting conference information. In order to replay or archive the conference later, it is particularly important to accurately and synchronously record the content of the mainstream and auxiliary streams. However, most current video conferencing systems can only support the recording of a single video stream, and cannot record both the mainstream and auxiliary streams at the same time. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a dual-stream video conference recording method, a dual-stream video conference recording system, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0005] In order to solve the above problem, an embodiment of the present invention discloses a recording method of a dual-stream video conference, which is applied to a first server, and the method includes:

[0006] Get the audio and video stream data to be recorded;

[0007] Analyzing the first video stream data and the second video stream data from the audio and video stream data;

[0008] Recording the first video stream data and the second video stream data respectively to obtain a first video file and a second video file, and adding respective processing identification information, wherein the processing identification information is used to identify and distinguish different video files;

[0009] The first video file, the second video file and the corresponding processing identification information are stored in a database.

[0010] Optionally, analyzing the first video stream data and the second video stream data from the audio and video stream data includes:

[0011] Read each video frame data of the audio and video stream data;

[0012] Screening and classifying each of the video frame data according to the metadata of each of the video frame data to obtain first video frame data and second video frame data;

[0013] The first video frame data and the second video frame data are reconstructed into the first video stream data and the second video stream data respectively.

[0014] Optionally, before respectively reconstructing the first video frame data and the second video frame data into the first video stream data and the second video stream data, the method further includes:

[0015] Synchronize the first video frame data and the second video frame data according to the synchronization information in the metadata;

[0016] The synchronization information includes a timestamp or a frame number.

[0017] Optionally, when the first video stream data and the second video stream data are recorded respectively to obtain a first video file and a second video file, the method further includes:

[0018] Detecting a frame rate or a packet loss rate of the first video frame data and the second video frame data;

[0019] Determine whether the first video frame data and / or the second video frame data is lost according to the frame rate or the packet loss rate;

[0020] If the first video frame data or the second video frame data is lost, sending video stream check notification information;

[0021] If both the first video frame data and the second video frame data are lost, the recording process is stopped and the video stream check notification information is sent.

[0022] Optionally, obtaining the audio and video stream data to be recorded includes:

[0023] Acquire the audio and video stream data from the second server;

[0024] The audio and video stream data is obtained by performing alternating frame encoding on the first video stream data and the second video stream data.

[0025] Optionally, when the first video stream data and the second video stream data are recorded respectively to obtain a first video file and a second video file, the method further includes:

[0026] Sending recording status notification information to the second server so that the second server forwards the recording status notification information to the client, and the client is used to update the recording status of the dual-stream video conference according to the recording status notification information.

[0027] Optionally, the processing identification information includes at least one of the following:

[0028] Video stream identification information, video start and end time information, and video thumbnail information.

[0029] The embodiment of the present invention further discloses a dual-stream video conference recording system, which is applied to a first server, and the system includes:

[0030] An audio and video stream data acquisition module, used to acquire the audio and video stream data to be recorded;

[0031] A video stream data analysis module, used for analyzing the audio and video stream data to obtain first video stream data and second video stream data;

[0032] a video stream data recording module, used to record the first video stream data and the second video stream data to obtain a first video file and a second video file, respectively, and add respective processing identification information, wherein the processing identification information is used to identify and distinguish different video files;

[0033] The video file storage module is used to store the first video file, the second video file and the corresponding processing identification information in a database.

[0034] Optionally, the video stream data analysis module includes:

[0035] A video frame data reading module, used to read each video frame data of the audio and video stream data;

[0036] A video frame data screening and classification module, used for screening and classifying each of the video frame data according to the metadata of each of the video frame data to obtain first video frame data and second video frame data;

[0037] The video stream data reconstruction module is used to reconstruct the first video frame data and the second video frame data into the first video stream data and the second video stream data respectively.

[0038] Optionally, the system further comprises:

[0039] a video frame data synchronization processing module, configured to perform synchronization processing on the first video frame data and the second video frame data according to the synchronization information in the metadata before the video stream data reconstruction module reconstructs the first video frame data and the second video frame data into the first video stream data and the second video stream data respectively;

[0040] The synchronization information includes a timestamp or a frame number.

[0041] Optionally, the system further comprises:

[0042] a video frame data detection module, configured to detect a frame rate or a packet loss rate of the first video frame data and the second video frame data when the video stream data recording module respectively records the first video stream data and the second video stream data to obtain a first video file and a second video file;

[0043] a loss situation determination module, used to determine whether the first video frame data and / or the second video frame data is lost according to the frame rate or the packet loss rate;

[0044] A check notification sending module, configured to send video stream check notification information if the first video frame data or the second video frame data is lost;

[0045] The recording stop module is used to stop the recording process and send the video stream check notification information if both the first video frame data and the second video frame data are lost.

[0046] Optionally, the audio and video stream data acquisition module is used to acquire the audio and video stream data from a second server;

[0047] The audio and video stream data is obtained by performing alternating frame encoding on the first video stream data and the second video stream data.

[0048] Optionally, the system further comprises:

[0049] A recording status update module is used to send recording status notification information to the second server when the video stream data recording module records the first video stream data and the second video stream data respectively to obtain the first video file and the second video file, so that the second server forwards the recording status notification information to the client, and the client is used to update the recording status of the dual-stream video conference according to the recording status notification information.

[0050] Optionally, the processing identification information includes at least one of the following:

[0051] Video stream identification information, video start and end time information, and video thumbnail information.

[0052] An embodiment of the present invention also discloses an electronic device, comprising: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enables the electronic device to execute the dual-stream video conference recording method as described above.

[0053] The embodiment of the present invention further discloses a computer-readable storage medium, wherein the computer program stored in the storage medium enables a processor to execute the dual-stream video conference recording method as described above.

[0054] The embodiments of the present invention include the following advantages:

[0055] The recording solution for dual-stream video conferencing provided by an embodiment of the present invention is applied to a first server. The audio and video stream data to be recorded is obtained, and then the first video stream data and the second video stream data are analyzed from the audio and video stream data. Next, the first video stream data and the second video stream data are recorded respectively to obtain a first video file and a second video file, and respective processing identification information is added. The processing identification information is used to identify and distinguish different video files. Finally, the first video file, the second video file and the corresponding processing identification information are stored in a database.

[0056] The embodiment of the present invention can simultaneously record two video streams, namely, the first video stream data (such as the mainstream) and the second video stream data (such as the auxiliary stream), to ensure that the speaker's screen and screen sharing content (such as a presentation document) are completely preserved. The problem that the prior art can only record a single video stream is solved, and the loss of important conference information is avoided, thereby improving the recording quality of the dual-stream video conference. By processing the first video stream data and the second video stream data separately and adding processing identification information, the synchronization of multiple video streams during the recording process is ensured. This enables the speaker's screen and the presentation content to be accurately aligned during playback, improving the playback experience and information consistency. The first video stream data and the second video stream data are recorded as independent video files respectively, and processing identification information is added to each video file to distinguish the type of video stream and its corresponding content, which simplifies the subsequent video file storage, retrieval and use, and improves management efficiency. By storing the recorded first video file and the second video file and their corresponding processing identification information (such as timestamp, video type, etc.) together in a database, the traceability of the dual-stream video conference recording is improved. Users can easily perform accurate retrieval based on information such as meeting time and video type, supporting efficient archiving and subsequent calls, and enhancing the flexibility and practicality of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1It is a flowchart of the steps of a recording method of a dual-stream video conference according to an embodiment of the present invention;

[0058] Figure 2 It is a schematic diagram of a recording interface of a dual-stream recording solution for video conferencing based on a visual network according to an embodiment of the present invention;

[0059] Figure 3 It is a structural block diagram of a dual-stream video conference recording system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] The recording scheme for dual-stream video conferencing proposed in the embodiment of the present invention realizes the synchronous recording of the mainstream (speaker's screen) and the auxiliary stream (such as screen sharing, presentation documents) by simultaneously processing and recording multiple video streams. Specifically, the audio and video stream data to be recorded is obtained, the mainstream and the auxiliary streams are analyzed and distinguished, and they are recorded independently to generate mainstream video files and auxiliary stream video files, and processing identification information, such as timestamp and synchronization information, is added to each video file to ensure accurate alignment during subsequent playback. After the recording is completed, the main and auxiliary stream video files and their corresponding processing identification information are stored in the database, supporting effective file management and retrieval functions. This solution solves the problem in the prior art that multiple video streams cannot be recorded simultaneously and the main and auxiliary streams are not synchronized, and improves recording efficiency and information integrity.

[0062] Reference Figure 1 , shows a flowchart of a method for recording a dual-stream video conference according to an embodiment of the present invention. The method for recording a dual-stream video conference can be applied to a first server. The method for recording a dual-stream video conference can specifically include the following steps:

[0063] Step 101, obtaining audio and video stream data to be recorded.

[0064] In the embodiment of the present invention, the audio and video stream data refers to the conference audio and video data stream transmitted through the network, which usually includes the voices of conference participants, the screen of the speaker, the shared screen of the presentation document, etc.

[0065] In a dual-stream video conference, the sources of audio and video stream data are usually divided into multiple parts. For example, when the conference speaker is speaking, the video images captured by his camera and the sound recorded by the microphone will form the mainstream audio and video stream. At the same time, when the speaker shares the screen to show the slides, a secondary audio and video stream will be formed. The recording system needs to obtain these audio and video stream data from the conference server or terminal device, usually through the Real-time Transport Protocol (RTP) for data transmission.

[0066] For example, suppose a company is conducting a remote project discussion, and the speaker is speaking through a camera while showing a slide of the project progress on the screen. At this time, the video conference recording system needs to obtain two streams, one is the main stream of the speaker, and the other is the auxiliary stream of the screen sharing. These streams are transmitted to the video conference recording server through the network and prepared for subsequent recording and processing.

[0067] Step 102: Analyze the audio and video stream data to obtain first video stream data and second video stream data.

[0068] In an embodiment of the present invention, after obtaining the audio and video stream data, the next step is to analyze and distinguish different video stream data. Usually, there are multiple video streams in a video conference, such as the speaker's video stream, screen sharing stream, or presentation document stream. In this step, it is necessary to separate the mainstream video stream data and the auxiliary video stream data from the obtained audio and video stream data. These different video stream data contain different contents and need to be processed separately.

[0069] The first video stream data usually refers to the mainstream, that is, the camera picture of the speaker or host, which is the core content of the meeting and shows the speaker's facial expressions, gestures and other information. The second video stream data is generally the auxiliary stream, which contains screen sharing, presentation documents or other display content, such as slide files, web pages or video presentations.

[0070] For example, in an online training, the host explains the course while showing the handouts through screen sharing. After the video conference recording system obtains two audio and video streams, it needs to distinguish which one is the speaker's image and which one is the screen sharing content. At this time, the video conference recording system can make a judgment by analyzing the video frames. The speaker's image usually contains facial features captured by the camera, while the screen sharing image is mostly static or mainly text and graphics. Therefore, the video conference recording system can use image recognition technology or metadata of the video stream (such as video source information) to automatically distinguish between the main stream and the auxiliary stream.

[0071] Step 103: Record the first video stream data and the second video stream data respectively to obtain a first video file and a second video file, and add respective processing identification information.

[0072] In an embodiment of the present invention, after distinguishing the mainstream and auxiliary streams, the two video stream data are recorded separately. The recording process is to capture the video stream data and save it as a video file. Usually, the mainstream video file and the auxiliary stream video file are used to save the conference content respectively. In this way, the speaker's screen and the presentation content are stored separately, which is convenient for subsequent viewing and management.

[0073] For example, suppose a company is holding a remote work meeting, and the host is speaking while showing a sales report through a shared desktop. The video conference recording system distinguishes the host's camera image and the screen sharing image through the previous step, and starts recording the two video streams simultaneously. The host's image is saved as the first video file, and the screen sharing content is saved as the second video file.

[0074] To facilitate subsequent management and processing, the video conference recording system adds processing identification information to each video file. The processing identification information is used to identify and distinguish different video files. These processing identification information include but are not limited to: the source of the video file, the timestamp of the recording, the type of stream (main stream or auxiliary stream), and the frame number, etc. This information can synchronize the main stream and auxiliary stream during subsequent playback, and the video files can be classified and retrieved based on these processing identification information.

[0075] The functions of the above-mentioned processing identification information include but are not limited to:

[0076] Source information: The source from which the recorded video file is obtained (such as camera or screen sharing).

[0077] Timestamp: Records the start and end time of the video to ensure accurate alignment of the main and auxiliary streams during subsequent playback.

[0078] Frame Numbering: Ensures that each frame in a video file has a unique number to facilitate frame-level operations during playback and editing.

[0079] Step 104: store the first video file, the second video file and the corresponding processing identification information in a database.

[0080] In the embodiment of the present invention, after recording is completed, the video conference recording system stores the video files of the main stream and auxiliary stream and their corresponding processing identification information in the database. The purpose of this step is to ensure the long-term preservation of the conference recording content and to facilitate the retrieval of the corresponding files through the query function of the database.

[0081] After the meeting, the speaker's camera image is saved as a video file (such as MP4 format), and the shared screen image is saved as another video file. Both video files have their own processing identification information, and the video conference recording system will upload these video files and their processing identification information to the storage database. For example, an enterprise's internal database will save all meeting recordings, and the administrator can query by meeting time, host name, file type (main stream or auxiliary stream), etc.

[0082] The recording solution for dual-stream video conferencing provided by an embodiment of the present invention is applied to a first server. The audio and video stream data to be recorded is obtained, and then the first video stream data and the second video stream data are analyzed from the audio and video stream data. Next, the first video stream data and the second video stream data are recorded respectively to obtain a first video file and a second video file, and respective processing identification information is added. The processing identification information is used to identify and distinguish different video files. Finally, the first video file, the second video file and the corresponding processing identification information are stored in a database.

[0083] The embodiment of the present invention can simultaneously record two video streams, namely, the first video stream data (such as the mainstream) and the second video stream data (such as the auxiliary stream), to ensure that the speaker's screen and screen sharing content (such as a presentation document) are completely preserved. The problem that the prior art can only record a single video stream is solved, and the loss of important conference information is avoided, thereby improving the recording quality of the dual-stream video conference. By processing the first video stream data and the second video stream data separately and adding processing identification information, the synchronization of multiple video streams during the recording process is ensured. This enables the speaker's screen and the presentation content to be accurately aligned during playback, improving the playback experience and information consistency. The first video stream data and the second video stream data are recorded as independent video files respectively, and processing identification information is added to each video file to distinguish the type of video stream and its corresponding content, which simplifies the subsequent video file storage, retrieval and use, and improves management efficiency. By storing the recorded first video file and the second video file and their corresponding processing identification information (such as timestamp, video type, etc.) together in a database, the traceability of the dual-stream video conference recording is improved. Users can easily perform accurate retrieval based on information such as meeting time and video type, supporting efficient archiving and subsequent calls, and enhancing the flexibility and practicality of the storage system.

[0084] In an exemplary embodiment of the present invention, an implementation method of analyzing the first video stream data and the second video stream data from the audio and video stream data is to read each video frame data of the audio and video stream data; filter and classify each video frame data according to the metadata of each video frame data to obtain the first video frame data and the second video frame data; and reconstruct the first video frame data and the second video frame data into the first video stream data and the second video stream data, respectively.

[0085] In actual application, in video conferencing, video stream data is transmitted in the form of frames. Video frame is the smallest unit of video stream, and each frame represents a still picture of the video at a specific time point. Video stream is composed of a series of frames in time sequence to form a smooth dynamic picture. Therefore, when analyzing and processing audio and video streams, it is necessary to read each video frame data first.

[0086] The process of reading video frames involves extracting video frames from video stream data. Since video conferencing systems usually use real-time transport protocols to transmit data, video conferencing recording systems receive each frame in the audio and video stream at a certain time interval. When reading each frame of data, the metadata attached to it is also included, such as timestamp, frame number, video source information, etc.

[0087] After reading the video frame data, the next step is to distinguish the main video frame from the auxiliary video frame by analyzing the metadata. Metadata is a series of information attached to the video frame to describe the properties of the frame. Common metadata includes video source (such as camera or screen sharing), frame type (such as key frame or normal frame), timestamp (recording the specific time when the frame was captured), etc. Based on these metadata, video frames can be effectively screened and classified.

[0088] For example, in an online meeting, the speaker's camera image is captured as the mainstream video frame, and the screen sharing content is captured as the auxiliary video frame. In order to distinguish these two types of frames, you can filter them by the "Source" field in the metadata: the speaker's video frame usually comes from the camera, and the screen sharing video frame comes from the computer display content. Therefore, by reading the "Source" field, these frames can be classified into mainstream frames and auxiliary frames.

[0089] In addition, the sequence consistency of video frames can be ensured based on the timestamp of the frames. Timestamps are used to mark the time when the video frames were generated, ensuring that the frames can be arranged in the correct time order when they are played. During the classification process, timestamps can also help synchronize the content of the main stream and the auxiliary stream. For example, if the speaker is explaining the content on the screen, the timestamps of the main stream video frame and the auxiliary stream video frame can help ensure that the speech is synchronized with the screen content.

[0090] After the screening and classification are completed, the first video frame (main stream) and the second video frame (auxiliary stream) are reconstructed to form two independent video streams. The process of reconstructing the video stream is actually to rearrange the video frames in chronological order and package them into complete video stream data. Since the classification has been done based on timestamp and source information, this process ensures that the main stream and auxiliary stream are consistent in content and can be played independently.

[0091] The reconstruction process of the video stream usually combines video encoding technology, such as H.264 encoding technology, to compress the video frames into an efficient streaming format for subsequent storage and transmission. When constructing the first video stream (main stream), the video conference recording system will combine all frames from the speaker's camera in chronological order. The same process applies to the auxiliary stream (screen sharing content). During the playback of the video stream, the video conference recording system will also synchronize the main stream and auxiliary stream through timestamps to ensure that the speaker's speech is consistent with the shared screen content during the conference playback.

[0092] For example, in an online seminar, the host shared a video file and gave a lecture at the same time. The video conference recording system obtained the host's camera image (main stream) and the playback content of the video file (auxiliary stream). Through the reconstruction step, the two types of frames are packaged into two independent video streams, so that in the subsequent conference playback, the viewer can see the host's expression and movements, and can also watch the shared video content at the same time.

[0093] The embodiment of the present invention can accurately distinguish different video streams and reconstruct them into independent video files by analyzing metadata of video frames. This allows users to choose to watch the speaker or screen sharing content as needed during conference playback, and the two can maintain a high degree of synchronization. In addition, the classification and reconstruction steps improve the processing efficiency of video streams, avoid confusion between different streams, and ensure the accuracy and integrity of the recording.

[0094] In an exemplary embodiment of the present invention, before reconstructing the first video frame data and the second video frame data into the first video stream data and the second video stream data respectively, an implementation method is to synchronize the first video frame data and the second video frame data according to the synchronization information in the metadata; wherein the synchronization information includes a timestamp or a frame number.

[0095] In actual applications, in video conferences, the main stream and auxiliary stream often need to be presented at the same time. For example, when the speaker is explaining, the screen displays related presentations or data charts. If there is a time asynchrony between the two, even if the video and audio quality is high, the audience will be confused and even unable to understand the content. Therefore, it is very important to ensure the synchronization of the main stream and auxiliary stream videos.

[0096] Before performing synchronization processing, it is necessary to extract metadata of the first video frame data and the second video frame data, especially synchronization information. Metadata is information attached to the video frame, which provides relevant attributes of the frame. Among them, each frame of video has a timestamp to record the exact time when the frame was captured. By comparing the timestamps, it can be determined whether the content presented by the two video streams at the same time is consistent. The frame number gives each video frame a unique identifier, which can be used to mark the relative order of the frames. During playback, the frame number can ensure that the order of the video stream is not confused. By extracting and comparing these synchronization information, the time relationship between the two video streams can be analyzed, providing a basis for subsequent synchronization processing.

[0097] After obtaining the necessary synchronization information, the actual synchronization process will be carried out. This process can be divided into the following steps:

[0098] 1. Analyze timestamps or frame numbers: Extract the timestamps or frame numbers of the main stream and auxiliary stream respectively, and establish the time relationship between the two video streams. Assume that the timestamp of the main stream is [0,1,2,3], and the timestamp of the auxiliary stream is [0,1,1.5,2,3]. Through analysis, it is recognized that the auxiliary stream has delays at certain time points.

[0099] 2. Adjust the order of frames: If the timestamps of some frames are delayed, adjustments will be made to rearrange the frames of the auxiliary stream to ensure that the main stream and auxiliary stream can be presented at the same time during playback. For example, if the auxiliary stream fails to update in time at 1.5 seconds, you can choose to insert a static frame between 1 second and 2 seconds to maintain consistency.

[0100] 3. Synthesize synchronized video streams: After the synchronization process is completed, the adjusted frames will be reconstructed into independent video streams to ensure that the main stream and auxiliary streams can be displayed simultaneously during playback.

[0101] For example, suppose that at an online product launch conference, the host explains through the camera (main stream) and shows the function demonstration of the new product (auxiliary stream). In this process, there is a slight difference between the timestamp of the main stream video and the timestamp of the auxiliary stream video. For example, due to network delay, the auxiliary stream only shows an important picture at 3 seconds. By implementing the above synchronization processing, the timestamps of the two video streams will be compared, and the playback order of the auxiliary stream will be adjusted so that the auxiliary stream video appears synchronously at 3 seconds, ensuring that the audience will not miss any important information when watching.

[0102] The synchronous processing in the embodiment of the present invention ensures that the audience can clearly understand the relationship between the demonstration content and the explanation content, thereby enhancing the viewing experience. At the same time, the use of timestamps and frame numbers provides a convenient basis for subsequent possible editing and retrieval. The synchronous processing will effectively improve the efficiency and accuracy of information transmission, promote the audience's sense of participation and understanding, and thus achieve better communication effects.

[0103] In an exemplary embodiment of the present invention, when recording the first video stream data and the second video stream data respectively to obtain the first video file and the second video file, an implementation method is to detect the frame rate or packet loss rate of the first video frame data and the second video frame data; determine whether the first video frame data and / or the second video frame data are lost according to the frame rate or the packet loss rate; if the first video frame data or the second video frame data is lost, send video stream check notification information; if both the first video frame data and the second video frame data are lost, stop the recording process and send video stream check notification information.

[0104] In actual application, while recording the first video stream data and the second video stream data, the quality indicators of the two video streams, such as frame rate and packet loss rate, are monitored. By detecting these indicators, the integrity of the video data can be judged in time, and corresponding measures can be taken when loss occurs. Among them, frame rate (Frame Rate): refers to the number of frames displayed per second, usually in frames per second (Frames Per Second, referred to as FPS). A high frame rate can provide smoother video pictures, especially in fast-moving scenes. Generally speaking, 30FPS is considered to be the minimum requirement for video playback, and 60FPS can provide a smoother viewing experience. Packet loss rate (Packet Loss Rate): refers to the ratio of the number of data packets lost during video transmission to the total number of transmitted data packets. The packet loss rate is usually expressed as a percentage. A high packet loss rate can cause video freezes, unclear images, or complete loss of part of the content, affecting the audience's viewing experience.

[0105] When recording video streams, it is necessary to detect the frame rate and / or packet loss rate of the first video stream (main stream) and the second video stream (auxiliary stream) in real time. By using specialized monitoring algorithms and tools, the time and status of each new video frame can be recorded and the real-time frame rate can be calculated. Specifically, the received video frames are counted, and the number of frames received is counted within a fixed period of time (such as one second). Assuming that 45 frames are successfully received within 1 second, the frame rate of the video stream is 45FPS. When recording each video packet, the sent packets are counted at the same time. When it is detected that the expected packet is not received, it will be marked as lost and the packet loss rate will be calculated. For example, if 5 packets are lost out of 100 packets sent, the packet loss rate is 5%.

[0106] By monitoring these two indicators in real time, the quality of the video stream can be judged instantly, thus ensuring that subsequent recording work can proceed smoothly. Once an abnormal frame rate or packet loss rate is detected, a judgment will be made immediately. If it is found that the frame rate of the first video frame data or the second video frame data is significantly lower than expected (for example, lower than 30FPS) or the packet loss rate exceeds a certain threshold (such as 5%), it will be considered that the video stream has been lost.

[0107] For example, in an online training course, the speaker gave a speech about product introduction. During the recording process, the video conference recording system detected that the frame rate of the speaker's video stream (main stream) dropped to 15FPS, and the packet loss rate of the auxiliary stream (PPT presentation) reached 10%. This means that the audience will experience screen freezes and content loss when watching, affecting the overall viewing experience.

[0108] If the loss of the first video stream or the second video stream is found, a video stream check notification message will be sent. The notification can be an automatically generated message sent to the technical support team of the conference to prompt them to check the relevant video streams and network connections.

[0109] If both the first and second video streams are detected to be lost, the recording will be automatically stopped to protect the quality of the recorded content. This process helps to avoid recording incomplete or substandard video files and ensures that the final output file has a high quality standard.

[0110] By monitoring the frame rate and packet loss rate in real time, the embodiment of the present invention can quickly discover potential problems and take corresponding measures to ensure the integrity and fluency of the recorded content. In addition, timely inspection notification information can help the technical team respond quickly and solve potential problems to avoid affecting the user's viewing experience.

[0111] In an exemplary embodiment of the present invention, an implementation method for obtaining the audio and video stream data to be recorded is to obtain the audio and video stream data from a second server; wherein the audio and video stream data is obtained by alternating frame encoding of the first video stream data and the second video stream data.

[0112] Alternate frame coding refers to the alternating coding of multiple video streams (such as the main stream and auxiliary stream) to form a single audio and video stream. In this process, the data of the main stream and the auxiliary stream are arranged alternately so that the content of both can be retained during subsequent processing or recording. For example, assuming there is a main stream video frame A and an auxiliary stream video frame B, the result of alternating frame coding may be a sequence like this: A1 (main stream), B1 (auxiliary stream), A2 (main stream), B2 (auxiliary stream)... This method can not only reduce the demand for transmission bandwidth, but also facilitate subsequent decoding and processing. Through alternating frame coding, it can ensure that the key information of the main stream and auxiliary stream is captured in the same time period.

[0113] The second server is usually a dedicated server that is responsible for processing and storing video and audio data from multiple sources. In actual applications, the second server may receive audio and video streams from multiple terminals and integrate them into an audio and video stream that can be recorded.

[0114] For example, in a product launch conference, suppose that at the conference, the speaker gives a lecture through the camera (main stream) and shows the relevant slides of the product (auxiliary stream). In this process, the video stream may form the following sequence after alternating frame encoding:

[0115] Mainstream video frame 1 (e.g. the content presented by the speaker)

[0116] Content stream video frame 1 (e.g. a slideshow of a product introduction)

[0117] Mainstream video frame 2 (the speaker is describing the features of the product)

[0118] Auxiliary stream video frame 2 (showing the usage scenario of the product)

[0119] In the second server, these frame data are integrated into a stream and encoded in an alternating manner, so that the real-time interaction between the speaker and the slides can be preserved during recording.

[0120] The embodiment of the present invention effectively integrates the video streams of the main stream and the auxiliary stream through alternating frame coding, ensuring that the contents of both are captured and saved simultaneously during the recording process. This not only improves the efficiency of data transmission and reduces the bandwidth requirement, but also ensures the smoothness and stability of the video. By obtaining audio and video stream data from the second server, the audio and video content can be better managed and scheduled, providing users with a high-quality viewing experience.

[0121] In an exemplary embodiment of the present invention, when recording the first video stream data and the second video stream data to obtain the first video file and the second video file, an implementation method is to send recording status notification information to the second server so that the second server forwards the recording status notification information to the client, and the client is used to update the recording status of the video conference according to the recording status notification information. The recording status notification information refers to the message about the current recording status sent by the video conference recording system to the relevant server and client during the video conference recording process. Such information generally includes the start, pause, stop, error and other states of the recording.

[0122] As the hub of information transmission, the second server is responsible for receiving information from the first server and forwarding relevant information. After the client initiates recording or updates the recording status, the second server receives the recording status notification information from the first server. The notification content of the notification information may include the recording status, timestamp, and other related metadata. For example, when the user starts recording, the notification information sent may be "Recording started, timestamp: 2024-10-10 10:00:00". Next, after receiving the recording status notification information, the second server will parse and process it. The second server needs to determine the type of information (such as start, pause, or stop) and prepare to forward it to the relevant client. This process may use a message queue or event system to ensure accurate transmission of information. For example, during a meeting, if the second server receives a "recording pause" notification, it will immediately send this status information to the management client of the video conference. After receiving the recording status notification information from the second server, the client needs to update the recording status of the video conference. This process includes:

[0123] After receiving the forwarded recording status notification information, the client will parse it and identify the current recording status and its changes. The client will update the user interface according to the new recording status, such as changing the color of the recording indicator or updating the displayed status information. This update not only improves the user experience, but also reminds participants of the progress of the current recording. Through the client's updates, participants can understand the recording status in real time and decide whether to adjust their speeches or questions. This real-time feedback mechanism helps to improve the interactivity and participation of the meeting.

[0124] By sending recording status notification information to the second server, the embodiment of the present invention can effectively manage the recording process and reduce recording problems caused by delayed status updates. In addition, timely status feedback can not only improve the sense of participation of participants, but also enhance the monitoring ability of the technical support team on the recording process. If an abnormality is found in the recording status, the technical support team can quickly intervene to solve potential problems, thereby ensuring the smooth progress of the video conference.

[0125] In an exemplary embodiment of the present invention, the processing identification information includes at least one of the following: video stream identification information, video start and end time information, and video thumbnail information. Among them, the video stream identification information is used to clarify the source and content of each video stream. Since there are usually multiple video streams (such as mainstream and auxiliary streams) in a video conference, these video streams represent different conference contents. Through the video stream identification information, a unique identifier can be assigned to each video stream, so that it can be easily distinguished and identified during subsequent processing or playback. The video start and end time information is used to record the start and end time of each video. Video conferencing is usually a continuous process, and the recording process may include multiple stages or segments. In order to accurately locate the content of a specific time period, the start and end time information is particularly important. It can not only help users find specific conference content later, but also provide accurate jump points during playback. Video thumbnail information refers to a static picture extracted from the video stream, which is used as a representative preview of the video. Thumbnails can help users intuitively understand the video content without playing it first.

[0126] The embodiment of the present invention can more effectively manage and process recorded audio and video files by adding processing identification information to the video stream. The integration of processing identification information makes the recording and management process of video conferences more efficient and intelligent. It not only simplifies the retrieval and playback operations of the video, but also enhances the system's ability to organize the recorded content, improves the visualization of the video content, and ultimately brings a better experience to the user.

[0127] Based on the above description of the embodiment of a dual-stream video conference recording method, a dual-stream video conference recording solution based on a visual network is introduced below. The execution process of the dual-stream video conference recording solution based on a visual network is roughly as follows:

[0128] After the video conference starts, set the video conference recording service system to be used. Select the corresponding recording service system by configuring the recording service number.

[0129] Select the main video terminal, auxiliary video terminal and 4-channel audio terminals to be recorded. The 4-channel audio terminals are the host terminal, speaker 1 terminal, speaker 2 terminal and speaker 3 terminal by default. You can also manually change the terminals of other participants.

[0130] When audio and video recording starts, the video conference management client sends two video streams (main stream and auxiliary stream) and four audio streams to the scheduling service system through one conference recording service number. The scheduling service system sends the video and audio numbers to be recorded to the recording service system. The scheduling service system alternately encodes the main stream and auxiliary stream video streams (one frame for the main stream and one frame for the auxiliary stream), synthesizes one video stream and pushes it to the recording service system.

[0131] During recording, the recording and broadcasting service system detects the status of the main and auxiliary video streams. If one of the two streams has no stream, it will prompt "Please check if there is no stream in the main or auxiliary stream" and the other stream will be recorded normally. When it is detected that both the main and auxiliary streams have no stream, the recording and broadcasting service system will stop recording and return an error prompt to the client.

[0132] When the recording and broadcasting service system detects that the video streams are normal, it sends a recording instruction to the scheduling service system, and the scheduling service system notifies the client to update the current recording status.

[0133] When the client sends a stop recording command to the scheduling service system, the scheduling service system notifies the recording service system to stop recording. The recording service system stops recording the current audio and video, saves the currently recorded audio and video files, and forms a recording file with the conference name plus time and saves it.

[0134] Reference Figure 2 , showing a schematic diagram of a recording interface of a dual-stream recording solution for video conferencing based on a visual network according to an embodiment of the present invention. The recording interface includes a video area and an audio area. Among them, the video area may include a mainstream display area and an auxiliary stream display area. The mainstream display area may display video stream 1 (name to be determined) and the terminal name. The auxiliary stream display area may display video stream 2 (name to be determined) and the auxiliary stream of the terminal name.

[0135] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0136] Reference Figure 3 , shows a structural block diagram of a dual-stream video conference recording system according to an embodiment of the present invention. The dual-stream video conference recording system can be applied to a first server, and the dual-stream video conference recording system can specifically include the following modules.

[0137] The audio and video stream data acquisition module 31 is used to acquire the audio and video stream data to be recorded;

[0138] A video stream data analysis module 32, configured to analyze the audio and video stream data to obtain first video stream data and second video stream data;

[0139] A video stream data recording module 33, used to record the first video stream data and the second video stream data to obtain a first video file and a second video file, and add respective processing identification information, wherein the processing identification information is used to identify and distinguish different video files;

[0140] The video file storage module 34 is used to store the first video file, the second video file and the corresponding processing identification information in a database.

[0141] In an exemplary embodiment of the present invention, the video stream data analysis module 32 includes:

[0142] A video frame data reading module, used to read each video frame data of the audio and video stream data;

[0143] A video frame data screening and classification module, used for screening and classifying each of the video frame data according to the metadata of each of the video frame data to obtain first video frame data and second video frame data;

[0144] The video stream data reconstruction module is used to reconstruct the first video frame data and the second video frame data into the first video stream data and the second video stream data respectively.

[0145] In an exemplary embodiment of the present invention, the system further comprises:

[0146] a video frame data synchronization processing module, configured to perform synchronization processing on the first video frame data and the second video frame data according to the synchronization information in the metadata before the video stream data reconstruction module reconstructs the first video frame data and the second video frame data into the first video stream data and the second video stream data respectively;

[0147] The synchronization information includes a timestamp or a frame number.

[0148] In an exemplary embodiment of the present invention, the system further comprises:

[0149] a video frame data detection module, configured to detect a frame rate or a packet loss rate of the first video frame data and the second video frame data when the video stream data recording module 33 respectively records the first video stream data and the second video stream data to obtain a first video file and a second video file;

[0150] a loss situation determination module, used to determine whether the first video frame data and / or the second video frame data is lost according to the frame rate or the packet loss rate;

[0151] A check notification sending module, configured to send video stream check notification information if the first video frame data or the second video frame data is lost;

[0152] The recording stop module is used to stop the recording process and send the video stream check notification information if both the first video frame data and the second video frame data are lost.

[0153] In an exemplary embodiment of the present invention, the audio and video stream data acquisition module 31 is used to acquire the audio and video stream data from the second server;

[0154] The audio and video stream data is obtained by performing alternating frame encoding on the first video stream data and the second video stream data.

[0155] In an exemplary embodiment of the present invention, the system further comprises:

[0156] A recording status update module is used to send recording status notification information to the second server when the video stream data recording module 33 records the first video stream data and the second video stream data respectively to obtain the first video file and the second video file, so that the second server forwards the recording status notification information to the client, and the client is used to update the recording status of the dual-stream video conference according to the recording status notification information.

[0157] In an exemplary embodiment of the present invention, the processing identification information includes at least one of the following:

[0158] Video stream identification information, video start and end time information, and video thumbnail information.

[0159] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0161] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0162] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0166] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0167] The above is a detailed introduction to a dual-stream video conference recording method and a dual-stream video conference recording system provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A recording method for a dual-stream video conference, characterized in that: Applied to a first server, the method comprises: Get the audio and video stream data to be recorded; Analyzing the first video stream data and the second video stream data from the audio and video stream data; Recording the first video stream data and the second video stream data respectively to obtain a first video file and a second video file, and adding respective processing identification information, wherein the processing identification information is used to identify and distinguish different video files; The first video file, the second video file and the corresponding processing identification information are stored in a database.

2. The method according to claim 1, characterized in that The step of analyzing the first video stream data and the second video stream data from the audio and video stream data comprises: Read each video frame data of the audio and video stream data; Screening and classifying each of the video frame data according to the metadata of each of the video frame data to obtain first video frame data and second video frame data; The first video frame data and the second video frame data are reconstructed into the first video stream data and the second video stream data respectively.

3. The method according to claim 2, characterized in that Before respectively reconstructing the first video frame data and the second video frame data into the first video stream data and the second video stream data, the method further includes: Synchronize the first video frame data and the second video frame data according to the synchronization information in the metadata; The synchronization information includes a timestamp or a frame number.

4. The method according to claim 2, characterized in that: When the first video stream data and the second video stream data are recorded respectively to obtain the first video file and the second video file, the method further includes: Detecting a frame rate or a packet loss rate of the first video frame data and the second video frame data; Determine whether the first video frame data and / or the second video frame data is lost according to the frame rate or the packet loss rate; If the first video frame data or the second video frame data is lost, sending video stream check notification information; If both the first video frame data and the second video frame data are lost, the recording process is stopped and the video stream check notification information is sent.

5. The method according to claim 1, characterized in that The step of obtaining the audio and video stream data to be recorded includes: Acquire the audio and video stream data from the second server; The audio and video stream data is obtained by performing alternating frame encoding on the first video stream data and the second video stream data.

6. The method according to claim 5, characterized in that When the first video stream data and the second video stream data are recorded respectively to obtain the first video file and the second video file, the method further includes: Sending recording status notification information to the second server so that the second server forwards the recording status notification information to the client, and the client is used to update the recording status of the dual-stream video conference according to the recording status notification information.

7. The method according to claim 1, characterized in that The processing identification information includes at least one of the following: Video stream identification information, video start and end time information, and video thumbnail information.

8. A dual-stream video conference recording system, characterized in that: Applied to a first server, the system comprises: An audio and video stream data acquisition module, used to acquire the audio and video stream data to be recorded; A video stream data analysis module, used for analyzing the audio and video stream data to obtain first video stream data and second video stream data; a video stream data recording module, used to record the first video stream data and the second video stream data to obtain a first video file and a second video file, respectively, and add respective processing identification information, wherein the processing identification information is used to identify and distinguish different video files; The video file storage module is used to store the first video file, the second video file and the corresponding processing identification information in a database.

9. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to execute the dual-stream video conference recording method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the dual-stream video conference recording method as described in any one of claims 1 to 7.

Images