Video synthesis method and device, electronic equipment, storage medium and program

Abstract

The embodiment of the invention provides a video synthesis method and device, electronic equipment, a storage medium and a program, and the method comprises the steps: determining the number of video code streams of to-be-processed video data, and determining the video frame rate and playing duration of different video code streams in the to-be-processed video data according to the number of the video code streams; obtaining each video code stream corresponding to the to-be-processed video data, and generating decoding data of each video code stream; and performing picture synthesis on the decoded data based on the video frame rate and the playing duration. According to the embodiment of the invention, the video streams used for picture synthesis can be dynamically adjusted according to the number of the video streams, the problem of picture abnormity caused by picture synthesis can be solved, the picture synthesis effect can be improved, and the user experience is enhanced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more particularly to a video synthesis method, apparatus, electronic device, storage medium, and program. Background Technology

[0002] With the rapid development of internet technology, people's communication methods have also undergone significant changes. Video conferencing systems have also developed rapidly. As the user base grows, the demands for video conferencing systems are becoming increasingly diverse. Users need to be able to save meeting content for post-meeting summaries, reviews, and record organization. Our company's recording server is designed for recording meetings. Because the operations during a meeting are constantly changing, the recorded file may contain multiple audio and video streams. Some video streams may have resolution variations, making the recorded file unplayable with common media players and unavailable for playback on the recording server. Therefore, after recording, the file needs to be transcoded into a universal video file to ensure playback on various devices and platforms. Currently, due to the large number of video streams in the recorded file, to ensure a uniform transcoding format, the recording server mixes the audio and composites the video, ensuring that each video file contains only one video and one audio stream. However, since the number of video streams changes constantly, if video frame composition is performed according to a fixed number of video streams, a series of unexpected problems will occur in the composite image. For example, if a video stream has no data at the beginning but is added in the middle of the meeting, the composite image will appear black. Or, if there is no data in the middle or at the end of the meeting, it will cause the image to freeze for a long time. Therefore, how to perform flexible and effective frame composition of meeting videos to ensure normal playback is an urgent problem to be solved. Summary of the Invention

[0003] This invention provides a video compositing method, apparatus, electronic device, storage medium, and program that can dynamically adjust the video stream used for image compositing based on the number of video streams, thereby solving the problem of image anomalies caused by image compositing, improving the image compositing effect, and enhancing the user experience.

[0004] According to one aspect of the present invention, a video synthesis method is provided, wherein the method includes:

[0005] Determine the number of video bitstreams in the video data to be processed, and determine the video frame rate and playback duration of different video bitstreams within the video data to be processed based on the number of video bitstreams;

[0006] Obtain each video bitstream corresponding to the video data to be processed, and generate decoding data for each video bitstream;

[0007] The decoded data are composited based on the video frame rate and the playback duration.

[0008] According to another aspect of the present invention, a video synthesis apparatus is provided, wherein the apparatus comprises:

[0009] The information pre-reading module is used to determine the number of video bitstreams in the video data to be processed, and to determine the video frame rate and playback duration of different video bitstreams in the video data to be processed based on the number of video bitstreams.

[0010] The bitstream decoding module is used to acquire each of the video bitstreams corresponding to the video data to be processed, and to generate decoding data for each of the video bitstreams;

[0011] The image compositing module is used to compose images of each of the decoded data based on the video frame rate and the playback duration.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video synthesis method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video synthesis method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the video synthesis method described in any embodiment of the present invention.

[0018] The technical solution of this invention identifies the number of video streams in the video data to be processed, identifies the video frame rate and playback duration of different video streams according to the number of video streams, obtains and decodes each video stream to obtain decoded data of each video stream, and performs image composition on each decoded data according to the video frame rate and playback duration. It can dynamically adjust the video stream used for image composition according to the number of video streams, solve the problem of image anomalies caused by image composition, improve the image composition effect, and enhance the user experience.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a video synthesis method provided according to Embodiment 1 of the present invention;

[0022] Figure 2 This is a flowchart of a video synthesis method provided according to Embodiment 2 of the present invention;

[0023] Figure 3 This is a flowchart of a video synthesis method provided in Embodiment 3 of the present invention;

[0024] Figure 4 This is a schematic diagram of a video file information reservation process according to Embodiment 4 of the present invention;

[0025] Figure 5 This is an example diagram of a video synthesis method provided in Embodiment 4 of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of a video synthesis device according to Embodiment 5 of the present invention;

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the video synthesis method of this invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a video compositing method according to Embodiment 1 of the present invention. This embodiment is applicable to situations where the number of video bitstreams changes dynamically during the image compositing process. The method can be executed by a video compositing device, which can be implemented in hardware and / or software, and can be configured in a server or server cluster. Figure 1 As shown, the method includes:

[0032] Step 110: Determine the number of video bitstreams in the video data to be processed, and determine the video frame rate and playback duration of different video bitstreams within the video data to be processed based on the number of video bitstreams.

[0033] The video data to be processed can be video data to be composited, and can include video with dynamically changing bitrates, such as web conference videos and live streams. The number of video streams can be the number of independent video stream tracks contained within a video file, or the total number of video streams within the video data to be processed. The video frame rate can be the number of image frames transmitted per unit time in the video stream. The video frame rate can be the overall bitrate of the video stream during the entire transmission process, or it can be a partial bitrate within different time segments. For example, in a conference video, user A joins the conference at 15-30 minutes and 40-50 minutes respectively. The video frame rate can be calculated for the entire bitrate transmission process for user A, or it can be calculated for different time segments of the video stream. The playback duration can be the time from the start of video stream transmission or playback within the processed video data. The playback duration can be expressed as one or more time segments, with different time segments indicating the duration of video stream transmission or playback.

[0034] In this embodiment of the invention, video data to be processed can be obtained. The video data to be processed can be real-time generated video data or pre-saved video data. The number of video bitstreams in the video data to be processed can be read, and the video frame rate and playback duration of each video bitstream in the video data to be processed can be counted according to the number of video bitstreams.

[0035] Step 120: Obtain the video bitstreams corresponding to the video data to be processed, and generate the decoding data for each video bitstream.

[0036] In this embodiment of the invention, when different video bitstreams are received within the video data to be processed, the received video bitstreams can be decoded separately to obtain decoded data. It is understood that different video bitstreams can be implemented based on different decoders.

[0037] Step 130: Perform image synthesis on each decoded data based on the video frame rate and playback duration.

[0038] Specifically, after obtaining the decoded data, the playback duration and video frame rate corresponding to each decoded data can be determined. The decoded data can be composited according to their respective playback durations. For example, if there are two decoded data in the range of 1 to 10 minutes, these two decoded data can be used to composite the video. During the composite process, the video frame rates of the two decoded data can be used to generate the composite video. For example, the frame rate of the composite video can be determined based on the maximum value of the two video frame rates.

[0039] In this embodiment of the invention, the number of video streams to be processed is identified, and the video frame rate and playback duration of different video streams are identified according to the number of video streams. Each video stream is obtained and decoded to obtain the decoded data of each video stream. The decoded data is then composited according to the video frame rate and playback duration. The video stream used for composited images can be dynamically adjusted according to the number of video streams, which can solve the problem of image anomalies caused by composited images, improve the composited image effect, and enhance the user experience.

[0040] Example 2

[0041] Figure 2 This is a flowchart of a video synthesis method according to Embodiment 2 of the present invention. Based on the above embodiments, this embodiment describes the statistical process of video frame rate and playback duration. This method can be executed by a video synthesis device, which can be implemented in hardware and / or software, and can be configured in a server or server cluster. Figure 2 As shown, the method includes:

[0042] Step 210: Read the header file of the video data to be processed and extract the number of video bitstreams carried in the header file.

[0043] The header file can be the beginning of a video file and can store the core metadata of the video file. The header file can carry information such as format identifiers, number of video streams, and stream sequence number.

[0044] In this embodiment of the invention, the header file of the video data to be processed can be read, and the number of video streams carried in the header file can be extracted. The number of video streams can indicate the number of video streams contained in the video data to be processed.

[0045] Step 220: Read the video bitstream of the video data to be processed, and determine the video frame rate and playback duration based on the timestamp information and the number of video frames in each video bitstream.

[0046] The timestamp information can indicate the absolute position or relative order of video frames on the video timeline of the video data to be processed, and the timestamp information can mark the time order in which video frames are processed by the decoder.

[0047] In this embodiment of the invention, each video stream of the video data to be processed can be received, the timestamp information and the number of video frames in each video stream can be counted, one or more time periods composed of the timestamp information of each video stream can be used as the playback duration, and the ratio of the number of video frames in each video stream to the time length corresponding to the playback duration can be used as the video frame rate of the video stream.

[0048] Step 230: Create a decoder for each video bitstream in the video data to be processed.

[0049] Specifically, there can be multiple video streams of video data to be processed, and a separate decoder can be created for each video stream. For example, the decoder can be instantiated separately according to the bitstream sequence number of each video stream.

[0050] Step 240: Input each received video stream into its corresponding decoder for decoding to obtain the video stream decoding data and stream information. The stream information includes at least one of the following: size information, current timestamp, and video frame rate. Save the decoding data and stream information of each video stream.

[0051] In this embodiment of the invention, the identification information of each video stream can be parsed to obtain the stream sequence number or encoding format identifier, etc. Based on the preset correspondence between video streams and decoders, the data packets of each video stream are distributed to the corresponding decoder according to the stream sequence number. The decoder performs decoding operations on the received stream data packets. During the decoding process, the original pixel data of the video frame can be restored based on the encoding rules of the stream to obtain the decoded data. During the decoding process, the stream information of the stream data can also be extracted, such as size information, current timestamp, and video frame rate. The size information can be the width and height of the decoded video frame, reflecting the spatial dimension attribute of the video image. The current timestamp indicates the timestamp information of the current decoded frame in the video stream, representing the position of the frame on the video timeline. The video frame rate can be the frame rate of the current decoded frame in the video stream. The obtained decoded data and stream information of the video stream can be saved for subsequent image compositing. The decoded data of each video stream can be saved to a queue.

[0052] Step 250: Perform image synthesis on each decoded data based on the video frame rate and playback duration.

[0053] This invention, in its embodiments, reads the header file of the video data to be processed and parses the header file to obtain the number of video streams. It then reads the video streams of the video data to be processed and counts the timestamp information and the number of video frames within each video stream. Based on the timestamp information and the number of video frames, it determines the video frame rate and playback duration of each video stream. A decoder is created for each video stream, and the received video streams are input into the decoders for decoding, thereby obtaining decoded data and stream information. The decoded data and stream information are saved, and the decoded data is composited according to the video frame rate and playback duration. This invention can count the playback duration of different video streams using their video timestamps, accurately determine the video stream used for image compositing based on their respective playback durations, solve the image anomaly problem caused by image compositing, improve the image compositing effect, and enhance the user experience.

[0054] Furthermore, based on the above embodiments of the invention, determining the video frame rate and playback duration based on the timestamp information and the number of video frames within each video stream includes:

[0055] For each video stream, time segment information is generated, and the timestamp of the first video frame within the category is set as the start and end times of the time segment information. The number of segmented video frames for the time segment information is initialized to 1. For each video stream, it is determined whether the time difference between the timestamp of each video frame and the end time of the time segment information is less than a threshold time. If so, the end time is updated to the timestamp of the video frame, and the number of segmented video frames is incremented by 1. If not, and the video frame is a keyframe, new time segment information is generated for the video stream, the timestamp of the current video frame is set as the start and end times of the new time segment information, and the number of segmented video frames for the new time segment information is set to 1. The time segment information of each video stream is saved as the playback duration, and the video frame rate is calculated based on the start and end times and the number of segmented video frames for each time segment information.

[0056] Among them, the time segment information can be the time period during which each video stream is continuously played. The playback duration of the video stream can be composed of multiple time segment information. The time interval between different time segment information can be greater than a threshold time. This threshold time can be used to measure the time interval during which the video stream is paused. The number of segmented video frames can be the number of video frames within each time segment information.

[0057] In this embodiment of the invention, a first time segment information can be generated for each video stream, and the timestamp information of the first video frame in the video stream can be used as the start and end times of the time segment information. The number of video frames in the time segment information is set to 1. The timestamp information of each subsequent video frame can be compared with the end time of the current time segment information. If the time difference between the timestamp information and the end time is less than a threshold time, it indicates that there is no pause between the current video frame and the previous video frame. In this case, the timestamp information of the current video frame can be used as the end time of the current time segment information, and the number of video frames in the current time segment information can be incremented by 1. If the time difference between the current timestamp information and the end time is greater than or equal to the threshold, it indicates that there is a pause between the current video frame and the previous video frame. If the current video frame is a keyframe, a new time segment information can be created for the video stream, and the timestamp information of the current video frame can be set as the start and end times of the newly created time segment information. The number of video frames in the newly created time segment information is also set to 1. For each video stream, one or more time segments of its continuous playback time can be statistically analyzed using the methods described above. These time segments can include the start time, end time, and the number of video frames in each segment. For each video stream, one or more corresponding time segments can be obtained. This time segment information can be used as the playback duration of the video stream. The video frame rate can be statistically analyzed based on the start time, end time, and number of video frames in each time segment. For example, the duration of each video stream's playback can be determined based on the start and end times of each time segment, and the ratio of each time segment to the duration can be used as the video bitrate.

[0058] In some embodiments of the invention, the video frame rate is calculated based on the start time, end time, and number of video frames in each time segment, including:

[0059] For each time segment, the time length between the end time and the start time is determined, and the ratio of the number of segmented video frames to the time length is used as the segmented video frame rate of the time segment. For each video stream, the segmented video frame rate of each time segment is obtained as the video frame rate.

[0060] Specifically, for each time segment, the length of time between the end time and the start time can be calculated. The ratio of the number of video frames in that time segment to the length of time can be used as the segment video frame rate for that time segment. The segment video frame rate of each time segment in each video stream can be used as the video frame rate of that video stream.

[0061] Example 3

[0062] Figure 3 This is a flowchart of a video compositing method according to Embodiment 3 of the present invention. This embodiment of the present invention describes the process of compositing video data. (See also...) Figure 3 The video synthesis method provided in this embodiment of the invention includes the following steps:

[0063] Step 310: Determine the number of video bitstreams in the video data to be processed, and determine the video frame rate and playback duration of different video bitstreams in the video data to be processed based on the number of video bitstreams.

[0064] Step 320: Obtain the video bitstreams corresponding to the video data to be processed, and generate the decoding data for each video bitstream.

[0065] Step 330: Read the video stream with the highest video bitrate in sequence according to each playback duration as the target video stream.

[0066] In this embodiment of the invention, video streams can be obtained sequentially according to the playback duration. If the current time is within the playback duration of a video stream, then the video stream is marked as the target video stream. If the current time is within the playback duration of multiple video streams, then the video stream with the highest video bitrate among the multiple video streams at that time is obtained, and that video stream can be marked as the target video stream.

[0067] Step 340: When the target video stream is decoded, generate the composite image according to the target video stream.

[0068] Specifically, it can be determined whether the target video stream has been decoded, that is, whether the decoded data of the target video stream has been obtained. If so, the images can be composited according to the target video stream. For example, the frame rate of the target video stream can be obtained, and the images of the video data to be composited can be composited according to the frame rate of the target video stream.

[0069] Step 350: If the difference between the timestamp information of the current frame of the target decoded data and the current frame of other decoded data is greater than the first time threshold, then the target decoded data will not be composited and the target decoded data will be cleared.

[0070] Specifically, during the image compositing process, the current frame of the decoded data can be compared to determine the time difference between the timestamps of each current frame. If the time difference between the timestamp of a current frame and the timestamps of other current frames is greater than a first time threshold, it can be determined that the video stream corresponding to the current frame is in a paused state. The target decoded data corresponding to the current frame can then be composited and the target decoded data can be cleared. Clearing the target decoded data can include initializing the decoder corresponding to the target decoded data.

[0071] This invention, in its embodiments, obtains the number of video streams in the video data to be processed, determines the video frame rate and playback duration of different video streams within the video data to be processed according to the data stream data, receives the video streams within the video data to be processed, and decodes the video streams into decoded data. For each decoded data, the video stream with the highest video bitrate is sequentially determined as the target video stream for the corresponding playback duration. A composite image is generated based on the decoded target video stream. If the difference in timestamp information between the current frame in the target decoded data of the target video stream and the current frame of other decoded data is greater than a first time threshold, then the target decoded data is used for image compositing, and the target decoded data is cleared. This invention can statistically determine the playback duration of video streams through the video timestamps of different video streams, accurately determine the video stream used for image compositing based on their respective playback durations, solve the image anomaly problem caused by image compositing, improve the image compositing effect, and enhance the user experience.

[0072] Furthermore, based on the above embodiments of the invention, when the target video stream is decoded, generating a composite image according to the target video stream includes:

[0073] Obtain the target video frame rate and target playback duration of the target video bitstream; determine the number of decoded streams that have been acquired at the current moment, obtain the picture style corresponding to the number of decoded streams, and set the duration of the picture style as the playback duration; perform picture composition on each acquired decoded data according to the target video frame rate, target playback duration, and picture style.

[0074] In this embodiment of the invention, the target video frame rate and target playback duration can be obtained for the target video bitstream. The number of bitstreams with currently acquired decoded data can be obtained as the number of decoded streams. The picture style corresponding to the number of decoded streams can be obtained. The picture style may include, but is not limited to, picture layout, picture color, and picture resolution. The picture style corresponding to the number of decoded streams can be used as the picture style of the image generated after picture synthesis. The duration of the picture style is set as the target playback duration. The currently acquired decoded data can be synthesized according to the picture style, and the video frame rate of the synthesized picture can be set as the target video frame rate.

[0075] Example 4

[0076] To address the issues of black screens and prolonged image stuttering after video compositing during file transcoding, this invention provides a video compositing method. The method, as described in this embodiment, pre-reads the recorded ASF file, recording the start and end timestamps of each video stream. During transcoding, based on the timestamp information of the decoded video and the video streams, a compositing mode is selected, ensuring that the transcoded MP4 file has neither prolonged image stuttering nor black screens, thus improving the viewing experience. Simultaneously, it ensures that other functional modules of the system are largely unaffected, improving the system's usability. This video compositing method may include the following steps:

[0077] S1. Pre-read the video bitrate in the recording file.

[0078] S2. Pre-read the bitstream in the video file, read the video data, record the number of video frames, start and end timestamp information, and calculate the frame rate of each video bitstream segment.

[0079] S3. Read the video file, read a new video stream, and create a decoder.

[0080] S4. Read video frame data and decode the video frames using the corresponding decoder. The decoded data for each video stream is YUV, the timestamp of the current frame, and the frame rate information.

[0081] In this embodiment of the invention, after pre-reading the video information of the recording file, the reading of the recording file can be restarted. For each new video stream read, a decoder is created. For each read video frame data, the corresponding decoder is called to decode the video frame. The decoded data for each video stream includes YUV, width and height, the timestamp of the current frame, and frame rate information. This decoded data, width and height, timestamp, and frame rate information are stored in a queue. Newly decoded YUV data is placed at the tail of the queue. Simultaneously, if the queue length is greater than or equal to 2, redundant data is deleted from the head.

[0082] S5. Compare the frame rates of the video streams, select the video stream with the highest frame rate at the same time point, and perform a frame synthesis once the video stream is decoded.

[0083] Specifically, the process involves reading the frame rate data of the current video stream, selecting the stream with the highest frame rate, decoding that stream, and then performing a frame composite. This embodiment of the invention allows for the selection of the stream with the highest frame rate for frame composite, ensuring that the higher frame rate stream does not experience frame loss during the composite process. For example, if stream A has 60 frames per second and stream B has 20 frames per second, if the frame rate is less than 60, stream A will lose video data. If it is 60, stream A is guaranteed to be unaffected, and stream B may experience some repetition, which will not impact the user's viewing experience.

[0084] S6. Read the YUV data from the decoder and the timestamp information of the current frame. Based on the number of YUV data entries read, as well as the start and end timestamps of the current video stream, select the image compositing method and perform image compositing.

[0085] In this embodiment of the invention, YUV data from all decoders can be read. The initial width and height of the YUV data for each decoder are set to 16. Width and height information related to the decoded YUV is obtained. If the obtained width and height information differs from the initial value, a new video stream has been added, requiring modification of the image composition style to incorporate the new stream into the composed image. Here, adding a new video stream corresponds to adding a sub-image; for example, if the original style was a 2-image style, adding a video stream changes it to a 3-image style. The timestamp information of the decoder's YUV and the current frame is read. The timestamps of other streams are compared with the current decoded stream data. If the timestamp difference is greater than 60 seconds, the data of that stream is considered paused and should be removed from the image composition. The decoder's YUV data is cleared, and the initial width and height of the YUV are set to 16. This way, when the stream has data again, it can be added to the image composition in real time.

[0086] S7. After the images are composited, the composite images are encoded in h264 and written to an mp4 file.

[0087] Among them, see Figure 4 The process of pre-reading the recording file can be implemented as follows: Obtain the ASF header object of the recording file to get the number of video streams and the stream sequence number i. Read the bitstream in the recording file, determine if it is video data. If it is a video frame, record the timestamp of the first frame as the start timestamp of the frame, t(i)(i+0)(start), according to the stream sequence number i, and set t(i)(i+0)(stop) = t(i)(i+0)(start). Read the video frame stream sequence number i. If it is not the first frame, record the timestamp of the current frame in t(i)(i+0)(stop). Read the video frame stream sequence number i, calculate the difference g between the current frame and t(i)(i+0)(stop). If the difference g is greater than 60000, ... If the current frame is frame I, then a new start timestamp t(i)(i+1)(start) is recorded, and a new timestamp t(i)(i+1)(stop) for the current frame is also recorded. All start timestamps t(i)(i+j)(start) and all end timestamps t(i)(i+j)(stop) for each video stream can be obtained in the above way. Here, j represents the j-th timestamp record of video frame stream sequence number i. Each timestamp record can record a period of continuous playback of the stream, and the time interval between different timestamp records is greater than 60000.

[0088] In one exemplary implementation, after the meeting ends, a video recording is generated. This recording cannot detect the start or stop of the video stream. If transcoding is performed directly without detecting the start or stop of a stream, then after that stream stops, the composite image of that stream will only be stuck on the last frame, or other data will be used to fill the gap. Such image quality is unsatisfactory. Therefore, transcoding needs to detect the start and stop of the video stream using the method provided in this embodiment. Here, a threshold needs to be set. For example, if there is no video within one minute, the stream can be considered stopped before being restarted during transcoding. Since the start of a video stream always begins with a keyframe, the video frame type can be determined. If it is a keyframe, the difference between the timestamp of this frame and the previous timestamp on this stream is compared. If the threshold time is reached, the start and stop time of this stream can be recorded during pre-reading of the recording file, and the frame rate information can be calculated. Figure 5 This document describes in detail the scenario where a video file contains both a main stream and a secondary stream. A new video stream is read, a decoder is created, and the process checks if the video stream reading has ended. If not, the decoded data is placed in a queue. It then checks if the stream has the highest frame rate; if so, the queue data is read, the images are composited, encoded, and written to an MP4 file. The timestamp information in the queue is iterated to see if it matches the pre-read stop timestamp; if so, the decoded YUV data in the queue is cleared. If the video stream reading has ended, the entire image compositing process terminates. Since the operation of participants in a video conferencing system can be complex, some typical scenarios are explained in detail. The specific bitstream details during the meeting are as follows:

[0089] a. Initially, only mainstream videos are available; after a period of time, auxiliary streams need to be enabled.

[0090] b. The auxiliary flow continued for a period of time; the auxiliary flow was then shut down.

[0091] c. Restart the auxiliary stream, continue for a period of time, then turn it off.

[0092] d. The meeting continues, and at the end of the meeting, the main stream is shut down.

[0093] At the start of the meeting, only the mainstream videos were available, so the mainstream videos were decoded and then composited into single frames.

[0094] When the auxiliary stream is enabled, the decoded YUV data is placed in two queues. If the auxiliary stream frame rate is higher than the mainstream frame rate, the image composition style is switched and two images are composed each time the auxiliary stream information is read.

[0095] After the auxiliary stream is closed, the auxiliary stream data is cleared. At this time, only one queue in the YUV queue has data. Switch the screen composition style and perform single-screen composition on the main stream.

[0096] When the secondary stream is restarted, if the main frame rate is higher than the secondary stream, then every time the main information is read, the image compositing style is switched and the two images are composited.

[0097] After the auxiliary stream stops completely, clear the auxiliary stream data. At this point, only one queue in the YUV queue has data. Switch the screen composition style and perform single-screen composition on the main stream.

[0098] The above explains the two streams: the main stream and the secondary stream. This method also applies to multi-stream scenarios when subscribing to RTC. By pre-reading files, the start and stop of the video stream can be detected, and the image composition style can be dynamically adjusted to improve the playback quality of the transcoded file.

[0099] Example 5

[0100] Figure 6 This is a schematic diagram of a video synthesis device according to Embodiment 5 of the present invention. Figure 6 As shown, the device includes:

[0101] The information pre-reading module 410 is used to determine the number of video bitstreams in the video data to be processed, and to determine the video frame rate and playback duration of different video bitstreams in the video data to be processed based on the number of video bitstreams.

[0102] The bitstream decoding module 420 is used to acquire each of the video bitstreams corresponding to the video data to be processed, and generate decoding data for each of the video bitstreams.

[0103] The image compositing module 430 is used to compose images of each of the decoded data based on the video frame rate and the playback duration.

[0104] In this embodiment of the invention, the information pre-reading module identifies the number of video streams in the video data to be processed, identifies the video frame rate and playback duration of different video streams according to the number of video streams, the stream decoding module obtains and decodes each video stream to obtain the decoded data of each video stream, and the picture composition module performs picture composition on each decoded data according to the video frame rate and playback duration. The video stream used for picture composition can be dynamically adjusted according to the number of video streams, which can solve the picture abnormality problem caused by picture composition, improve the picture composition effect, and enhance the user experience.

[0105] Furthermore, based on the above embodiments of the invention, the information pre-reading module 410 includes:

[0106] The video stream count unit is used to read the header file of the video data to be processed and extract the number of video streams carried in the header file.

[0107] The synthesis parameter unit is used to read the video bitstream of the video data to be processed, and determine the video frame rate and the playback duration based on the timestamp information and the number of video frames in each video bitstream.

[0108] Based on the above embodiments of the invention, the synthesis parameter unit is specifically used for: generating time segment information of the video stream for each video stream, setting the timestamp information of the first video frame within the category as the start and end times of the time segment information, and initializing the number of segmented video frames of the time segment information to 1; determining whether the time difference between the timestamp information of each video frame and the end time of the time segment information is less than a threshold time for each video stream; if so, updating the end time to the timestamp information of the video frame and controlling the number of segmented video frames to increment by 1; if not, and the video frame is a keyframe, generating new time segment information for the video stream, setting the timestamp information of the current video frame as the start and end times of the new time segment information, and setting the number of segmented video frames of the new time segment information to 1; saving each time segment information of each video stream as the playback duration, and calculating the video frame rate based on the start time, end time, and number of segmented video frames of each time segment information.

[0109] Based on the above embodiments of the invention, the synthesis parameter unit calculates the video frame rate according to the start time, end time, and number of segmented video frames of each time segment information, including: for each time segment information, determining the time length between the end time and the start time, and using the ratio of the number of segmented video frames to the time length as the segmented video frame rate of the time segment information; for each video stream, obtaining the segmented video frame rate of each time segment information as the video frame rate.

[0110] Based on the above embodiments of the invention, the stream decoding module 420 includes:

[0111] The decoder unit is used to create a decoder for each of the video bitstreams in the video data to be processed.

[0112] An information extraction unit is used to input each of the received video streams into the corresponding decoder for decoding to obtain the decoded data and stream information of the video streams, wherein the stream information includes at least one of size information, current timestamp, and video frame rate.

[0113] A data storage unit is used to store the decoding data and the bitstream information of each of the video bitstreams.

[0114] Based on the above embodiments of the invention, the image compositing module 430 includes:

[0115] The target stream acquisition unit is used to sequentially read the video stream with the largest video bitrate as the target video stream according to each playback duration.

[0116] The image compositing unit is used to generate the composite image according to the target video bitstream when the target video bitstream is decoded.

[0117] Based on the above embodiments of the invention, the image compositing unit is used to obtain the target video frame rate and target playback duration of the target video bitstream; determine the number of decoded streams that have obtained the decoded data at the current time, obtain the image style corresponding to the number of decoded streams, and set the duration of the image style as the target playback duration; and perform image compositing on each of the obtained decoded data according to the target video frame rate, the target playback duration, and the image style.

[0118] Based on the above embodiments of the invention, it further includes: a screen adjustment module, used to, if the difference between the timestamp information of the current frame of the target decoded data and the current frame of other decoded data is greater than a first time threshold, not to perform the screen compositing on the target decoded data and to clear the target decoded data.

[0119] The video synthesis apparatus provided in the embodiments of the present invention can execute the video synthesis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0120] Example 6

[0121] Figure 7 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0122] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video compositing methods.

[0125] In some embodiments, the video compositing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video compositing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video compositing method by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0131] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0132] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0133] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video synthesis method, characterized in that, The method includes: Determine the number of video bitstreams in the video data to be processed, and determine the video frame rate and playback duration of different video bitstreams within the video data to be processed based on the number of video bitstreams; Obtain each video bitstream corresponding to the video data to be processed, and generate decoding data for each video bitstream; The decoded data are composited based on the video frame rate and the playback duration.

2. The method according to claim 1, characterized in that, The step of determining the number of video bitstreams in the video data to be processed, and determining the video frame rate and playback duration of different video bitstreams within the video data to be processed based on the number of video bitstreams, includes: Read the header file of the video data to be processed, and extract the number of video bitstreams carried in the header file; The video bitstream of the video data to be processed is read, and the video frame rate and the playback duration are determined based on the timestamp information and the number of video frames in each video bitstream.

3. The method according to claim 2, characterized in that, The step of determining the video frame rate and the playback duration based on the timestamp information and the number of video frames within each video stream includes: For each video stream, time segment information of the video stream is generated, and the timestamp information of the first video frame in the category is set as the start and end time of the time segment information. The number of segmented video frames of the time segment information is initialized to 1. For each video stream, determine whether the time difference between the timestamp information of each video frame and the end time of the time segment information is less than a threshold time. If so, the end time is updated to the timestamp information of the video frame, and the number of segmented video frames is incremented by 1. If not, and the video frame is a keyframe, then generate new time segment information for the video stream, set the timestamp information of the current video frame as the start and end time of the new time segment information, and set the number of video frames in the new time segment information to 1. The time segment information of each of the video streams is saved as the playback duration, and the video frame rate is calculated based on the start time, end time and number of video frames in each time segment information.

4. The method according to claim 3, characterized in that, The step of calculating the video frame rate based on the start time, end time, and number of video frames in each time segment includes: For each of the aforementioned time segments, the time length between the end time and the start time is determined, and the ratio of the number of segmented video frames to the time length is used as the segmented video frame rate of the time segment information. For each of the video streams, the segmented video frame rate of each time period sub-information is obtained as the video frame rate.

5. The method according to claim 1, characterized in that, The step of acquiring each video bitstream corresponding to the video data to be processed and generating decoding data for each video bitstream includes: A decoder is created for each video bitstream within the video data to be processed. Each of the received video streams is input to the corresponding decoder for decoding to obtain the decoded data and stream information of the video stream, wherein the stream information includes at least one of size information, current timestamp and video frame rate; The decoding data and bitstream information of each video stream are saved.

6. The method according to claim 1 or 5, characterized in that, The step of combining the decoded data based on the video frame rate and the playback duration includes: The video stream with the highest video bitrate is read sequentially according to each playback duration as the target video stream; When the target video stream is decoded, the composite image is generated according to the target video stream.

7. The method according to claim 6, characterized in that, The step of generating the composite image according to the target video stream after the target video stream decoding is completed includes: Obtain the target video frame rate and target playback duration of the target video stream; Determine the number of decoded streams that have acquired the decoded data at the current moment, obtain the picture style corresponding to the number of decoded streams, and set the duration of the picture style as the target playback duration; The obtained decoded data are composited according to the target video frame rate, the target playback duration, and the picture style.

8. The method according to claim 1, 6, or 7, characterized in that, Also includes: If the difference between the timestamp information of the current frame of the target decoded data and the current frame of other decoded data is greater than the first time threshold, then the image compositing is not performed on the target decoded data, and the target decoded data is cleared.

9. A video synthesis device, characterized in that, The device includes: The information pre-reading module is used to determine the number of video bitstreams in the video data to be processed, and to determine the video frame rate and playback duration of different video bitstreams in the video data to be processed based on the number of video bitstreams. The bitstream decoding module is used to acquire each of the video bitstreams corresponding to the video data to be processed, and to generate decoding data for each of the video bitstreams; The image compositing module is used to compose images of each of the decoded data based on the video frame rate and the playback duration.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video synthesis method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video synthesis method according to any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the video synthesis method according to any one of claims 1-8.

Images