Bitstream scheduling method, bitstream composition method, and bitstream scheduling system

Abstract

The application provides a code stream scheduling method, a system and a code stream synthesis method. The code stream scheduling method comprises: judging whether to schedule a current video code stream to a synthesis media processing module according to video code stream information of the current video code stream and a code stream processing state of the synthesis media processing module; if not, scheduling the current video code stream to an idle preprocessing media processing module according to the video code stream information of the current video code stream and the code stream processing state of each preprocessing media processing module, so that the current video code stream is preprocessed by the idle preprocessing media processing module; and scheduling the preprocessed video code stream to the synthesis media processing module, so that the synthesis media processing module synthesizes the preprocessed video code stream. The application realizes code stream scheduling of multimedia processing modules, so that normal synthesis is avoided due to the limitation of single media chip coding and decoding capacity and bus bandwidth.

Description

Technical Field

[0001] This application relates to the field of video stream transmission, and in particular to a stream scheduling method, a stream synthesis method, and a stream scheduling system. Background Technology

[0002] With the widespread use of converged communication technologies in public security, education, healthcare, and law enforcement departments, the types of media being integrated into these systems are becoming increasingly complex, especially with the increasing prevalence of high-definition video sources. The demands for image compositing are also rising, leading to a wider range of scenarios requiring complex image compositing outputs with high resolution and multiple channels. While ordinary dedicated media processing chips can perform encoding and decoding on a single chip, simplifying the process, the increasing resolution and bitrate of video sources places higher demands on decoding capabilities and network bandwidth. The limited encoding / decoding capabilities and bus bandwidth of single-node media processing chips are insufficient to meet the application requirements of higher-resolution, multi-channel image compositing.

[0003] Therefore, how to implement bitstream scheduling of multimedia processing modules to avoid the inability to perform normal image synthesis due to the limitations of single media chip encoding / decoding capabilities and bus bandwidth is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, this application provides a bitstream scheduling method, a bitstream synthesis method, and a bitstream scheduling system, which realizes the synthesis of images from more video streams through bitstream scheduling of the multimedia processing module.

[0005] According to one aspect of this application, a bitstream scheduling method is provided, comprising:

[0006] Based on the video stream information of the current video stream to be synthesized and the stream processing status of the synthesized media processing module, determine whether to schedule the current video stream to be synthesized to the synthesized media processing module;

[0007] If not, based on the video stream information of the current video stream to be synthesized and the stream processing status of each preprocessing media processing module, the current video stream to be synthesized is scheduled to an idle preprocessing media processing module so that the idle preprocessing media processing module can preprocess the current video stream to be synthesized.

[0008] The preprocessed video stream is scheduled to the composite media processing module so that the composite media processing module can synthesize the preprocessed video stream.

[0009] In some embodiments of this application, determining whether to schedule the current video stream to be synthesized to the synthesis media processing module based on the video stream information of the current video stream to be synthesized and the stream processing status of the synthesis media processing module includes:

[0010] Based on the video bitstream information of the current video bitstream to be synthesized, predict the total number of bitstreams processed by the synthesized media processing module when the synthesized media processing module preprocesses the current video bitstream to be synthesized.

[0011] Determine whether the total number of streams processed by the synthetic media processing module is less than the number of streams threshold;

[0012] If so, the current video stream to be synthesized will be scheduled to the synthesized media processing module;

[0013] If not, the currently synthesized video stream will be scheduled to an idle preprocessing media processing module for preprocessing.

[0014] In some embodiments of this application, scheduling the current video stream to be synthesized to an idle preprocessing media processing module based on the video stream information of the current video stream to be synthesized and the stream processing status of each of the preprocessing media processing modules includes:

[0015] Based on the video bitstream information of the current video bitstream to be synthesized, it is sequentially determined whether each of the preprocessing media processing modules has reached the maximum transcoding capacity when preprocessing the current video bitstream to be synthesized, until a preprocessing media processing module has not reached the maximum transcoding capacity when preprocessing the current video bitstream to be synthesized.

[0016] The preprocessing media processing module that has not reached its maximum transcoding capacity is designated as an idle preprocessing media processing module.

[0017] In some embodiments of this application, the number of channels threshold is a decoding number threshold, which is calculated based on the resolution and frame rate of the video stream to be synthesized, the target resolution and frame rate of the preprocessing, the maximum target number of synthesis channels, and the maximum decoding capability of a single synthesis media processing module; or

[0018] The number of channels threshold is a bitrate number of channels threshold, which is calculated based on the bitrate of the video bitstream, the target bitrate of preprocessing, the maximum target number of synthesis channels, and the maximum bitrate of a single synthesis media processing module. The target bitrate of preprocessing is calculated based on the target resolution of preprocessing.

[0019] In some embodiments of this application, the number of paths threshold is a smaller threshold between the number of decoding paths threshold and the number of bitrate paths threshold.

[0020] In some embodiments of this application, the preprocessing step includes converting the resolution and / or bitrate of the video bitstream to be synthesized to a target resolution and / or bitrate, wherein the target resolution and / or bitrate is determined based on the position and / or image parameters of the synthesis window, the position and / or image parameters of each synthesis window are determined based on the number of synthesis channels of the synthesis media processing module, and the synthesis media processing module allocates a synthesis channel to each video bitstream synthesized by the synthesis media processing module.

[0021] In some embodiments of this application, the compositing window and / or image parameters change with the number of compositing channels, and in response to the change in the compositing window and / or image parameters, the target resolution and / or bitrate of the video bitstream corresponding to the existing compositing channel are updated.

[0022] In some embodiments of this application, in response to the difference between the resolution / bitrate of the video stream to be synthesized and the target resolution / bitrate, the step of determining whether to schedule the current video stream to be synthesized to the synthesized media processing module based on the bitstream processing status of the synthesized media processing module is executed.

[0023] In response to the fact that the resolution / bitrate of the video stream to be synthesized is the same as the target resolution / bitrate, the current video stream to be synthesized is scheduled to the synthesis media processing module so that the synthesis media processing module can synthesize the current video stream to be synthesized.

[0024] According to another aspect of this application, a method for synthesizing bitstreams is also provided, comprising:

[0025] In response to the signaling control and scheduling module receiving the video stream information of a newly added video stream to be synthesized, the signaling control and scheduling module updates the synthesis window and / or screen parameters based on the video stream information of the newly added video stream to be synthesized and the video stream information of the existing video streams to be synthesized.

[0026] Based on the updated compositing window and / or screen parameters, determine the target resolution and / or bitrate of the newly added video stream to be composited, and update the target resolution and / or bitrate of the existing video streams to be composited.

[0027] The signaling control and scheduling module determines whether the newly added video stream to be synthesized should be preprocessed by the synthesizing media processing module or the preprocessing media processing module based on the video stream information of the newly added video stream to be synthesized, the stream processing status of the synthesizing media processing module, and the stream processing status of each preprocessing media processing module.

[0028] The signaling control and scheduling module determines the synthetic media processing module or the preprocessing media processing module to preprocess the newly added video stream to be synthesized based on the destination resolution and / or bitrate of the newly added video stream to be synthesized.

[0029] The preprocessing media processing module and / or the compositing media processing module preprocess the existing video bitstream to be synthesized based on the target resolution and / or bitrate of the existing video bitstream to be synthesized.

[0030] The signaling control scheduling module schedules the pre-processed video stream to the composite media processing module;

[0031] The synthetic media processing module synthesizes a pre-processed video stream.

[0032] According to one aspect of this application, a stream scheduling system is also provided, including a preprocessing media processing module, a composite media processing module, a signaling control scheduling module, and a video stream forwarding module, wherein...

[0033] The signaling control and scheduling module is used for:

[0034] Based on the bitstream processing status of the synthesized media processing module, the bitstream processing status of each of the preprocessing media processing modules, and the video bitstream information of the current video bitstream to be synthesized, a media processing module is determined to perform the preprocessing steps of the current video bitstream to be synthesized. The media processing module is either the synthesized media processing module or an idle preprocessing media processing module.

[0035] The identifier of the determined media processing module is sent to the video stream forwarding module;

[0036] Send a preprocessing instruction to the determined media processing module;

[0037] Send a synthesis command to the synthesis media processing module.

[0038] The video stream forwarding module is used for:

[0039] Negotiate media stream information with the signaling control and scheduling module;

[0040] Based on the identifier of the media processing module sent by the signaling control and scheduling module, the video stream to be synthesized is forwarded to the media processing module.

[0041] In response to the preprocessing of the current video stream to be synthesized by the preprocessing media module, the video stream output by the media processing module is forwarded to the synthesizing media processing module according to the identifier of the synthesizing media processing module sent by the signaling control scheduling module.

[0042] The preprocessing media processing module is used to preprocess the received video bitstream according to the preprocessing instructions;

[0043] The composite media processing module is used to preprocess the received video stream according to the preprocessing instructions, and to composite the received video stream according to the composite instructions.

[0044] According to another aspect of this application, a stream scheduling device is also provided, comprising:

[0045] The judgment module is used to determine whether to schedule the current video stream to be synthesized to the synthesis media processing module based on the video stream information of the current video stream to be synthesized and the stream processing status of the synthesis media processing module.

[0046] The bitstream preprocessing scheduling module, when the first judgment module determines no, is used to schedule the current video bitstream to be synthesized to an idle preprocessing media processing module according to the video bitstream information of the current video bitstream to be synthesized and the bitstream processing status of each preprocessing media processing module, so that the idle preprocessing media processing module can preprocess the current video bitstream to be synthesized.

[0047] The bitstream synthesis scheduling module is used to schedule the pre-processed video bitstream to the synthesis media processing module so that the synthesis media processing module can synthesize the pre-processed video bitstream.

[0048] According to another aspect of this application, an electronic device is also provided, the electronic device comprising: a processor; and a storage medium having a computer program stored thereon, the computer program being executed by the processor to perform the steps described above.

[0049] According to another aspect of this application, a storage medium is also provided, on which a computer program is stored, the computer program being executed by a processor to perform the steps described above.

[0050] Therefore, the solution provided in this application has the following advantages compared with the prior art:

[0051] This application decouples video stream preprocessing from video stream synthesis. It determines whether the synthesis media processing module should perform preprocessing based on the video stream information of the current video stream to be synthesized and the stream processing status of the synthesis media processing module. If not, it schedules the video stream to be synthesized to an idle preprocessing media processing module based on the video stream information and the stream processing status of the preprocessing media processing module. The idle preprocessing media processing module then preprocesses the video stream to be synthesized and schedules the preprocessed video stream to the synthesis media processing module, which synthesizes the preprocessed video stream. Thus, by scheduling the video streams of the preprocessing media processing module and the synthesis media processing module, it ensures that the streams can be synthesized normally, avoiding the problem of failure to synthesize normally due to limitations in the encoding / decoding capabilities and bus bandwidth of a single media chip. Attached Figure Description

[0052] The above and other features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0053] Figure 1 A flowchart of a bitstream scheduling method according to an embodiment of this application is shown.

[0054] Figure 2 A flowchart of a stream scheduling method according to a specific embodiment of this application is shown.

[0055] Figure 3 A schematic diagram of a stream scheduling system according to a specific embodiment of this application is shown.

[0056] Figure 4 A flowchart of a bitstream synthesis method according to an embodiment of this application is shown.

[0057] Figure 5 A block diagram of a stream scheduling apparatus according to an embodiment of this application is shown.

[0058] Figure 6 This illustration schematically depicts a computer-readable storage medium according to an exemplary embodiment of the present disclosure.

[0059] Figure 7 The schematic diagram illustrates an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0060] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0061] Furthermore, the accompanying drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0062] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0063] To overcome the shortcomings of the existing technology, this application provides a bitstream scheduling method, bitstream scheduling device, and system to achieve bitstream scheduling of the multimedia processing module, thereby avoiding the inability to synthesize properly due to limitations in the encoding / decoding capabilities and bus bandwidth of a single media chip. The media processing module described in this application can be a media processing chip.

[0064] First see Figure 1 , Figure 1 A flowchart of a bitstream scheduling method according to an embodiment of this application is shown. Figure 1 The following steps are shown:

[0065] Step S110: Based on the video stream information of the current video stream to be synthesized and the stream processing status of the synthesized media processing module, determine whether to schedule the current video stream to be synthesized to the synthesized media processing module.

[0066] If step S110 determines no, then step S120 is executed: based on the video stream information of the current video stream to be synthesized and the stream processing status of each preprocessing media processing module, the current video stream to be synthesized is scheduled to an idle preprocessing media processing module so that the idle preprocessing media processing module can preprocess the current video stream to be synthesized.

[0067] If step S110 determines that the desired outcome is achieved, the current video stream to be synthesized can be directly scheduled to the synthesizing media processing module, where it will perform preprocessing. In other embodiments, if the video stream to be synthesized is scheduled to the synthesizing media processing module, the preprocessing step can be skipped and the video stream can be synthesized directly.

[0068] Step S130: The preprocessed video stream is scheduled to the composite media processing module so that the composite media processing module can synthesize the preprocessed video stream.

[0069] Specifically, the preprocessing media processing module and the compositing media processing module can be the same or different media processing modules. In other words, the same media processing module can perform only the compositing of the video stream or only the preprocessing of the video stream, or it can perform both the compositing and preprocessing of the video stream simultaneously. The media processing module can be a physical media processing chip, but this application is not limited to this.

[0070] The bitstream scheduling method provided in this application decouples the preprocessing of the video bitstream from the synthesis of the video bitstream. It determines whether the synthesis media processing module should perform bitstream preprocessing based on the video bitstream information of the current video bitstream to be synthesized and the bitstream processing status of the synthesis media processing module. If not, it schedules the video bitstream to be synthesized to an idle preprocessing media processing module based on the video bitstream information of the current video bitstream to be synthesized and the bitstream processing status of the preprocessing media processing module. The idle preprocessing media processing module then preprocesses the video bitstream to be synthesized and schedules the preprocessed video bitstream to the synthesis media processing module, which synthesizes the preprocessed video bitstream. Thus, by scheduling the bitstreams of the preprocessing media processing module and the synthesis media processing module, it ensures that the bitstream can be synthesized normally, avoiding the problem of failure to synthesize normally due to limitations in the encoding / decoding capabilities and bus bandwidth of a single media chip.

[0071] Specifically, when the number of video streams to be synthesized is within the processing capacity of the compositing media processing module, video stream synthesis and preprocessing can be performed within the same module. This avoids signaling latency, stream latency, and resource waste caused by cross-media processing module access, thus saving project costs. When the number of video streams to be synthesized is outside the processing capacity of the compositing media processing module, a single module cannot simultaneously perform video stream synthesis and preprocessing. In this case, preprocessing can be performed by other preprocessing media processing modules based on their load conditions.

[0072] In some embodiments, whether to schedule the current video stream to be synthesized to the synthesis media processing module can be determined by comparing the total number of streams processed by the synthesis media processing module with a first threshold number when the synthesis media processing module preprocesses the current video stream to be synthesized.

[0073] In the above embodiment, step S110 can be implemented by the following steps: based on the video bitstream information of the current video bitstream to be synthesized, predict the total number of bitstream processing channels of the synthesizing media processing module when the synthesizing media processing module preprocesses the current video bitstream to be synthesized; determine whether the total number of bitstream processing channels of the synthesizing media processing module is less than the number of channels threshold; if yes, schedule the current video bitstream to be synthesized to the synthesizing media processing module; if no, schedule the current video bitstream to be synthesized to the idle preprocessing media processing module for preprocessing.

[0074] Specifically, the number of streams threshold is the upper limit of the number of video streams that the compositing and processing media processing module is allowed to preprocess simultaneously.

[0075] In some embodiments, in practical applications, the decoding resource consumption in the composite media processing module is often greater than the encoding resource consumption. Therefore, the number of channels threshold can be the decoding number threshold, which is calculated based on the resolution and frame rate of the video bitstream, the target resolution and frame rate of the preprocessing, the maximum target number of composite channels, and the maximum decoding capability of a single composite media processing module.

[0076] For example, the decoding path threshold can be calculated using the following formula:

[0077]

[0078] Wherein, the resolution of the video stream (stream source) is w1*h1, and the frame rate of the video stream (stream source) is f1; the target resolution of the preprocessing is w2*h2, and the target frame rate of the preprocessing is f2; the actual maximum number of target synthesis channels is N; the maximum decoding capability of a single synthesis media processing module is E, and ρ is a calculation coefficient. This application is not limited thereto, and other calculation methods for the number of decoding channels threshold are all within the protection scope of this application.

[0079] In other embodiments, in practical applications, the total uplink bitrate in the composite media processing module is often greater than the total downlink bitrate. The number of channels threshold can be a bitrate number of channels threshold, which is calculated based on the bitrate of the video stream, the target bitrate of preprocessing, the maximum target number of composite channels, and the maximum bitrate of a single composite media processing module.

[0080] For example, the bitrate threshold can be calculated using the following formula:

[0081]

[0082] Wherein, the bitrate of the video stream (stream source) is b, the target resolution of the preprocessing is w²*h², the actual maximum number of target synthesis channels is N, the maximum decoding bitrate of a single synthesis media processing module is B, and δ is a calculation coefficient. In this embodiment, the target bitrate and target resolution of the preprocessing can maintain a proportional relationship, thus, the target bitrate of the preprocessing is calculated based on the target resolution of the preprocessing. For example, bitrate = 1.28 * target resolution, then δ is 1.28. This application is not limited to this, and other proportional coefficients and other calculation methods for the bitrate channel threshold are all within the protection scope of this application.

[0083] In some other embodiments, the number of paths threshold can be a smaller threshold between the number of decoding paths threshold and the number of bitrate paths threshold.

[0084] Therefore, as the number of video streams to be synthesized increases, when the encoding / decoding consumption or uplink / downlink bitrate of the compositing media processing module reaches a threshold, new video streams to be synthesized can be pre-scheduled to other available pre-media processing modules for pre-processing. This pre-processing pre-compiles resolution and bitrate reduction, reducing the decoding consumption of the compositing media processing module. The pre-processed video stream is then transmitted over the network and returned to the compositing media processing module where the compositing task is located, where the final image synthesis is completed. Through the collaborative processing of multiple media processing modules, the image synthesis effect of more video streams is achieved.

[0085] In some embodiments, whether the preprocessing media processing module is idle can be determined based on whether the preprocessing media processing module reaches its maximum transcoding capacity when preprocessing the current video bitstream to be synthesized.

[0086] In the above embodiment, step S120 can be implemented by the following steps: based on the video bitstream information of the current video bitstream to be synthesized, sequentially determine whether each of the preprocessing media processing modules has reached the maximum transcoding capacity when preprocessing the current video bitstream to be synthesized, until a preprocessing media processing module has not reached the maximum transcoding capacity when preprocessing the current video bitstream to be synthesized; the preprocessing media processing module that has not reached the maximum transcoding capacity is designated as an idle preprocessing media processing module.

[0087] Specifically, for each new video stream to be synthesized, the compositing media processing module allocates a compositing channel to that video stream. This compositing channel instructs the compositing media processing module to send the video stream through it for synthesis. The synthesized video stream's composite frame includes multiple compositing windows, each corresponding to a compositing channel. Specifically, the compositing media processing module is used to synthesize and splice video streams from multiple video sources into a single composite stream. The composite stream can have various splicing formats (e.g., compositing styles). Splicing formats may include, but are not limited to, dual-screen, quad-screen, etc. Each small window in the composite stream's frame corresponds to a compositing channel. Therefore, the preprocessing step may include converting the resolution and / or bitrate of the video stream to be synthesized to a target resolution and / or bitrate, wherein the target resolution and / or bitrate is determined based on the position and / or frame parameters of the compositing windows. The position and / or frame parameters of each compositing window can be determined based on the number of compositing channels in the compositing media processing module. For example, if there are two compositing channels, the compositing window can be a horizontally arranged dual-screen display; if there are four compositing channels, the compositing window can be a four-screen display arranged in two rows and two columns. Therefore, based on the number of compositing channels, the position and size of the compositing window for each video stream, as well as the image parameters (such as bitrate and / or resolution) within each window, can be determined. Thus, the size of each compositing window and the image parameters (image style parameters) within each window can be used to determine the target resolution and / or bitrate for the video stream preprocessing corresponding to that compositing window.

[0088] This application is not limited to this; video stream preprocessing may also include other preprocessing steps.

[0089] Furthermore, when the video bitstream to be synthesized increases / decreases (synthesis channels increase / decrease), or when the video bitstream information to be synthesized changes, the number of synthesis windows changes accordingly, thus changing the synthesis windows and / or image parameters of the synthesized image. At this time, the target resolution and / or bitrate of the video bitstream corresponding to the existing synthesis channels can be updated. When the video bitstream increases, the target resolution and / or bitrate of the preprocessing of the newly added video bitstream can also be set to achieve dynamic updates of the bitstream preprocessing.

[0090] Specifically, since the target resolution / bitrate of each video stream is determined based on the number and arrangement of the composite windows in the composite image, when the resolution / bitrate of the video stream matches the target resolution / bitrate, preprocessing of the video stream is unnecessary. The current video stream to be composited is then scheduled to the composite media processing module for synthesis. In other words, when the resolution / bitrate of the video stream to be composited differs from the target resolution / bitrate, the step of determining whether to schedule the current video stream to be composited to the composite media processing module based on its stream processing status can be executed, thereby avoiding redundant judgment and scheduling steps.

[0091] See below. Figure 2 , Figure 2 A flowchart of a stream scheduling method according to a specific embodiment of this application is shown. Figure 2 The following steps are shown:

[0092] Step S301: Enable screen composition.

[0093] Step S302: Add a synthesis channel.

[0094] Specifically, the composite media processing module is used to combine and splice video streams from multiple video sources into a single composite stream. Each composite window in the composite stream corresponds to a composite channel. Since image compositing requires adding composite elements, step S302 is executed by the composite media processing module. Each time a new video stream to be composited is added, the composite media processing module allocates a composite channel for that video stream. This composite channel is used to instruct the composite media processing module to send the video stream to be composited through this channel. Furthermore, when the composite media processing module receives a new video stream to be composited from the signaling control and scheduling module, it allocates a composite channel for that video stream and feeds it back to the signaling control and scheduling module, so that the signaling control and scheduling module can send the video stream to be composited to the composite media processing module through the composite channel for composited processing.

[0095] Step S303: Adjust the image composition style.

[0096] Specifically, the image compositing style can be set according to the number of video streams to be composited, the display parameters of the display device, user preferences, etc. The image compositing style may include, but is not limited to, the layout of the compositing channels, such as the number of compositing windows, the position of the compositing windows, the bitrate and / or resolution of the video streams in each compositing window, etc. For example, the compositing channel layout can be a dual-window layout (side-by-side windows; picture-in-picture windows); a triple-window layout (one window on top and two windows below; one window on the left and two windows on the right), etc., and this application is not limited to these.

[0097] Step S304: Based on the video bitstream information of the current video bitstream to be synthesized, predict the total number of bitstream processing channels of the synthesized media processing module when the synthesized media processing module preprocesses the current video bitstream to be synthesized, and determine whether the total number of bitstream processing channels of the synthesized media processing module is greater than the number of channels threshold of the synthesized media processing module.

[0098] If step S304 determines that it is yes, it means that the composite media processing module can no longer perform preprocessing of the current video bitstream to be composited. Therefore, step S305 can be executed: determine whether the current composite channel already has a composite channel.

[0099] Specifically, in this embodiment, the target resolution / bitrate of each video stream synthesized by the compositing media processing module needs to be updated according to the picture style. Therefore, step S305 determines whether the current compositing channel is an existing compositing channel. For example, when the style changes from one window to two windows, the target resolution / bitrate of the video stream in the existing compositing channel also needs to be adjusted. The new compositing channel is preprocessed directly based on the adjusted picture style. With more windows, the resolution of each small frame also needs to be adjusted. Therefore, step S305 determines whether there is an existing channel branch.

[0100] If step S305 determines otherwise, the current compositing channel is the compositing channel added for the newly added video stream, and step S306 can be executed: find an idle preprocessing media processing module. Based on the video stream information of the current video stream to be composed, it can be sequentially determined whether each preprocessing media processing module has reached its maximum transcoding capacity when preprocessing the current video stream to be composed, until a preprocessing media processing module does not reach its maximum transcoding capacity when preprocessing the current video stream to be composed; this preprocessing media processing module that has not reached its maximum transcoding capacity is designated as an idle preprocessing media processing module. When an idle preprocessing media processing module is found, step S307 can be executed: determine whether the resources of the idle preprocessing media processing module are sufficient to perform the preprocessing of the current video stream to be composed. Specifically, resources may include, but are not limited to, processing resources and storage resources.

[0101] If step S307 is incorrect, then step S306 can be re-executed: search for an available preprocessing media processing module again.

[0102] If step S307 determines that it is yes, then step S308 is executed: the current video stream to be synthesized is scheduled to the determined preprocessing media processing module.

[0103] After step S308, step S309 is executed: the determined preprocessing media processing module preprocesses the current video stream to be synthesized according to the target resolution and / or bitrate determined by the aforementioned picture compositing style. After step S309, step S311 is executed.

[0104] If step S305 determines "yes," it means the current compositing channel is an existing compositing channel. Then, step S310 can be executed: reset the preprocessing task for the existing compositing channel according to the new compositing style and the position of the channel in the compositing frame, and preprocess the corresponding video stream to be composited according to the reset preprocessing task. Step S311 is then executed after step S310.

[0105] Step S311: Send the preprocessed video stream to the composite media processing module. Step S313 is then executed after step S311.

[0106] If step S304 is incorrect, then step S312 is executed: the composite media processing module preprocesses the current video stream to be composited.

[0107] Step S313: The preprocessed video stream is decoded and synthesized by the composite media processing module.

[0108] Step S314: Output the composite image.

[0109] The above is merely an illustrative representation of the bitstream scheduling method of this application. When applied to video conferencing, with an initial number of N participants, the video streams of the N participants can be composited according to the above process. When a new participant joins the video conference, the composite style of the existing and new video streams can be adjusted, and the above steps are executed to determine whether the number of channels for a single composite media processing module has been reached after the addition of the new participant. If not, the composite media processing module is used for processing; if so, the existing channels are preprocessed to the target resolution based on the new image style and channel position; if not, other preprocessing media processing modules are searched, and idle resources are used to provide preprocessing requests for that channel. In other words, the composite style is dynamically adjusted in real time with the addition of a new participant, and the corresponding media processing module is dynamically adapted. The video conferencing scenario is merely illustrative, and this application is not intended to limit it. Other scenarios requiring video stream composite, such as surveillance scenarios, are within the scope of protection of this application.

[0110] In a specific implementation, taking a certain composite media processing module as an example, the maximum decoding capability of a single composite media processing module is (4090*2688@120 frames per second), supporting a maximum of 16 channels (1080p@30 frames per second) decoding, and the maximum encoding capability is (4096*2688@60 frames per second). The multi-stream real-time encoding capability is (4090*2688@30 frames per second) + (1080p@30 frames per second) + (1080p@30 frames per second) + (1080p@30 frames per second). The maximum decoding bitrate is 300Mbps, and the maximum encoding output bitrate is 200Mbps.

[0111] In video merging scenarios, taking a source media stream with a maximum of 25 channels / H264 (transmission protocol) / 4M / (1080p@30 frames per second) and a destination media stream with a maximum of (4090*2688@30 frames per second) as an example, a single frame is dynamically added to the merging task of 25 frames:

[0112] Based on the above parameters, the threshold number of channels for a single compositing media processing module in this compositing task can be calculated to be 10.

[0113] When the number of video synthesis channels is 10 or less, the total decoding consumption is less than (1080p@30 frames per second) * 10, the total uplink bitrate is less than 40M, pre-decoding and synthesis are completed on the synthesis media processing module, and finally the synthesized output is output.

[0114] When the number of video compositing channels increases to 11, the newly added channel resources can be pre-allocated to other idle pre-processing media processing modules for resolution and bitrate reduction pre-transcoding. After pre-transcoding is completed in the new pre-processing media processing module, the bitstream is transcoded to the compositing media processing module for video compositing and output. Existing pre-decoding tasks will dynamically adjust the target resolution of pre-decoding based on changes in image style and window position. As the number of video compositing channels continues to increase, the above steps are repeated until the compositing task of 25 video bitstreams is completed.

[0115] The above examples illustrate multiple implementations of this application. This application is not intended to limit itself. In each implementation, the addition, omission, or change of the order of steps are all within the protection scope of this application. Each implementation can be implemented individually or in combination.

[0116] See below. Figure 3 , Figure 3 A schematic diagram of a stream scheduling system according to a specific embodiment of this application is shown.

[0117] The bitstream scheduling system includes a preprocessing media processing module 220A, a composite media processing module 220B, a signaling control scheduling module 210, and a video bitstream forwarding module 230.

[0118] The signaling control and scheduling module 210 can be used to determine, based on the bitstream processing status of the synthesized media processing module, the bitstream processing status of each of the preprocessing media processing modules, and the video bitstream information of the current video bitstream to be synthesized, a media processing module for performing the preprocessing steps of the current video bitstream to be synthesized, wherein the media processing module is the synthesized media processing module or an idle preprocessing media processing module; send the identifier of the determined media processing module to the video bitstream forwarding module; send a preprocessing instruction to the determined media processing module; and send a synthesis instruction to the synthesized media processing module. The signaling control and scheduling module 210 may include a scheduling decision unit and a data collection unit. The scheduling decision unit can be used to execute video bitstream scheduling decisions, and the data collection unit can collect the current bitstream processing status of each media processing module.

[0119] The video stream forwarding module 230 is used to negotiate media stream information with the signaling control and scheduling module; according to the identifier of the media processing module sent by the signaling control and scheduling module, it forwards the current video stream to be synthesized to the media processing module; in response to the current video stream to be synthesized being preprocessed by the preprocessing media module, according to the identifier of the synthesizing media processing module sent by the signaling control and scheduling module, it forwards the video stream output by the media processing module to the synthesizing media processing module.

[0120] The media stream information negotiation includes sending information such as video stream size, video stream resolution, and video stream bitrate to the signaling control and scheduling module.

[0121] The preprocessing media processing module 220A is used to preprocess the received video stream according to the preprocessing instructions.

[0122] The composite media processing module 220B is used to preprocess the received video stream according to the preprocessing instructions, and to composite the received video stream according to the composite instructions. The composited video stream is sent to the stream receiving device 240 for playback via the video stream forwarding module 230.

[0123] Both the preprocessing media processing module 220A and the compositing media processing module 220B may include a media processing chip and a media processing unit.

[0124] Therefore, based on the configuration of the multimedia processing module, the bitstream scheduling system completes real-time judgment and dynamic scheduling migration through the signaling control scheduling module, the media processing module receives the signaling and completes the corresponding encoding and decoding processing, and the media stream forwarding module completes the forwarding of the bitstream, together realizing the adaptive preprocessing picture synthesis task.

[0125] The following is combined with Figure 4 This application describes a bitstream synthesis method. The bitstream synthesis method includes the following steps:

[0126] Step S501: In response to the signaling control scheduling module receiving the video stream information of the newly added video stream to be synthesized, the signaling control scheduling module updates the synthesis window and / or screen parameters according to the video stream information of the newly added video stream to be synthesized and the video stream information of the existing video stream to be synthesized.

[0127] Step S502: Based on the updated compositing window and / or screen parameters, determine the target resolution and / or bitrate of the newly added video stream to be composited, and update the target resolution and / or bitrate of the existing video stream to be composited.

[0128] Step S503: The signaling control scheduling module determines whether the newly added video stream to be synthesized will be preprocessed by the synthesizing media processing module or the preprocessing media processing module based on the video stream information of the newly added video stream to be synthesized, the stream processing status of the synthesizing media processing module, and the stream processing status of each preprocessing media processing module.

[0129] Step S504: The synthetic media processing module or preprocessing media processing module determined by the signaling control scheduling module preprocesses the newly added video bitstream to be synthesized based on the destination resolution and / or bitrate of the newly added video bitstream to be synthesized.

[0130] Step S505: The preprocessing media processing module and / or the compositing media processing module preprocess the existing video bitstream to be synthesized based on the target resolution and / or bitrate of the existing video bitstream to be synthesized.

[0131] Step S506: The signaling control scheduling module schedules the preprocessed video stream to the composite media processing module;

[0132] Step S507: The synthetic media processing module synthesizes the pre-processed video bitstream.

[0133] In the bitstream synthesis method provided in this application:

[0134] 1) Decouple the preprocessing of the video stream from the synthesis of the video stream. Determine whether the synthesis media processing module should perform the preprocessing of the video stream based on the video stream information of the newly added video stream to be synthesized and the stream processing status of the synthesis media processing module. If not, schedule the video stream to be synthesized to an idle preprocessing media processing module based on the video stream information of the newly added video stream to be synthesized and the stream processing status of the preprocessing media processing module. The idle preprocessing media processing module will then perform preprocessing on the video stream to be synthesized. The preprocessed video stream will then be scheduled to the synthesis media processing module, which will synthesize the preprocessed video stream. Thus, by scheduling the video streams of the preprocessing media processing module and the synthesis media processing module, the synthesis of the video stream can be ensured to be synthesized normally, thereby avoiding the problem of failure to synthesize normally due to the limitations of the encoding and decoding capabilities and bus bandwidth of a single media chip.

[0135] 2) By dynamically determining the target resolution and / or bitrate of the newly added video stream to be synthesized, the target resolution and / or bitrate of the currently being processed video stream to be synthesized can also be adjusted. Thus, dynamic stream scheduling can be achieved based on the dynamically adjusted target resolution and / or bitrate and the stream processing status of each media processing module, thereby improving the efficiency of stream preprocessing and synthesis.

[0136] The following is combined with Figure 5 The present application describes a stream scheduling device 400. The stream scheduling device 400 includes a judgment module 410, a stream preprocessing scheduling module 420, and a stream synthesis scheduling module 430.

[0137] The judgment module 410 is used to determine whether to schedule the current video stream to be synthesized to the synthesis media processing module based on the video stream information of the current video stream to be synthesized and the stream processing status of the synthesis media processing module.

[0138] When the first judgment module determines that it is not true, the bitstream preprocessing scheduling module 420 is used to schedule the current video bitstream to be synthesized to an idle preprocessing media processing module according to the video bitstream information of the current video bitstream to be synthesized and the bitstream processing status of each preprocessing media processing module, so that the idle preprocessing media processing module can preprocess the current video bitstream to be synthesized.

[0139] The bitstream synthesis scheduling module 430 is used to schedule the pre-processed video bitstream to the synthesis media processing module so that the synthesis media processing module can synthesize the pre-processed video bitstream.

[0140] The bitstream scheduling device provided in this application decouples the preprocessing of the video bitstream from the synthesis of the video bitstream. It determines whether the synthesis media processing module should perform bitstream preprocessing based on the video bitstream information of the current video bitstream to be synthesized and the bitstream processing status of the synthesis media processing module. If not, it schedules the video bitstream to be synthesized to an idle preprocessing media processing module based on the video bitstream information of the current video bitstream to be synthesized and the bitstream processing status of the preprocessing media processing module. The idle preprocessing media processing module then preprocesses the video bitstream to be synthesized and schedules the preprocessed video bitstream to the synthesis media processing module, which synthesizes the preprocessed video bitstream. Thus, by scheduling the bitstreams of the preprocessing media processing module and the synthesis media processing module, it ensures that the bitstreams can be synthesized normally, avoiding the problem of failure to synthesize normally due to limitations in the encoding / decoding capabilities and bus bandwidth of a single media chip.

[0141] This application can implement the stream scheduling device 400 through software, hardware, firmware, or any combination thereof. Figure 5 This application merely illustrates the bitstream scheduling device 400 provided in this application. Without departing from the concept of this application, the splitting, merging, and addition of modules are all within the protection scope of this application.

[0142] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by, for example, a processor, can implement the steps of the bitstream scheduling method described in any of the above embodiments. In some possible implementations, various aspects of this application can also be implemented as a program product comprising program code, which, when run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this application described in the bitstream scheduling method section of this specification.

[0143] refer to Figure 6 As shown, a program product 800 for implementing the above-described method according to an embodiment of this application is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0144] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0145] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0146] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the tenant's computing device, partially on the tenant's device, as a standalone software package, partially on the tenant's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the tenant's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0147] In exemplary embodiments of this disclosure, an electronic device is also provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to perform the steps of the stream scheduling method described in any of the above embodiments by executing the executable instructions.

[0148] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0149] The following reference Figure 7 To describe an electronic device 600 according to this embodiment of the present application. Figure 7 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0150] like Figure 7 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0151] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the above-described bitstream scheduling method section of this specification according to various exemplary embodiments of this application. For example, the processing unit 610 can perform, as follows: Figure 1 The steps are shown in the figure.

[0152] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0153] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0154] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0155] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable tenants to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0156] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the above-described stream scheduling method and stream synthesis method according to the embodiments of this disclosure.

[0157] This application decouples video stream preprocessing from video stream synthesis. It determines whether the synthesis media processing module should perform preprocessing based on the video stream information of the current video stream to be synthesized and the stream processing status of the synthesis media processing module. If not, it schedules the video stream to be synthesized to an idle preprocessing media processing module based on the video stream information and the stream processing status of the preprocessing media processing module. The idle preprocessing media processing module then preprocesses the video stream to be synthesized and schedules the preprocessed video stream to the synthesis media processing module, which synthesizes the preprocessed video stream. Thus, by scheduling the video streams of the preprocessing media processing module and the synthesis media processing module, it ensures that the streams can be synthesized normally, avoiding the problem of failure to synthesize normally due to limitations in the encoding / decoding capabilities and bus bandwidth of a single media chip.

[0158] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A code stream scheduling method, characterized in that, include: Based on the video bitstream information of the current video bitstream to be synthesized and the bitstream processing status of the synthesizing media processing module, determine whether to schedule the current video bitstream to be synthesized to the synthesizing media processing module, including: based on the video bitstream information of the current video bitstream to be synthesized, predict the total number of bitstream processing paths of the synthesizing media processing module when the synthesizing media processing module preprocesses the current video bitstream to be synthesized; and determine whether the total number of bitstream processing paths of the synthesizing media processing module is less than a path threshold. If so, the current video stream to be synthesized will be scheduled to the synthesized media processing module; If not, based on the video stream information of the current video stream to be synthesized and the stream processing status of each preprocessing media processing module, the current video stream to be synthesized is scheduled to an idle preprocessing media processing module so that the idle preprocessing media processing module can preprocess the current video stream to be synthesized. The preprocessed video stream is scheduled to the composite media processing module so that the composite media processing module can synthesize the preprocessed video stream.

2. The stream scheduling method as described in claim 1, characterized in that, The step of scheduling the current video stream to be synthesized to an idle preprocessing media processing module based on the video stream information of the current video stream to be synthesized and the stream processing status of each of the preprocessing media processing modules includes: Based on the video bitstream information of the current video bitstream to be synthesized, it is sequentially determined whether each of the preprocessing media processing modules has reached the maximum transcoding capacity when preprocessing the current video bitstream to be synthesized, until a preprocessing media processing module has not reached the maximum transcoding capacity when preprocessing the current video bitstream to be synthesized. The preprocessing media processing module that has not reached its maximum transcoding capacity is designated as an idle preprocessing media processing module.

3. The stream scheduling method as described in claim 1, characterized in that, The number of channels threshold is the number of decoding channels threshold, which is calculated based on the resolution and frame rate of the video stream to be synthesized, the target resolution and frame rate of the preprocessing, the maximum number of target synthesis channels, and the maximum decoding capability of a single synthesis media processing module. or The number of channels threshold is a bitrate number of channels threshold, which is calculated based on the bitrate of the video bitstream, the target bitrate of preprocessing, the maximum target number of synthesis channels, and the maximum bitrate of a single synthesis media processing module. The target bitrate of preprocessing is calculated based on the target resolution of preprocessing.

4. The stream scheduling method as described in claim 3, characterized in that, The number of paths threshold is the smaller of the decoding path threshold and the bitrate path threshold.

5. The stream scheduling method as described in claim 1, characterized in that, The preprocessing step includes converting the resolution and / or bitrate of the video stream to be synthesized to the target resolution and / or bitrate, wherein the target resolution and / or bitrate is determined according to the position and / or image parameters of the synthesis window, the position and / or image parameters of each synthesis window are determined according to the number of synthesis channels of the synthesis media processing module, and the synthesis media processing module allocates a synthesis channel to each video stream synthesized by the synthesis media processing module.

6. The stream scheduling method as described in claim 5, characterized in that, The compositing window and / or image parameters change with the number of compositing channels. In response to the change in the compositing window and / or image parameters, the target resolution and / or bitrate of the video stream corresponding to the existing compositing channel are updated.

7. The stream scheduling method as described in claim 1, characterized in that, In response to the difference between the resolution / bitrate of the video stream to be synthesized and the target resolution / bitrate, the step of determining whether to schedule the current video stream to be synthesized to the synthesized media processing module based on the bitstream processing status of the synthesized media processing module is executed. In response to the fact that the resolution / bitrate of the video stream to be synthesized is the same as the target resolution / bitrate, the current video stream to be synthesized is scheduled to the synthesis media processing module so that the synthesis media processing module can synthesize the current video stream to be synthesized.

8. A method for synthesizing code streams, characterized in that, include: In response to the signaling control and scheduling module receiving the video stream information of a newly added video stream to be synthesized, the signaling control and scheduling module updates the synthesis window and / or screen parameters based on the video stream information of the newly added video stream to be synthesized and the video stream information of the existing video streams to be synthesized. Based on the updated compositing window and / or screen parameters, determine the target resolution and / or bitrate of the newly added video stream to be composited, and update the target resolution and / or bitrate of the existing video streams to be composited. The signaling control scheduling module determines, based on the video stream information of the newly added video stream to be synthesized, the stream processing status of the synthesizing media processing module, and the stream processing status of each preprocessing media processing module, whether the synthesizing media processing module or the preprocessing media processing module should preprocess the newly added video stream to be synthesized. This includes: predicting the total number of streams processed by the synthesizing media processing module when it preprocesses the current video stream to be synthesized, based on the video stream information of the current video stream to be synthesized; determining whether the total number of streams processed by the synthesizing media processing module is less than a threshold; if so, determining that the synthesizing media processing module should preprocess the newly added video stream to be synthesized; if not, determining that the idle preprocessing media processing module should preprocess the newly added video stream to be synthesized. The signaling control and scheduling module determines the synthetic media processing module or the preprocessing media processing module to preprocess the newly added video stream to be synthesized based on the destination resolution and / or bitrate of the newly added video stream to be synthesized. The preprocessing media processing module and / or the compositing media processing module preprocess the existing video bitstream to be synthesized based on the target resolution and / or bitrate of the existing video bitstream to be synthesized. The signaling control scheduling module schedules the pre-processed video stream to the composite media processing module; The synthetic media processing module synthesizes a pre-processed video stream.

9. A stream scheduling system, characterized in that, It includes a pre-processing media processing module, a composite media processing module, a signaling control and scheduling module, and a video stream forwarding module. The signaling control and scheduling module is used for: Based on the bitstream processing status of the composite media processing module, the bitstream processing status of each of the preprocessing media processing modules, and the video bitstream information of the current video bitstream to be composited, a media processing module is determined to perform the preprocessing steps of the current video bitstream to be composited. The media processing module is either the composite media processing module or an idle preprocessing media processing module. This includes: based on the video bitstream information of the current video bitstream to be composited, predicting the total number of bitstream processing paths of the composite media processing module when it preprocesses the current video bitstream to be composited; determining whether the total number of bitstream processing paths of the composite media processing module is less than a path threshold; if so, determining that the composite media processing module will perform the preprocessing steps of the current video bitstream to be composited; if not, determining that an idle preprocessing media processing module will perform the preprocessing steps of the current video bitstream to be composited. The identifier of the determined media processing module is sent to the video stream forwarding module; Send a preprocessing instruction to the determined media processing module; Send a synthesis command to the synthesis media processing module. The video stream forwarding module is used for: Negotiate media stream information with the signaling control and scheduling module; Based on the identifier of the media processing module sent by the signaling control and scheduling module, the video stream to be synthesized is forwarded to the media processing module. In response to the preprocessing of the current video stream to be synthesized by the preprocessing media module, the video stream output by the media processing module is forwarded to the synthesizing media processing module according to the identifier of the synthesizing media processing module sent by the signaling control scheduling module. The preprocessing media processing module is used to preprocess the received video bitstream according to the preprocessing instructions; The composite media processing module is used to preprocess the received video stream according to the preprocessing instructions, and to composite the received video stream according to the composite instructions.

Images