Cloud live production method, apparatus, device, and computer storage medium

Abstract

Embodiments of the present invention relate to the field of cloud live production technology, and disclose a cloud live production method. The method includes: parsing a first video stream output to a preview window to obtain synchronization mark information; the synchronization mark information is pre-added identification information for characterizing whether the pictures of the preview window and the main monitor window are consistent; when the synchronization mark information characterizes that the pictures of the preview window and the main monitor window are consistent, updating a target video stream output to the main monitor window according to the first video stream. By the above method, the accuracy of frame synchronization of the pictures of the main monitor and the preview in cloud live production is improved in the embodiments of the present invention.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of cloud video switchers, and in particular, to a cloud video switching method, apparatus, device, and computer storage medium. Background Art

[0002] Existing cloud video switchers generally adopt a completely independent pre-monitoring and main-monitoring stream scheme. That is, when a certain video is output to the pre-monitoring, the cloud video switching service processes the video and outputs it as a pre-monitoring stream. In the cloud video switching management page, the pre-monitoring window pulls the pre-monitoring stream for playback. When a request to output the pre-monitoring to the main monitoring is initiated on the cloud video switching management page, the cloud video switching service opens the video stream that the current pre-monitoring is processing again, and after opening, performs the same processing as the pre-monitoring, and finally outputs it as the main monitoring stream. Finally, the main monitoring window on the cloud video switching management page pulls the foregoing main monitoring stream for playback.

[0003] The inventors found in the process of implementing the embodiments of the present invention that: the current cloud video switching method has the problem that the picture frames of the pre-monitoring and the main monitoring are not synchronized, resulting in a poor cloud video switching experience. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a cloud video switching method for solving the problem that the picture frames of the pre-monitoring and the main monitoring in the prior art are not synchronized, resulting in a poor cloud video switching experience.

[0005] According to one aspect of the embodiments of the present invention, a cloud video switching method is provided. The method includes:

[0006] Analyze a first video stream output to a pre-monitoring window to obtain synchronization marker information; the synchronization marker information is pre-added identification information for characterizing whether the pictures of the pre-monitoring window and the main monitoring window are consistent;

[0007] When the synchronization marker information characterizes that the pictures of the pre-monitoring window and the main monitoring window are consistent, update a target video stream output to the main monitoring window according to the first video stream.

[0008] In an optional manner, the method further includes:

[0009] Obtain a picture processing request sent by a user;

[0010] Determine the synchronization marker information according to the picture processing request;

[0011] Add the synchronization marker information to the original video stream pushed to the pre-monitoring window to obtain the first video stream.

[0012] In an optional manner, the method further includes:

[0013] Determine whether the images of the preview window and the main monitor window are consistent according to the image processing request;

[0014] If they are consistent, set the synchronization mark information to the first mark; the first mark is used to indicate that the images of the preview window and the main monitor window are the same;

[0015] If they are inconsistent, set the synchronization mark information to the second mark; the second mark is used to indicate that the images of the preview window and the main monitor window are different.

[0016] In an alternative manner, the image processing request includes an image switching request; the image switching request is used to request switching the preview image to the main monitor image; the method further includes:

[0017] When receiving the image switching request, determine that the images of the preview window and the main monitor window are consistent.

[0018] In an alternative manner, the image processing request further includes an image editing request; the image editing request is used to perform editing processing on the preview image; the method further includes:

[0019] When receiving the image editing request, process the preview image according to the image editing request to obtain a processed image;

[0020] Determine whether the processed image is the same as the currently displayed image of the main monitor window;

[0021] If they are not the same, determine that the images of the preview window and the main monitor window are inconsistent.

[0022] In an alternative manner, the method further includes:

[0023] When determining that the synchronization mark information is the first mark, perform a copy process on the first video stream to obtain a second video stream;

[0024] Update the target video stream to the second video stream.

[0025] In an alternative manner, the method further includes:

[0026] When determining that the synchronization mark information is the second mark, do not update the target video stream.

[0027] According to another aspect of the embodiments of the present invention, there is provided a cloud live production device, including:

[0028] A parsing module for parsing a first video stream output to a preview window to obtain synchronization marker information; the synchronization marker information is pre-added identification information for indicating whether the pictures of the preview window and the main monitoring window are consistent.

[0029] An updating module for updating a target video stream output to the main monitoring window according to the first video stream when the synchronization marker information indicates that the pictures of the preview window and the main monitoring window are consistent.

[0030] According to another aspect of the embodiments of the present invention, a cloud live production device is provided, including:

[0031] A processor, a memory, a communication interface, and a communication bus, where the processor, the memory, and the communication interface complete communication with each other through the communication bus;

[0032] The memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operations of the cloud live production method embodiment as described in any one of the foregoing.

[0033] According to still another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, where at least one executable instruction is stored in the storage medium, and the executable instruction causes a cloud live production device to execute the operations of the cloud live production method embodiment as described in any one of the foregoing.

[0034] In the embodiments of the present invention, a first video stream output to a preview window is parsed to obtain synchronization marker information; the synchronization marker information is pre-added identification information for indicating whether the pictures of the preview window and the main monitoring window are consistent; when the synchronization marker information indicates that the pictures of the preview window and the main monitoring window are consistent, the target video stream output to the main monitoring window is updated according to the first video stream. In the embodiments of the present invention, the main monitoring and preview are combined into one input stream, and further, two video streams and two audio streams are multiplexed in one input stream as the inputs of the main monitoring and preview respectively. Thus, when receiving the first video stream input to the preview window, the internal stream switching and copying are notified according to the synchronization signal identifier pre-added in the first video stream, realizing complete end-to-end preview and main monitoring picture frame synchronization, and since the same input source is decoded only once, the system consumption is reduced.

[0035] The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to be able to understand the technical means of the embodiments of the present invention more clearly, it can be implemented according to the content of the description. And in order to make the above and other purposes, features, and advantages of the embodiments of the present invention more obvious and understandable, the following specifically describes the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are only used to illustrate the embodiments and are not considered to limit the present invention. Moreover, throughout the drawings, the same reference numerals are used to denote the same components. In the drawings:

[0037] Figure 1 A schematic flowchart of the cloud live production method provided by an embodiment of the present invention is shown;

[0038] Figure 2 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0039] Figure 3 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0040] Figure 4 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0041] Figure 5 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0042] Figure 6 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0043] Figure 7 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0044] Figure 8 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0045] Figure 9 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0046] Figure 10 A schematic flowchart of the cloud live production method provided by another embodiment of the present invention is shown;

[0047] Figure 11 A schematic structural diagram of the cloud live production device provided by an embodiment of the present invention is shown;

[0048] Figure 12 A schematic structural diagram of the cloud live production equipment provided by an embodiment of the present invention is shown. Detailed Embodiments

[0049] Hereinafter, exemplary embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0050] Before describing the embodiments of the present invention, relevant terms, the data processing methods of the main monitor and preview monitor in the existing cloud live production, and the problems thereof will be described first.

[0051] Preview monitor: Preview the picture being produced, where the production in progress may include adding text, stickers, etc. If the user is satisfied with the production, they can send a picture push request to push the preview picture to the main monitor for live broadcast. If not satisfied, they can continue to produce (re-edit the text, stickers, etc., and then push it back to the preview) until satisfied and then push it to the main monitor.

[0052] Main monitor: Monitor the picture being broadcast.

[0053] The existing cloud live production generally adopts a completely independent scheme for the preview monitor and main monitor, which generally includes the following process:

[0054] Step A: When a certain video stream is pushed to the preview monitor, the cloud live production service opens this video stream and starts transcoding and special effect processing, and finally outputs it as a preview stream.

[0055] Step B: The preview window on the cloud live production management page pulls the preview stream in Step A for playback.

[0056] Step C: When the user initiates a request to push the preview to the main monitor on the cloud live production management page, the cloud live production service opens the video stream being processed by the current preview monitor again. After opening, the same transcoding and special effect processing as the preview monitor are performed, and finally it is output as a main monitor stream; among them, when the pictures of the preview monitor and the main monitor are the same, the transcoding strategy (or the video processing effect) adopted is the same, but different encoders are still required. When the pictures of the preview monitor and the main monitor are different, the transcoding strategies are different and different encoders are required.

[0057] Step D: On the cloud live production management page, the main monitor window pulls the main monitor stream in the aforementioned Step C for playback.

[0058] There are at least the following problems in the prior art:

[0059] 1. Since the preview stream and the main monitor stream need to open the input source separately, it may cause out-of-sync when reading the stream of the input source; and decoding the same input source twice causes waste of resources.

[0060] 2. At the same time, the preview stream and the main monitor stream perform transcoding and special effect processing on the input source separately. Since they work on different encoders, the transcoded pictures output may be out-of-sync.

[0061] 3. Since the preview monitor and the main monitor on the cloud live production management page use two players to play two streams respectively, due to the player cache, etc., it may cause the output pictures to be out-of-sync.

[0062] In summary, the prior art has the problem that the images of the main monitor and the preview monitor are out of sync.

[0063] Figure 1 The flowchart of the cloud live production method provided by the embodiment of the present invention is shown. This method is executed by a computer processing device. The computer processing device may include a mobile phone, a laptop, etc. As Figure 1 shown, the method includes the following steps:

[0064] Step 10: Parse the first video stream output to the preview window to obtain synchronization marker information; the synchronization marker information is pre-added identification information for characterizing whether the images of the preview window and the main monitor window are consistent.

[0065] In an embodiment of the present invention, the preview window and the main monitor window are displayed on the cloud live production management page. The cloud live production management page is used to receive the data stream sent by the cloud live production platform and display it to the user, so that the user can perform cloud live production operations on the data stream on the cloud live production management page. The cloud live production management page and the cloud live production platform are part of the cloud live production system in the embodiment of the present invention. The cloud live production system in the embodiment of the present invention includes multiple modules such as a stream receiving module, a transcoding module, and a cloud live production management page.

[0066] The entire business process of cloud live production in the embodiment of the present invention may at least include the following: Configure the relevant input source push stream URL (Uniform Resource Locator) on the cloud live production management page, and then use devices such as cameras to push streams to the input source push stream URL. Thus, the stream receiving service in the cloud live production service will receive the push stream. When the cloud live production management page switches the input source to the preview according to the user's cloud live production operation, the decoding service in the cloud live production service will decode the input source, and then according to the current request of the cloud live production management page for special effects processing on this stream (such as adding text, pictures, etc.), add the special effects corresponding to the special effects processing request to the decoded stream, and then re-encode and output the stream with special effects and push it to the stream receiving service. The data stream received by the stream receiving service is the input stream of the cloud live production management page, and then the cloud live production management page can pull the input stream for playback.

[0067] In the embodiment of the present invention, the main monitor and the preview are combined into one input stream, and further, two video streams and two audio streams are multiplexed in one input stream as the inputs of the main monitor and the preview respectively. Thus, when receiving the first video stream input to the preview window, the internal stream switching and copying are notified according to the synchronization signal identifier pre-added in the first video stream, realizing complete end-to-end frame synchronization of the preview and the main monitor images. Specifically, it can be determined whether the main monitor window and the preview window are consistent according to the user's cloud live production requirements, so as to correspondingly set the synchronization identifier information in the first video stream.

[0068] Among them, when determining the synchronization identification information according to the user's cloud live broadcast requirements, the screen switching of the main monitor only depends on the instruction of the cloud live broadcast console "pushing the preview to the main monitor". When this instruction is initiated, if the preview and the screen currently being played on the main monitor are the same, the actual screen of the main monitor will not switch. When this instruction is initiated, if the preview and the screen currently being played on the main monitor are different, the actual screen of the main monitor will switch to the same as the preview.

[0069] In an embodiment of the present invention, before step 10, it further includes:

[0070] Step 101: Obtain the screen processing request sent by the user.

[0071] In an embodiment of the present invention, the screen processing request can be sent by the user to the cloud live broadcast management page for processing the screen on the main monitor window and / or the preview window. The processing can include screen switching, editing the screen, etc. For example, the screen processing request can be used to request the display screen on the current window to switch from screen A to screen B, or the original screen is A, and then add special effects such as text and pictures to screen A.

[0072] Step 102: Determine the synchronization mark information according to the screen processing request.

[0073] In an embodiment of the present invention, since the main monitor screen and the preview screen use the same input stream, when the main monitor screen required by the user is the same as the currently displayed preview screen, the preview screen can be directly copied and then pushed to the main monitor screen, thereby realizing the synchronization of the main monitor screen and the preview screen. The screen processing request can represent the user's cloud live broadcast requirements, that is, whether it is necessary to immediately display the current preview screen on the main monitor screen. Therefore, the synchronization identification information can be correspondingly set according to the screen processing request received on the cloud live broadcast management page. Thus, according to the synchronization identification information, when the user needs the main monitor and the preview screen to be the same, the first video stream of the preview screen can be copied to obtain the target video stream to be pushed to the main monitor screen. At this time, since the target video stream is directly copied from the first video stream, the main monitor screen and the preview screen are synchronized.

[0074] Specifically, the synchronization identification information can be SEI, that is, Supplemental Enhancement Information. By pre-adding SEI representing whether the screens of the preview window and the main monitor window are the same in the first video stream, when pulling in the main monitor window, SEI can be parsed. When SEI represents that the screens of the preview window and the main monitor window are the same, the video stream currently being used in the preview window can be directly copied to the corresponding input channel of the main monitor window, thereby ensuring the synchronization of the main monitor and the preview.

[0075] Therefore, in one embodiment of the present invention, step 102 further includes:

[0076] Step 1021: Determine whether the images of the preview window and the main monitor window are consistent according to the screen processing request.

[0077] In one embodiment of the present invention, the user using the cloud live production service can be defined as a director, that is, the user is the subject who performs cut-in operations on the cloud live production management page. The director is only responsible for the cloud live production service, that is, the director only needs to care about when to switch to which screen, and does not need to understand the underlying implementation logic of the service. Therefore, after receiving the cut-in instruction sent by the director on the cloud live production management page, the cloud live production service will determine whether the images of the current preview and the main monitor are consistent, and then determine whether it is necessary to push the video stream corresponding to the current preview screen after copying it as it is to the main monitor window to synchronize the main monitor and preview screens.

[0078] Therefore, the corresponding screen on the cloud live production management page can be processed according to the screen processing request, and it is determined whether the currently displayed preview and main monitor screens after processing are consistent.

[0079] In one embodiment of the present invention, the screen processing request includes a screen switching request; the screen switching request is used to request to switch the preview screen to the main monitor screen;

[0080] Step 1021 further includes:

[0081] Step 103: When receiving the screen switching request, determine that the images of the preview window and the main monitor window are consistent.

[0082] In one embodiment of the present invention, the user can edit the image of the preview window according to their own cloud live production requirements and push the preview screen to the main monitor window. For example, after the user finishes editing the preview screen and is satisfied with the preview screen, the user sends a screen processing request to push the currently edited preview screen to the main monitor screen. Therefore, when receiving the screen switching request, it can be determined that what the user needs to see is that the main monitor screen is the same as the preview screen. Therefore, when receiving the request to push the preview screen to the main monitor window, it is determined that the images of the preview window and the main monitor window are consistent.

[0083] In one embodiment of the present invention, the screen processing request further includes a screen editing request; the screen editing request is used to perform editing processing on the preview screen;

[0084] After step 103, it further includes:

[0085] Step 1041: When receiving the screen editing request, process the preview screen according to the screen editing request to obtain a processed screen.

[0086] In an embodiment of the present invention, as described above, the user can first perform one or more screen processes on the preview screen in the preview window, such as adding special effects such as files and icons, or switching the input source of the preview window, such as switching the input source of the preview window from the previous data stream 1 to data stream 2. Therefore, the processed screen can be a screen with special effects added on the basis of data stream 1, or a screen after replacing data stream 1 with data stream 2.

[0087] Step 1042: Determine whether the processed screen is the same as the currently displayed screen of the main monitor window.

[0088] In an embodiment of the present invention, the processed screen is compared with the currently displayed screen of the main monitor window to determine whether they are the same, so as to judge whether the input source of the preview window can be directly copied and used by the main monitor window at this time.

[0089] Step 1043: If they are not the same, determine that the screens of the preview window and the main monitor window are inconsistent.

[0090] In an embodiment of the present invention, when the current main monitor screen and the processed preview screen are not the same, it means that the preview screen should not be played on the main monitor, that is, the main monitor screen and the preview screen required by the user are not the same.

[0091] Step 1022: If they are consistent, set the synchronization mark information to the first mark; the first mark is used to represent that the screens of the preview window and the main monitor window are the same.

[0092] In an embodiment of the present invention, the synchronization mark information can be SEI information, specifically, it can be a preset field, such as operationType (operation type), and the first mark can be setting operationType to 1.

[0093] Step 1023: If they are inconsistent, set the synchronization mark information to the second mark; the second mark is used to represent that the screens of the preview window and the main monitor window are not the same.

[0094] In an embodiment of the present invention, corresponding to the setting of the first mark, the second mark can be setting operationType to a value different from the first mark 1, such as 0 or 2.

[0095] Step 103: Add the synchronization mark information to the original video stream pushed to the preview window to obtain the first video stream.

[0096] In an embodiment of the present invention, in order to avoid the asynchronous situation that may occur during reading caused by the main monitoring window and the preview window opening the input source separately in the prior art, the embodiment of the present invention adopts sharing an input source for the main monitoring window and the preview window. To achieve sharing and considering that generally the preview screen is first displayed and processed, and then the preview screen is pushed to the main monitoring window, synchronous marking information can be added to the original video stream used by the preview window in the preview window, so as to obtain the first video stream currently shared and pushed to the main monitoring window. According to the synchronous marking information, it is determined whether the picture of the main monitoring window is consistent with that of the preview window, and when they are consistent, the first video stream is copied to obtain the target video stream of the main monitoring window.

[0097] It should be noted that in another embodiment of the present invention, when the main monitoring window is not opened, the cloud live production management page detects that the user initiates a guide cut request and forwards it to the cloud live production service. After receiving the request, the cloud live production service will determine that the current main monitoring is not opened, so it will not add synchronous identification information to the first video stream.

[0098] Step 20: When the synchronous marking information indicates that the pictures of the preview window and the main monitoring window are consistent, update the target video stream output to the main monitoring window according to the first video stream.

[0099] In an embodiment of the present invention, when the synchronous marking information indicates that the pictures of the preview window and the main monitoring window are consistent, the first video stream can be directly copied to the push channel of the main monitoring window, that is, to ensure that the target video stream is exactly the same as the first video stream, thereby realizing the synchronization of the main monitoring window and the preview window.

[0100] Therefore, in an embodiment of the present invention, step 20 further includes:

[0101] Step 201: When it is determined that the synchronous marking information is the first mark, perform a copy process on the first video stream to obtain a second video stream.

[0102] In an embodiment of the present invention, by directly copying the first video stream, there is no need to perform secondary decoding on the input source, ensuring that the picture on the main monitoring window is time-axis synchronized with the preview window from which it comes.

[0103] Step 202: Update the target video stream to the second video stream.

[0104] In an embodiment of the present invention, the second video stream copied from the first video stream is used as the target video stream and pushed to the main monitoring window, so that the main monitoring window and the preview window use the same input source, and the main monitoring and preview pictures are synchronized.

[0105] In one embodiment of the present invention, step 20 further includes:

[0106] Step 203: When it is determined that the synchronization marker information is the second marker, do not update the target video stream.

[0107] In one embodiment of the present invention, when the synchronization marker information is the second identifier indicating that the main monitoring screen is inconsistent with the preview screen, it means that the main monitoring screen and the preview screen are different, so there is no synchronization requirement. Therefore, there is no need to update the target video stream.

[0108] In still another embodiment of the present invention, the cloud live production process may further include at least the following:

[0109] Step 1: Assume that input source 1 is pushed to the preview. Then the cloud live production service starts to decode input source 1; obtain the video and audio of the decoded input source 1.

[0110] Step 2: At this time, it is determined that the main monitor is not turned on. Then directly perform relevant special effects and transcoding processing on the video of input source 1, and then output the transcoded video to video stream 1 in the output stream; transcode the audio stream of input source 1 and output it to audio stream 1 in the output stream. As shown in Figure 2 , the output stream contains two audio channels (audio stream 1, audio stream 2) and two video channels (video stream 1, video stream 2); at this time, there is no data in audio stream 2 and video stream 2 (because the main monitor is not turned on at this time). The cloud live production management page pulls this output stream for decoding and playing. Since the main monitor is not turned on, directly decode video stream 1 and render it in the preview window, and decode audio stream 1 for playing. There is no preview-main monitor frame synchronization problem.

[0111] Among them, the two audio channels in the two output streams respectively correspond to the audio of the preview and the main monitor, and the two video channels respectively correspond to the video of the preview and the main monitor. However, in the current situation, since the main monitor is not turned on, there is no data in audio stream 2 and video stream 2. In fact, in the output stream, 2 audio channels and 2 video channels are sufficient to cover all scene requirements, because the logic of copying or selecting different streams can be completed when switching the respective screens of the main monitor and the preview, and no more channels are needed.

[0112] It should be noted that regardless of the number of input sources, all the currently required input sources to be processed (such as using input source 1 and input source 2 to form a left-right layout screen, and the actual service will first combine these two streams into one stream) will be mixed into one stream in the cloud live production service first, and then processed according to the subsequent logic.

[0113] Step 3: When a command is sent on the cloud live production management page to push the preview to the main monitor, then as shown in Figure 3As shown, insert an SEI into video1 of the output stream. The SEI is of user-defined type and carries an operationType attribute with a value of 1, indicating that the preview and the main monitor use the same video stream at this time.

[0114] Step 4: The cloud live production platform pulls the output stream and continuously decodes it. When an SEI is encountered during the decoding of video1 and it is found that its operationType = 1, then as Figure 4 shown, copy a copy of the video of video1 for the main monitor to render, and at the same time copy a copy of the audio of audio1 for the main monitor to play; at this time, since the audio and video of the main monitor are copied from the preview, the main monitor and the preview are completely synchronized.

[0115] Step 5: At this time, assume that input source 2 is pushed to the preview. Then the cloud live production service first inserts an SEI into video2 of the output stream, with its operationType = 2, indicating that the preview and the main monitor use different video streams; at the same time as Figure 5 shown, start transcoding input source 2; at the same time, the already transcoding input source 1 continues to be transcoded; at the same time, the newly output video1, audio1, video2, and audio2 are aligned in PTS.

[0116] Step 6: As Figure 6 shown, the cloud live production platform pulls the output stream and continuously decodes it. When an SEI is encountered during the decoding of video2 and it is found that its operationType = 2, then output the video of video2 for the preview to render, and at the same time output the audio of audio2 for the preview to play; at this time, continue to output the video of video1 for the main monitor to render, and at the same time continue to output the audio of audio1 for the main monitor to play. And since the main monitor and the preview are different streams at this time, there is no frame synchronization problem between the main monitor and the preview.

[0117] Step 7: If the preview is pushed to the main monitor at this time, then as Figure 7 shown, insert an SEI into video2 of the output stream. The SEI is of user-defined type and has an operationType attribute with a value of 1, indicating that the preview and the main monitor use the same video stream at this time; at the same time, the newly output video1, audio1, video2, and audio2 are aligned in PTS.

[0118] Step 8: The cloud live production platform pulls the output stream and continuously decodes it, as Figure 8As shown in the figure, when SEI is encountered during the decoding of video2 and its operationType = 1 is found, at this time, a copy of the video of video2 is given to the main monitor for rendering, and at the same time, a copy of the audio of audio2 is given to the main monitor for playing; at this time, since the main monitor of the audio and video is copied from the preview monitor, the main monitor and the preview monitor are completely synchronized.

[0119] Step 9: If the input source 1 is pushed to the preview monitor again at this time, as Figure 8 shown, the cloud live production service first inserts SEI with operationType = 2 into video1 of the output stream, which means that different video streams are used for the preview monitor and the main monitor; at the same time, the input source 2 that is already being transcoded continues to be transcoded; at the same time, the input source 1 that is already being transcoded continues to be transcoded; at the same time, the newly output video1, audio1, video2, and audio2 are aligned in PTS.

[0120] That is, in the embodiments of the present invention, the value-taking rule of operationType is: if different video streams (or different video processing effects, such as layout merging) are used for the pictures to be played by the preview monitor and the main monitor, then operationType = 2; if the same video stream (or the same video processing effect, such as layout merging) is used for the pictures to be played by the preview monitor and the main monitor, then operationType = 1.

[0121] Step 10: The cloud live production platform pulls the output stream and continues to decode. As Figure 10 shown, when SEI is encountered during the decoding of video1 and its operationType = 2 is found, at this time, the video of video1 is output to the preview monitor for rendering, and at the same time, the audio of audio1 is given to the preview monitor for playing; at this time, the video output of video2 continues to be output to the main monitor for rendering, and at the same time, the audio of audio2 continues to be given to the main monitor for playing; since the main monitor and the preview monitor are different streams at this time, there is no frame synchronization problem.

[0122] Step 11: At this time, if other input sources are switched, or multiple input sources are merged and laid out, etc., the operation process and principle are the same as those in the above steps 1-10; thus, in a complete cloud live production process, the preview monitor and the main monitor maintain frame synchronization in all links, overcoming the problem that the preview monitor and the main monitor in the prior art cannot be completely frame-synchronized.

[0123] Among them, the process of realizing the synchronization of the main monitor and the preview monitor based on multiple streams can be as follows:

[0124] When multiple input sources converge, for example, when input source 1 and input source 2 are laid out side by side in a single screen (e.g., input source 1 on the left and input source 2 on the right) and pushed to preview, both input source 1 and input source 2 will be decoded simultaneously. The decoded images will then be mixed to obtain an image with a side-by-side layout, which is then placed in video1 of the output. The two-channel audio obtained after decoding will be mixed and placed in audio1 of the output.

[0125] Assume that the main monitor is not currently enabled, and then if you click to push to the main monitor, operationType = 1 will be inserted into video1.

[0126] The cloud live production platform continuously decodes the pulled output stream. When it encounters an SEI while decoding video1 and finds that its operationType = 1, at this time, a copy of the video in video1 will be copied for the main monitor to render, and at the same time, a copy of the audio in audio1 will be copied for the main monitor to play.

[0127] Then assume that the layout of the preview is now changed to (input source 2 on the left and input source 1 on the right) and pushed to preview. At this time, the images to be displayed on the preview and the main monitor will be inconsistent. At this time, the original stream with (input source 1 on the left and input source 2 on the right) remains unchanged and continues to be played on the main monitor. The cloud live production will newly layout input source 1 and input source 2 to form an image with (input source 2 on the left and input source 1 on the right). Then, operationType = 2 will be inserted into video2 first, and then this image will be inserted into video2. The mixed audio will be inserted into audio2.

[0128] In the embodiments of the present invention, by merging the preview and main monitor streams into one and using relevant notification mechanisms, internal stream switching and copying technologies ensure the frame synchronization of the preview and the main monitor. Specifically, the embodiments of the present invention adopt the method of sharing the same input source for the preview stream and the main monitor stream, eliminating the asynchrony of reading the input source; moreover, the input source is decoded only once, reducing resource waste; the preview and the main monitor on the cloud live production management page pull the same stream and then perform copy rendering, eliminating the asynchrony of the images on the playback side, and finally achieving complete end-to-end frame synchronization of the preview and the main monitor images.

[0129] In an embodiment of the present invention, the first video stream output to the preview window is parsed to obtain synchronization marker information; the synchronization marker information is pre-added identification information for characterizing whether the pictures of the preview window and the main monitor window are consistent; when the synchronization marker information characterizes that the pictures of the preview window and the main monitor window are consistent, the target video stream output to the main monitor window is updated according to the first video stream. In the embodiment of the present invention, the main monitor and the preview are combined into one input stream, and further two video streams and two audio streams are multiplexed in one input stream as the inputs of the main monitor and the preview respectively. Thus, when the first video stream input to the preview window is received, the internal stream switching and copying are notified according to the synchronization signal marker pre-added in the first video stream, achieving complete end-to-end preview and main monitor picture frame synchronization, and since the same input source is decoded only once, the system consumption is reduced.

[0130] Figure 11 The structural schematic diagram of the cloud live production device provided by the embodiment of the present invention is shown. As Figure 11 shown, the device 30 includes: a parsing module 301 and an updating module 302.

[0131] Among them, the parsing module 301 is configured to parse the first video stream output to the preview window to obtain synchronization marker information; the synchronization marker information is pre-added identification information for characterizing whether the pictures of the preview window and the main monitor window are consistent;

[0132] The updating module 302 is configured to update the target video stream output to the main monitor window according to the first video stream when the synchronization marker information characterizes that the pictures of the preview window and the main monitor window are consistent.

[0133] In the cloud live production device provided by the embodiment of the present invention, the first video stream output to the preview window is parsed to obtain synchronization marker information; the synchronization marker information is pre-added identification information for characterizing whether the pictures of the preview window and the main monitor window are consistent; when the synchronization marker information characterizes that the pictures of the preview window and the main monitor window are consistent, the target video stream output to the main monitor window is updated according to the first video stream. In the embodiment of the present invention, the main monitor and the preview are combined into one input stream, and further two video streams and two audio streams are multiplexed in one input stream as the inputs of the main monitor and the preview respectively. Thus, when the first video stream input to the preview window is received, the internal stream switching and copying are notified according to the synchronization signal marker pre-added in the first video stream, achieving complete end-to-end preview and main monitor picture frame synchronization, and since the same input source is decoded only once, the system consumption is reduced.

[0134] Figure 12The structural schematic diagram of the cloud live production device provided by the embodiments of the present invention is shown. The specific embodiments of the present invention do not limit the specific implementation of the cloud live production device.

[0135] As Figure 12 shown, the cloud live production device may include: a processor 402, a communications interface 404, a memory 406, and a communication bus 408.

[0136] Among them: the processor 402, the communications interface 404, and the memory 406 communicate with each other through the communication bus 408. The communications interface 404 is used to communicate with network elements of other devices such as clients or other servers. The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the above embodiments of the cloud live production method.

[0137] Specifically, the program 410 may include program code, and the program code includes computer-executable instructions.

[0138] The processor 402 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention. One or more processors included in the cloud live production device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0139] The memory 406 is used to store the program 410. The memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0140] The program 410 can specifically be called by the processor 402 to cause the cloud live production device to perform the following operations:

[0141] Parse the first video stream output to the preview window to obtain synchronization mark information; the synchronization mark information is pre-added identification information for characterizing whether the pictures of the preview window and the main monitor window are consistent;

[0142] When the synchronization mark information characterizes that the pictures of the preview window and the main monitor window are consistent, update the target video stream output to the main monitor window according to the first video stream.

[0143] In the embodiment of the present invention, the cloud live production device analyzes the first video stream output to the preview window to obtain synchronization mark information; the synchronization mark information is pre-added identification information used to characterize whether the pictures of the preview window and the main monitor window are consistent; when the synchronization mark information characterizes that the pictures of the preview window and the main monitor window are consistent, the target video stream output to the main monitor window is updated according to the first video stream. In the embodiment of the present invention, the main monitor and the preview are combined into one input stream, and further two video streams and two audio streams are multiplexed in one input stream as the inputs of the main monitor and the preview respectively. Thus, when receiving the first video stream input to the preview window, internal stream switching and copying are notified according to the synchronization signal identification pre-added in the first video stream, achieving complete end-to-end preview and main monitor picture frame synchronization, and reducing system consumption because the same input source is decoded only once.

[0144] The embodiment of the present invention provides a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction runs on the cloud live production device, the cloud live production device is enabled to execute the cloud live production method in any of the above method embodiments.

[0145] The executable instruction can specifically be used to enable the cloud live production device to perform the following operations:

[0146] Analyze the first video stream output to the preview window to obtain synchronization mark information; the synchronization mark information is pre-added identification information used to characterize whether the pictures of the preview window and the main monitor window are consistent;

[0147] When the synchronization mark information characterizes that the pictures of the preview window and the main monitor window are consistent, update the target video stream output to the main monitor window according to the first video stream.

[0148] The executable instruction stored in the computer storage medium provided by the embodiment of the present invention analyzes the first video stream output to the preview window to obtain synchronization mark information; the synchronization mark information is pre-added identification information used to characterize whether the pictures of the preview window and the main monitor window are consistent; when the synchronization mark information characterizes that the pictures of the preview window and the main monitor window are consistent, the target video stream output to the main monitor window is updated according to the first video stream. In the embodiment of the present invention, the main monitor and the preview are combined into one input stream, and further two video streams and two audio streams are multiplexed in one input stream as the inputs of the main monitor and the preview respectively. Thus, when receiving the first video stream input to the preview window, internal stream switching and copying are notified according to the synchronization signal identification pre-added in the first video stream, achieving complete end-to-end preview and main monitor picture frame synchronization, and reducing system consumption because the same input source is decoded only once.

[0149] An embodiment of the present invention provides a cloud live production device for performing the above-mentioned cloud live production method.

[0150] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a cloud live production device to perform the cloud live production method in any of the above method embodiments.

[0151] An embodiment of the present invention provides a computer program product. The computer program product includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions run on a computer, the computer is enabled to perform the cloud live production method in any of the above method embodiments.

[0152] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings provided herein. The structure required to construct such a system will be apparent from the above description. In addition, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the present invention described herein can be implemented using various programming languages, and the description of a particular language above is for the purpose of disclosing the best mode of the present invention.

[0153] In the specification provided herein, a large number of specific details are set forth. However, it can be understood that the embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0154] Similarly, it should be understood that, in order to streamline the present invention and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0155] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from those of the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and they can also be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all the features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise clearly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) can be replaced by an alternative feature that provides the same, equivalent, or similar purpose.

[0156] It should be noted that the above embodiments illustrate the present invention rather than limit the present invention, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the unit claims listing several devices, several of these devices can be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A cloud live production method, characterized in that, The method includes: Parsing a first video stream output to a preview window to obtain synchronization mark information; the synchronization mark information is pre-added identification information for characterizing whether the pictures of the preview window and the main monitor window are consistent; When the synchronization mark information characterizes that the pictures of the preview window and the main monitor window are consistent, updating a target video stream output to the main monitor window according to the first video stream.

2. The method according to claim 1, wherein Before parsing the first video stream output to the preview window to obtain synchronization mark information, it further includes: Obtaining a picture processing request sent by a user; Determining the synchronization mark information according to the picture processing request; Adding the synchronization mark information to the original video stream pushed to the preview window to obtain the first video stream.

3. The method according to claim 2, characterized in that, The determining the synchronization mark information according to the picture processing request further includes: Determining whether the pictures of the preview window and the main monitor window are consistent according to the picture processing request; If they are consistent, setting the synchronization mark information as a first mark; the first mark is used for characterizing that the pictures of the preview window and the main monitor window are the same; If they are inconsistent, setting the synchronization mark information as a second mark; the second mark is used for characterizing that the pictures of the preview window and the main monitor window are different.

4. The method according to claim 3, characterized in that The picture processing request includes a picture switching request; the picture switching request is used for requesting to switch the preview picture to the main monitor picture; the determining whether the pictures of the preview window and the main monitor window are consistent according to the picture processing request includes: When receiving the picture switching request, determining that the pictures of the preview window and the main monitor window are consistent.

5. The method according to claim 4, characterized in that The picture processing request further includes a picture editing request; the picture editing request is used for editing the preview picture; after determining that the pictures of the preview window and the main monitor window are consistent when receiving the picture switching request, it includes: When receiving the picture editing request, processing the preview picture according to the picture editing request to obtain a processed picture; Determining whether the processed picture is the same as the currently displayed picture of the main monitor window; If they are not the same, determining that the pictures of the preview window and the main monitor window are inconsistent.

6. The method according to claim 3, wherein The updating the target video stream output to the main monitor window according to the first video stream when the synchronization mark information characterizes that the pictures of the preview window and the main monitor window are consistent includes: When determining that the synchronization mark information is the first mark, performing a copy process on the first video stream to obtain a second video stream; Updating the target video stream to the second video stream.

7. The method according to claim 3, wherein After parsing the first video stream output to the preview window to obtain synchronization mark information, it further includes: When determining that the synchronization mark information is the second mark, not updating the target video stream.

8. A cloud live production device, characterized in that, The device includes: A parsing module, configured to parse a first video stream output to a preview window to obtain synchronization mark information; the synchronization mark information is pre-added identification information for characterizing whether the pictures of the preview window and the main monitor window are consistent; An update module, configured to update the target video stream output to the main monitoring window according to the first video stream when the synchronization marker information indicates that the picture of the pre-monitoring window is consistent with that of the main monitoring window.

9. A cloud live production device, characterized in that, It includes: A processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the operations of the cloud live streaming method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, At least one executable instruction is stored in the storage medium. When the executable instruction runs on the cloud live streaming device, it causes the cloud live streaming device to execute the operations of the cloud live streaming method according to any one of claims 1-7.

Images